Search Disabled |
|
|
|
Search Options | Help | Site Map | Cultivate Web Site | ||||
|
|
|||||
| Home | Current Issue | Index of Back Issues |
| Issue 9 Home | Editorial | Features | Regular Columns | News & Events | Misc. | ||
By Marcel van Dijk - February 2003
In 2001, for the first time in its history, the Amsterdam Municipal Records Office received and processed records that had been electronically created by a government authority. In the course of the project a great many obstacles were faced and ultimately resolved. In his article on the ups and downs of this difficult process, Marcel van Dijk gives us a flavour of the challenges encountered and the hard lessons learnt.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The Dutch Government is going digital, and the archival world is experiencing the much heralded consequences. Inside governmental administrations, document management, and with that good record management, is subject to heavy digital turbulence and insecurity. At stake were the following two issues: How can our governmental organisations, in an electronic environment, record and apply their information to support their management, and render an account of their actions? And further: how shall the transfer of electronic records to the governmental record offices be implemented? Last year, in the Netherlands a regulation was issued on the arrangement and accessibility of records with archival value. Special attention was given to electronic records, as our government's memory and our national electronic cultural heritage were at stake here!
Many a Dutch governmental authority will find it hard to comply with these regulations. The Amsterdam Municipal Records Office [1] often finds that many municipal authorities fail to practise good records management, especially where electronic records are concerned. We cannot emphasise enough the risks that are being run at times with the accounting for municipal implementation procedures and the keeping of our historical and cultural heritage.
The Amsterdam Municipal Records Office has some experience now appraising electronically created archives. The Amsterdam Projectbureau 'Year 2000', handling the Millennium problem, transmitted its records after closing down, eleven metres of paper records and ... nine CD-ROM's. According to Dutch regulations, the Projectbureau should have appraised these records as well as prepared them for transfer. But on that particular occasion, an agreement was made that the Municipal Records Office would accomplish this task. I shall only go into some of the details of our experience, nor do we claim to have the solution to everyone else's digital challenges out there. However I trust this article will serve as a general warning however, and be of benefit to the many municipal (and other) record creators and records offices.
This Projectbureau was installed in 1998 by the Municipal Board to secure the transfer into the new millennium. Key activities were appraising the reliability of hard- and software used in municipal departments in terms of the millennium rollover, briefing the municipal authorities on these matters and taking care of internal and external communications. These activities were, for a large part, electronically supported, and this of course reflected on the bureau's records.
 |
| Figure 1: 'Bugman', the mascot of the Bureau Year 2000 |
The Projectbureau had to deal with a variety of questions. What data was stored in the bureau's databases? What PowerPoint presentation was shown supporting the municipal executive's speech warning about the Millennium bug? And what about the Web site specifically set up to inform citizens and authorities? These questions could partly be answered by the paper documentation, but consulting the electronic documents provides many more insights and illustrations.
On the bureau's closure, its records were transmitted to the Records Office. The paper components had already been appraised to a fair degree. We subsequently processed them and provided further archival description and material preservation: business as usual.
 |
| Figure 2: Viruses can endanger the integrity of electronic records |
Initially, in global terms, stocktaking of the CD's gave us immediate cause for concern. In the process of making back-ups, the virus checker sounded the alarm. The ICT department successfully disinfected the CD's without affecting the files' integrity. However a far more serious problem lay in the structure in which the information was organised. The content of shared and individual workstations had been burned directly onto the CD's, structured in maps which referred back only partially to the structure of the paper records. As a consequence we were confronted with a whole raft of questions: What was on the CD's, and what was in the files? Had this Word document ever actually been sent as a letter? What purpose had this spreadsheet served? At what meeting had this picture been taken? And what software do I need to open this file? All too often the necessary context data was missing, that is: the metadata.
Administrating metadata of hard- and software with which electronic records are created is, of course, vital. But equally without the information on the context in which those records were received or created, it would soon be very hard to interpret such records. The contextual information can in many cases easily be attached by implementing electronic filing. It is therefore of utmost importance to start the business of filing as soon as possible in the overall process; this means basically providing the files, as well as the electronic documents stored into them, with a plain description of the content. Then, electronic records could _ provided procedures are implemented to cover issues such as version management and document status _ be appraised at directory level.
 |
| Figure 3: Poorly named files hamper the appraisal of records |
But as it was, in directories with names such as 'E-mail' or 'miscellaneous' (as we sometimes found in these records) we could never be sure what we would come across next, particularly when documents bore names such as '3rd concept' , 'A,dam ll' or 'ATT01368', (see figure 1). The documents in these directories therefore had to be opened and inspected one at a time. A very labour-, time- and money-consuming affair! Lack of structure and lack of explicit and interpretable metadata therefore lay very much at the root of the problems that confronted us in our aim to prepare these records for transfer to our electronic depot [2].
The global stocktaking of the content of the CD's took us, for reasons mentioned above, about one week. Thereafter we concluded that appraisal [3] in some cases was appropriate on the level of the CD, in some cases at directory level but more often than not at document level, depending on the way the directories had been defined. Moreover, we had to deal with a large variety of software and files: MS-Word and WordPerfect files, e-mail, Web pages, databases, spreadsheets, PowerPoint files, photographs in JPEG and other JPEG files such as logos and other images. We even found a computer game, made with the aim of preparing Amsterdam for the turn of the millennium. We defined a project plan by formulating the necessary work in relation to file type. The problem was, of course, that it was difficult to estimate how much time we would need to carry out the different activities, as we had no experience whatsoever of these file types. And, on top of that, we were confronted regularly with unexpected problems.
As to text files, these included Word- and WordPerfect files as well as e-mail messages and scanned letters and accounts. We decided on the guiding principle that the electronic files would serve as complementary to the paper records, in such a way that the paper records would be pre-eminent. Some of the paper records clearly had more authenticity than the electronic records, as they were signed or stamped. Furthermore, the paper records had already been appraised and an archival description had been made. Our next task therefore was to find out whether or not the electronic text files had already been included in the paper records. When this was the case, the electronic version was not used. Alternatively, and when we were dealing with a record with archival value, we printed the electronic file and added it to the paper records.
Accordingly the CD's had to be checked as to which documents on them needed to be retained and, more specifically, whether these documents were already included in the paper records. Both the paper records' archival description and the notes made at the first stocktaking of the CD's supported this check. Even so, a fairly large proportion of the files had to be opened and checked, as a direct consequence of the way documents had been filed and named. As mentioned earlier, it was sometimes very hard to determine with what kind of document we were dealing. In figure 3 we see an e-mail message with the text: 'Joep: three times; kind regards, Henk'. The three attachments were entitled 'acceptation protocol concept 02, 03 and 04'. Nowhere was a statement to be found concerning the context in which the protocol had been created, where we could find version 01, or the final version and whether or not, and to whom, the protocol had finally been sent.
These are random examples of the lack of information we encountered. Versions of the same text files were found in different directories and even on different CD's. A Word file could form part of the bureau's Web site, without being referenced or identified. Only after considerable time and effort, and after searching the paper records repeatedly, were we able to appraise the electronic records a little bit faster. This way of proceeding proved to be very labour-intensive, as a thousand files were involved. We considered other ways, like throwing all text files in a big 'basket' and sorting out the important files, helped by different sorting routines. However this proved not to be very effective, as the structure, still apparent in some places, would then completely disappear. Conversely sorting routines at directory level proved to be effective. Directories that seemed promising were checked at random in this manner. Moreover, we used several search engines, inserting arguments connected with functions, tasks and activities of the bureau.
In this way, the electronic records could be worked on quite thoroughly. The result: practically all the electronic text files were either already included in the paper records or for other reasons were without archival value. But this did not hold for all documents, (for example not for the communication plan and for documents concerning the organisation of a meeting with Turkish entrepreneurs); so ultimately it proved wise not to have skipped the stocktaking of all the electronic records!
As to the other file types, it was clear that, if we chose to save them, they had to be saved digitally. Files such as Web sites and databases could be printed, but they would lose their digital functionality. In particular with regard to the Year 2K Web site, including sound fragments and animations, we judged such a loss of functionality as unacceptable; the appraisal of these files had been made, to a significant degree, on the basis of their digital capabilities. Ultimately our solution was to copy the content of the CD's to a file server-disk.
In the Netherlands, the issue and rationale of archiving Web sites has only recently gained some public attention. Hence, we could only take limited advantage of earlier experience. In this particular case, the Web site had played such an important role in the bureau's primary goals, (guidance and communication), that we decided to retain completely the first and last versions. A number of intermediate versions, updates by which more or different information was added, and consequently representing a substantial change, were kept as well. In fact we found on the CD's only one complete site, plus 33 updates. These files had been stored to disk as zip files and had first to be unpacked. Thereafter, we were able to add the subsequent updates to the only complete version we had. In this way we re-created the various updates until we reached the last and 33rd.version, hoping it would function. This not only took considerable time, but also called for an enormous amount of disk space. Starting out with 299 zipped Web files, with a volume of 27.9 Mb, after unpacking we counted 2,967 files with a volume of 70.6 Mb. Once updating was complete, we counted 32,874 files with a volume of 370 Mb!
The result was not altogether satisfying: not all parts of the site functioned. On top of that, we found out that the oldest version of which we disposed was not in fact the first version of the Web site, something which an adequate set of metadata would have made clear from the start. Ultimately, we contacted the Web site's builder, in this case a small commercial designer. This firm provided us with the first version and the last (functioning) version of the site. Due to the relatively short interval between the design of the site and its transfer to the Municipal Records Office, valuable contact had still been possible with the designer. However over the same period, many comparably small ICT firms had been obliged to shut down and had disappeared. Who would have given the archivist the information he or she needed about such a Web site, built by such a small firm? This experience was one of the reasons why the Amsterdam Municipal Records Office began formulating a policy for the archiving of municipal sites. In this policy statement due attention will be given to adequate terms of transfer (e.g. probably earlier than required by existing regulations), required metadata, file formats and archiving of hyperlinks.
For some of the databases and spreadsheets found on the CD's, the lack of version management on the one hand and meta-information on the other, had an even greater impact then in the case of the text and Web files. Tables and schedules named by numbers only do not have the slightest meaning. Luckily, some spreadsheets and databases were accompanied by some information about the content and some technical information. On the accompanying document, the content of the files was made explicit, as well as which matches existed between the databases. Unfortunately the matches were omitted in the process of copying the databases onto the CD's, consequently most databases could not be opened.
Since it was these same databases which could give us a unique insight into the state of ICT at the municipal authority of Amsterdam on the verge of the New Millennium, we again felt obliged to contact the database designer. He provided us with a lot of information, from which it became clear that the databases contained a great deal of data, but also lacked a lot of data. Finally we preserved the database containing the result of the first stocktaking of the state of ICT in Amsterdam as executed by the bureau. This provides a very clear, though incomplete, picture of things before the bureau started its effort. Future researchers consulting these records shall have to bear their incompleteness in mind.
A number of files from the records of the Project Bureau Year 2000 have been selected for inclusion in the Amsterdam Municipal Records Offices' digital repository. These include pictures of publicity campaigns and gatherings, images, like the Bureau's logo, an educative computer game concerning the Millennium problem in Amsterdam, functioning Web site versions, some PowerPoint presentations and a database. Contents are disclosed by means of an archival description and a set of technical metadata which come with the records.
The project is far from over. Many questions remain unanswered. For example two substantial questions are: how are we going to keep these materials accessible and interpretable over the years to come? Moreover how are we going to provide the public with access to these records? In the digital age, management of - in principle - archival records is really a dynamic issue. This project shows in essence one thing very clearly: when archiving of electronic records is not properly planned and projected right from the start, as with a proper administration of the contextual, administrative and technical metadata, then any reconstruction afterwards is bound to be a most laborious and time-consuming affair with an uncertain outcome. The chances of success decrease with the passing of time. Good procedures and agreements with record creators beforehand together with a strict 'admission policy' are indispensable to ultimate success.
To make a long story short: everyone who wants to conserve history shall, far more than in the past, have to look to the future. And this does not only go for Amsterdam. For example, could the way electronic records are archived by the European Commission set an example here .. or perhaps not yet?
 |
| Figure 4: Electronic government...accessible archives? |
So in conclusion, the lessons learnt boil down to this: the storage of digital records has to be organised at the very moment of their creation; trying to manage them after that moment will cost everyone concerned a disproportionate and unjustifiable amount of effort.
I would very much like to thank Carolien Schönfeld, also of the Gemeentearchief Amsterdam, for her invaluable support on this article, both for her comments on its content and advice over points of translation.
Marcel van Dijk
Gemeentearchief Amsterdam - Amsterdam Municipal Records Office
Afdeling Verwerving en concerndiensten - Acquisition and Services Department
pb 51140
1007 EC Amsterdam
The Netherlands
Tel: +31 20 5720312
Fax: +31 20 5720326
URL: <http://www.gemeentearchief.amsterdam.nl/>
Email: mvandijk@gaaweb.nl
Marcel van Dijk has been working for the Municipal Archives of Amsterdam since 2000. In charge of the Municipal Archives, he supports and advises governmental organisations in Amsterdam which face problems in the sphere of record management.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
For citation purposes:
van Dijk, M "It Always Hurts the First Time: Experiences with transferred electronic records", Cultivate Interactive, issue
9, 7 February 2003
URL: <http://www.cultivate-int.org/issue9/amsterdammro/>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Related articles:
If you would like to view similar articles to this one click on a key word below:
< - Amsterdam Records Office - archiving - record appraisal - digital formats - record creation policy - >
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
|