Phase 1

Cataloging and Organizing the Documents

Before any digitization project can begin in earnest, the 400,000 (plus) documents of which the A.A.R.C. is in possession, must first be inventoried, catalogued and much better organized than is presently the case. Most, if not all, of the documents are already tagged with N.A.R.A.'s Records Identification Form codes. This allows each document to be automatically fed into scanners and "tracked" by a particular identification code.

Additional categorization and document preparation is necessary to:

1)	Verify that the documents are physically appropriate for automated scanning equipment; that they have R.I.F. numbers assigned, etc.

2)	If RIF identification is to be automated (unknown), apply press-on RIF tags to each document's first page, in one of a few pre-defined "allowable locations."

3)	Organize the documents chronologically in high-level subject categories (e.g., "Oswald in Mexico," "New Orleans," "Dallas," "the E.R. at Parkland Hospital," "the autopsy at Bethesda," and so forth) to elaborate further on their current, very simple delineation by the mere abbreviation of their issuing agency (CIA; FBI; HSCA, etc.).

4)	Prepare a prioritized list of document sets to be digitized. Document sets which can be identified as most important to the research community will be scheduled for digitization first.

This phase of the project necessarily entails a small team of people whose members are familiar with the subject matter and can be brought "up-to-speed" on our project's and the agencies' classification schemes. These people would, necessarily, have to spend some number of days or weeks in Washington (on-site) at the A.A.R.C. (or a suitable alternative, nearby location). This phase requires no special equipment other than a computer, for entering the master-document-set list info.

Phase 2:

Digitization and Basic Access

Digitization

Simply digitizing such a massive collection of documents is, in itself, a technical challenge. A single scanning station with an automatic feeder, one capable of processing 5 pages per minute and which is running 8 hours per day, would (statistically!) entail 2000 days of operation, more or less, to scan all of the ARRB's releases. So scale is an essential and rudimentary issue. The most practical options in this regard are as follows:

1)	Multiple Pentium-II-based scanning stations (2 or 3), located at the AARC or in some place where documents can be delivered. Each station consists of a PC running Windows NT, a medium- or high- speed sheet-fed scanner, and appropriate software (probably Adobe Acrobat Capture). The cost of each station would be $5000 - $8000, with more than half the cost devoted to the scanner itself. Each such station might be able to process 2-3 pages per minute. 3 such stations, manned 8 hours per day, might be able to scan the entire collection in somewhat more than a year.

2)	One or more "high end" scanning stations, capable of greater throughput, perhaps 5-10 pages per minute. The cost of such systems is unknown at this point.

3)	Out-source the scanning process to a service bureau. Such organizations have the equipment and personnel to tackle such jobs with relative ease, though at some expense. Note that this option presumably requires that each box of documents leave the AARC for some period of time (probably a few weeks).

The output of Phase 2 would be documents in Adobe Portable Document Format (PDF); one file per multi-page document. This format combines text and graphics, to give a version of each page which looks just like the original, but with text recognized and available to search engines. PDF has rapidly become a standard for document representation, and is viewable on all computer platforms. However, due to the poor quality of much of the document set, hand-cleanup of OCR (Optical Character Recognition) is simply not practical. The solution is to use Adobe's "Image with Hidden Text" PDF format variant. In this format, the image from which text is extracted is retained for viewing purposes. The text is made available to search engines, while a "photocopy" of the page is what is presented visually. The text available for searching would not be perfect--5% to 10% would likely be mis- recognized. Still, this makes the bulk of the document data available for full-text searching, something not even possible at all at the National Archives. In this format, a typical page would require up to 100K for storage if scanned at 300 dots per inch, which provides good visual detail. Thus a 5 million page collection would require roughly 500 gigabytes (100 DVD-ROM-sized optical disks) to store. This is an eminently practical storage size.

The computer system on which these documents are to be stored should also be directly connected to the Internet, to allow access to the collection as soon as possible (i.e., as it is being formed). This system should include:

1)	A fast PC workstation running Windows NT or Linux

2)	Web server software and connection to Internet via T1 or fractional T1 line

3)	Optical disk or magneto-optical disk jukebox. Plasmon sells magneto-optical disk jukeboxes in 540 and 810 gigabyte sizes (and larger).

4)	Good-sized hard disk (20-40 gigabytes) for eventual storage of full text search index for entire collection.

Basic Access

Even raw digital documents accessed by R.I.F. number, are useful to some researchers, if made available somehow. Here is a proposal for a very simple system that could be made available:

The first iteration of access requires the use of R.I.F. numbers. These may be obtained using the N.A.R.A. search engine or by other means (published lists created by other researchers, for instance). Researchers may submit such a list to the AARC in one of the following via email, letter, or via a simple interface on the AARC web site. The requested documents are supplied by the AARC in either of two methods, as desired by the researcher:

1)	Directly over the Internet (as PDF files). This is only appropriate for researchers with high-speed connections, and those requesting only a few pages at a time.

2)	Via CD-ROM. A second workstation, networked to the server, is set up to burn CD-ROMs on demand. Up to 5000 pages may fit onto a single CD-ROM, making this mechanism ideal for large requests.

It would also be desirable for knowledgeable people to identify a set of documents as an interesting collection, and make these available for ordering over the AARC web site. CD-ROM copies of popular data sets could be pre-burned instead of handled as orders come in.

Presuming that payments to the AARC for this service should at least cover costs, but not necessarily try to pay back the development of the system, then such nominal charges might be: $10 for cost of media and processing, and then 1 or 2 cents per page of document.

The personnel functions which would be needed, on site at the AARC offices or at some other location chosen to run this center (there is absolutely no requirement that this service be run at any particular location), during this phase include:

1)	Management of computer system; backups, etc.

2)	Development and maintenance of web site

3)	Overseeing the scanning of documents

4)	Running the scanners

5)	Processing of requests for CD-ROMs collections

A small staff of 2 or 3 people could run the entire project, assisted by interns or volunteers to help run the scanning stations (assuming scanning is not out- sourced). One of the staff members should be a skilled computer administrator and programmer.

Estimated one-time equipment costs are:

1)	$10,000 for web server PC, not counting optical jukebox

2)	$50,000 for 500 Gb optical or magneto-optical jukebox

3)	$5,000 for CD-burning workstation

(plus ongoing costs for T1 or frac-T1 Internet connection)

Additionally, if scanning is done in-house there is the need for multiple (2 or 3) scanning stations at a cost of $5000 to $8000 each.

Note that the Internet access and CD-ROM publication services in this phase could begin within a few months of the initiation of scanning, particularly if careful attention is paid to choosing which document sets should be scanned first.

Phase 3:

Indexing, Summarizing, and Structured Online Access

While the Phase 2 document access services are useful to some researchers, the volume of documents cries out for more organization and access methodologies. This could occur while Phase 2 is ongoing, and consists of the following:

Applying commercial search software to build a full-text index of the collection. This would be accessed by a commercial or custom-written "front end" which allows queries to be made on all or part of the database. The results of such queries are R.I.F. numbers, which can then be downloaded or ordered by the methods developed in Phase 2.

Creating short summaries of the most important documents. This can be partially automated, and supplemented by human hand for important selected documents. Simple custom software could be developed to look for "To" and "From" fields in FBI memos, for instance, and extract that information into automatically generated summaries. Search hits on documents which were tagged with a summary could then display that summary on demand.

The second aspect of this phase is to make its work product available. This means creating a web-based document search system more sophisticated than the "submit RIF numbers" of Phase 2. The reasonably simple but very powerful system envisioned would make use of both the full-text search index created automatically, and keywords and summary information attached to each document by custom software. This would result in a system somewhat similar to the NAIL system set up at N.A.R.A., wherein users may type in search parameters and receive the results. This system is more useful than that at N.A.R.A., however, because:

The document keyword and summary information is more useful. The "hits" of a given search can be identified more readily as being relevant.

For researchers with high-speed Internet connections or the need for only a few pages, documents may be downloaded on demand directly over the Internet. Note that at a slow connection speed of 28.8 kbps (a connection speed all too common today), each page would like take a half a minute. Researchers with cable modems or ADSL connections (more and more people all the time) could download pages in a few seconds each.

Documents identified as desired through this system may be "check- marked" and put into a document list. This list can then be submitted to the same process already developed in Phase 2, which is used to ship high-quality document page images on CD-ROM to the recipient.

Besides searching for documents via keywords, agency names, dates, and other such information, it would be useful to "organize" documents into some classification scheme. Actually, since the documents are inherently identified by the system via R.I.F. numbers, there is no reason why multiple such classification schemes couldn't co-exist. One might use originating agency as the highest-level breakdown, whereas another might use the investigative phase (Warren Commission, Garrison investigation, Church Committee, HSCA, etc.) as the primary organizing mechanism. The search engine envisioned would combine full text search and keyword search into one interface.

Phase 4:

Analysis and Presentation

Beyond the simple though large-scale digitization and basic classification of documents, it would be wonderful if the AARC could also serve as a repository for analysis of document sites or cross-document ideas. In paper format, journal articles or books are limited to citing testimony, exhibits, or documents in footnotes. On-line at the AARC, though, each such reference could be a hyperlink to the documents being discussed. Such "articles" could range from focused discussions of a small set of documents to length material such as a high-level overview of the 15 volumes of Warren Commission testimony, complete with hyperlinks to any particular appearance under discussion.

Here are some examples of the forms of analysis which could be stored on- line:

Document set descriptions - Focused articles on document sets, which describe the contents of that document set. If a serious effort is made to construct a "directory tree" as contemplated in phase 3, it would be great to allow researchers to contribute short (1 or a few pages) summaries of a given set.

Walkthroughs - The word "walkthrough" here refers to an article which describes briefly several documents or document sets. The walkthrough may try to "make a point" via such an ordered presentation of documents, or may simple try to collect under one roof an overall list of the documents relevant to a given issue. The walkthough would be in the form of an HTML page or pages, with hyperlinks to all the referenced documents.

Analyses - The term "analyses" here refers to what might normally be presented as a journal article or even a book, but here enhanced by the fact that all referenced documents are available via hyperlinks. [note: I like the idea of receiving unsolicited help with the task of interpretation and decoding over the Internet. It could very much be a two-way street.]

Discussion Threads - The AARC could host discussion threads among researches, and again enhance them by the ability for a given posting to be able to reference documents directly.

Documentary - In this setting, the term "documentary" is used to identify analyses which are primarily audio-visual. The AARC could aid researchers and educators in creating multi-media documentaries on-line. The body of photographic and audio exhibits for instance, could be available as source material for educational presentations on a given topic. The AARC might make available a "kit" which could be used by authors to put together such presentations in the proper format.

Obviously the sky is the limit, depending on how much the AARC views its role as a document archive versus an active education and research center, and how much money and resources can be applied to the tasks outlined here. The purpose of the above discussion is to show ways in which the base document set, once in digital form, can foster innovative research and educational uses.

No extra equipment is needed for this phase, except to the extent that the AARC web site becomes a heavily-trafficked site and needs upgrading for that reason.

Summary

This proposal has contained some indication of scope and cost, but is obviously not a fully fleshed out plan. That presumably would be the next step. Note also that there is no need to "do the entire collection" right from the get-go. Particularly if important document sets can be identified, a "proof of concept" project involving a few percent of the documents might be considered. There are certainly some fixed costs involved, but much of the costs here involve scanning time and are thus "per page" costs.

Such a project as delineated in this proposal could be an historic milestone, truly delivering to the public a documentary window onto a troubling time in American history. With today's technology, it is well within reach, and would provide access to virtually anyone. Once such an electronic database exists, there are a great many possibilities for turning it into a serious research tool as well as an educational forum. Such a document database, supplemented by the analyses and presentations of researchers, could be a framework for a new way of looking into the past.