How to Archive Files Long Term: Protecting Data for Decades

Two Different Problems: Readable Bits, Usable Files

Hard drives, SSDs and so on do not guarantee that the bits are still all in place after years, even in a climate-controlled space. For non-moving parts, error detection when reading is standard, but writing requires an extra level of checking. The National Archives recommend buying media that is robust, including integrity checks for both reading and writing, with specifications that don't obfuscate the bit checking techniques.

To address file corruption or accidental alteration, every file in a long-term archive needs a checksum taken as a reference value, which is then recomputed for each copy on a regular basis. If the checksums do not match, a corresponding uncorrupted copy is available as a replacement, as checksum tools like the Digital Preservation Coalition endorse. The DPC guidance states that checksums are utilized to determine the integrity of digital material to make sure they are correctly stored. Should there be any discovery of corruption, one of the other copies should be used to make a fresh replacement. These checksums and fixity tools are just one part of a long-term archive strategy.

To tackle the problem of format obsolescence, the Smithsonian Institution recommends of saving both an open format and the original for preservation. [TO VERIFY: evidence of software lock-in causing file incompatibility more than three decades later, for example in legacy financial reconciliation] Even if a format like Adobe PDF becomes a one-off software product, it will not become a parity format like binary TXT that all platforms will reliably read. Commit to making both a native copy and an open copy, and then to checking both for compliance, on a schedule. While storing alongside obscure data types, metadata is just as important in the event that a format becomes unreadable and retrieves the relevant information for file restoration.

The Only-Copy Trap

Waiting to preserve a file until after you need to restore it is a common cause of archive loss.

The University of Michigan Library's best practice is making three copies, using two different storage media, and keeping one offsite. Their guidance states that regular storage is paramount. To do this, the university recommends that institutions check their digital storage media annually, when they can also check that files are still accessible.

Randomly open a few files every year to ensure the archive is still usable, not just stored. Corruption, or loss, in one copy can be fixed by using another to make a new one, as an archive requires multiple copies of each file.

Why Checksums and Replacement Copies Matter

The DPC makes the case that misuse of checksums, such as skipping regular checks, is a leading cause of unnoticed file corruption. The mistake of counting on a checksum taken at the time of ingest, and assuming those bits have not changed, is a risky strategy.

Set up a timeline for computing the checksum anew for each copy, and have the old checksum in your Materials Management workbook somewhere. If bit errors have crept in, the checksums won't match, and you have a clean copy from your other systems to replace it. Store information about each copy in a monitoring system that tracks checksum matches, not just a "goal reached" message.

[TO VERIFY: a documented quote from an archive regulator or published court case criticizing checksum methods that do not assume bit corruption will happen, no matter how excellent a file system or storage system is.]

Why Restore Testing is Not Optional

The only way to know an archive is accessible is to open it. Beyond merely noting a checksum match, random opening and use of the archive proves that the data is not just there, but still operable.

While the University of Michigan mentions annual storage checks, there is an important difference between media integrity and the ability to open files on a modern platform. Guidebooks like the archives page of the University of Michigan most certainly cover opening files for testing, but the next though is, "And then what?" Rather than saving an archive of copies and losing it to format obsolescence, introduce accessible media and open formats into an active archive process. Staff creating the archive should consider what each file type is for, such as a PDF reader that outputs to TIFF for text search, and

Names, Metadata, and Finding the File Later

Every file should have a clear name, since filenames that are too long or use special characters can break. A naming convention written out in an organization's Materials Management tool helps collection staff name the file consistently. Folder structure should be logical. The various accepted methods for organizing archives are well documented, but a major factor in whether an archive is usable or not is the way it is organized. As Recollection Wisconsin notes, maintaining organized folders, labeling with consistent naming, and storing metadata alongside the data itself are essential practices for digital archival storage.

Which naming style is right for your collection depends on what files and how they are used. Having it written down in a management tool for future staff to follow is one thing, but more important than even a good naming strategy is having a full Materials Management plan for each file type. The Document Naming Tools wiki has examples, but the key is having a plan that is documented in a management tool, not just a document.

What to Do with Proprietary Formats

If a small organiations operates in one proprietary file format, the best way to safeguard its files against vendor lock-in is to have two copies: a native copy and an open format copy. This is a low cost way of preserving access to information, and universities like the Smithsonian, Recollection Wisconsin, and others use it to help prevent their files from being locked to a single piece of software.

A big organization or software publisher might not be able to take this approach, but it is absolutely necessary. Spreadsheets are locked in at different levels. Without an absolutely open dataset format, and shredding every one-off and extension that adds to the load, archiving long-term is impossible. The archives at places like Recollection Wisconsin are organized to help withstand software changes through strict attention to naming conventions, metadata, and opening formats.

[TO VERIFY: a primary source organization that has sanctioned a change in its admissions or rebate policies tied to keeping archives in an open format standard, like a well known publisher moving to ISO and sharding away the one-off and incompatible content files.]