the mind-boggling challege of long-term digital preservation

1

The Mind-Bending Challenge of Long-Term Digital Preservation

Gordon E.J. Hoke, CRM, IGP

Manager, Electronic Records, Abbott

Chicago Chapter, 11 February 2014

2

The views and opinions in this presentation belong exclusively to Gordon E.J. Hoke and do not reflect or express views and opinions of Abbott or ARMA International.

Disclaimer

3

Records Relative Longevity

• Ephemeral information• Convenience copies• Drafts and project materials• Transaction records• Serial records• Records within a tech generation• Persistent records

4

Ray Kurzweil, inventor of OCR, Speech-to-Text, the digital piano

Information lasts only so long as someone cares about it.

The conclusion I’ve come to…after several decades of careful consideration, is that there is no set of hardware and software standards existing today, nor any likely to come along, that will provide any reasonable level of confidence that the stored information will still be accessible (without unreasonable levels of effort) decades from now.

5

No Availability? No Value!

• Fifth of the Generally Accepted Recordkeeping Principles®

• There are no time limitations to availability• If the retention schedule calls for 10, 25, or

50 years…• If the retention schedule calls for

“permanent”……and what does “permanent” mean?

6

Persistent Records include…

• Buried utility lines and easements– Re: PG&E inability to locate leaking gas lines

• Medical and dental records– Forensic identification– Cross-generational research– Childhood disease with adult complications

• Insurance policies, deeds, investments, wills

7

Persistent Records include…• Research http://www.nobelprize.org/nobel_prizes/medicine/laureates/2012/

– The Nobel Prize in Medicine for 2012

– Sir John B. Gurden, research in 1962 > Shinya Yanamaka, b. 1962

• Art (incl. inventories), history• Student transcripts• Nuclear waste

– Spent nuclear fuel rods: half-life = 10,000 years

8

Challenges to Record’s Useful Life -- Storage

• Media longevity:– Magnetic Tape

• Gets brittle over time• Adhesive holding magnetic particles to tape fails• Is susceptible to magnetic fields

9

Challenges to Record’s Useful Life • Storage media longevity (cont.):

– Optical disks (CDs, DVDs, Blu-Ray, WORM platters)

• Format obsolescence• Substrate degradation• Separation of laminates• Bio-degradation of organic dyes• LG 2013 DVD disk “guaranteed for 1000 years”• JVC “ISO compliant” DVD disk, “up to 25 years”• “Pancake” Drive

10

Challenges to Record’s Useful Life #1

• Storage media longevity (cont.): – Solid state memory (including USB drives and

memory cards)• Failure of (plastic) housing• Small form factor increases loss and security risk• Interface compatibility• Cosmic rays• (There is promise in solid state external hard

drives )

11

Challenges to Record’s Useful Life #2

• Hardware lifespan– Hardware goes obsolete within a decade

• Audio cassettes (TRS-80 and Commodore 64)• Floppy drives?• ZIP Drives?• Sony’s 1991 12” WORM drive “guaranteed for 99

years”

12

Challenges to Record’s Useful Life #3

• Software Obsolescence– Operating systems– Applications, including mobile

• MS Office support has been about 10 years

– Access • Password management • Encryption keys

13

Challenges to Record’s Useful Life #3 (cont.)

• File formats – Clock starts ticking on release– Documents, Audio, Video– Medical– Engineering/Design

14

Challenges to Record’s Useful Life #4

• Governance– Migration/conversion, validation issues: “When you

move data, you lose data.”– Limitations of turning disks– Ensuring future

• Staff time• Budget• Expertise• Tools• Passwords• Executive support

15

“Technological obsolescence may not be the greatest problem! Organizational commitment, and the willingness to allocate sufficient resources may be an even bigger problem.”

David O. Stephens, CRM, and Roderick C. Wallace, CRM, in Electronic Records Retention: New Strategies for Data Life Cycle Management

16

Improvements through Behavior Management

• Policies, Processes, and Guidance• Education and training• Incentives – carrots and sticks• Installing limitations • Assigning responsibilities• Providing budget

17

Preservation Strategies #1

• Print to Paper– Acid-free paper– Inorganic toner/ink– Controlled storage conditions

• But what happens to the metadata, and how important is it…?

18

Preservation Strategies #2

• A computer museum– Keep those disks a-turning– Hardware/software in mothballs– Emulation: Make a new computer think like an

old computer

19

Preservation Strategies #3

• Migration/conversion– Appropriate in some situations– Consumes many resources– A strategy: identify core records and pour

resources into preserving those– Updating media is insufficient alone; whole

systems require regular renewal

20

Preservation Strategies #4

• Archival Formats– There are no guarantees of backward compatibility,

so balance the risk– Use what is most common: PDF, XML, JPEG, TIFF,

MPEG, FLV -- “standards are what people use”– Verity vs. file size: does every pixel require retention?– Format limitations: formulae, embedding, encryption,

digital signature

21

.

.

Preservation Strategies #5

Advantages

− Long-lived under controlled conditions, <500 years− Human readable with sunlight and lens

− May preserve metadata− Scanners can return images to digital

Microforms (film, fiche, et al)

22

Preservation Strategies #5

Microforms

Disadvantages− Color may be prohibitively expensive

− Chromatography for the most valuable images− Stereo-filters simulate color for less

− Inappropriate for audio and video− Spreadsheet/database use undeveloped

23

Craft a Unique Solution

• Evaluate– Volume of records – all records vs. core records– Range of retention periods– Available resources: budget, staff, technology– Acceptable level of risk– Prognostications

• Future resources• Viable formats• Organizational commitment

24

Craft a Unique Solution (cont.)

Which alternatives have the best chance to meet the retention schedule?

– What records can stay local and native?– What records lose value in relocation,

conversion, and/or migration?– What storage meets operational needs– When is dual storage worthwhile?– What is your risk appetite?

25

Craft a Unique Solution (concl.)

• Behavior management– Make compliance easy– Enforce policies

• Maintain availability• Encourage the development of archival

technologies• Extend record life until the next technology

26

Resources

• ARMA Book Store• Charles Dollar and Lori Ashley• National Archives and Records Administrat

ion (NARA)

• Society of American Archivists (SAA)• Library of Congress

27

Questions?

Gordon E.J. Hoke, CRM, IGP

ghoke@mindspring.com

1 (507) 254-6474

28

Finis