States struggling to deal with digital documents

Most state governments are not actively tackling the creeping problem of digital archives and long-term access to public documents, according to the head of an industry group.

Apart from a handful of cases, states have not devised comprehensive strategies for retaining "born digital" documents, said Doug Robinson, the executive director of the National Association of State CIOs (NASCIO). Such documents are created in electronic format and do not exist on paper.

"There are very, very few states that have enacted any legislation or directive that addresses the permanent access to records," Robinson said. "The challenge is that states are all over the map in what format they use in archiving born-digital content."

The state of Minnesota introduced a bill last month that would mandate the use of "open data formats" in state agencies by having them use standards-based products. By avoiding proprietary products and formats, the proposal's backers hope to ensure access to state information.

The bill also spells out criteria for what qualifies as a "standard" and proposes responsibilities for different IT-related state offices.

Minnesota's move toward long-term data access through standards follows the high-profile case of Massachusetts.

The office of the former chief information officer in Massachusetts caused waves across the industry when it said it had chosen the OpenDocument format among its standards for desktop applications--a format not supported by Microsoft Office. The state, which named a new CIO in January, is in the process of converting its systems in anticipation of a January, 2007 deadline.

There are a couple of reasons for the lack of state strategies, Robinson suggested. Most state IT executives are dealing with problems that require immediate attention, such as security or lowering costs by consolidating their servers, he said.

In addition, the jurisdiction among different agencies, both state and federal, is not always clear. In the case of Massachusetts, for example, the CIO's office ability to set technical standards has been challenged by legislators and the office of public records.

Yet states continue to generate millions of digital documents, as well as multimedia content such as video of "State of the state" speeches, Robinson said.

"Very few states have really adequately addressed this, although all these states recognize that there's a business problem that everyone needs to address," he said.

Andrew Updegrove, the attorney for standards body OASIS and a proponent of the OpenDocument format, said he wasn't aware of other attempts to legislate the use of standards.

But he said that having two states explicitly adopt standards policies for documents could encourage other states to follow, even if those efforts are low-profile.

"I wouldn't be surprised if it was a grass-roots kind of thing, like the Minnesota bill appears to have been," Updegrove said.

[mcepat]

open formats of documents

What about the databases that will store these documents? Will they be open too?

open office docs stored in oracle database

seems very open ):

[markdoiron]

Two Problems for Everyone

There are two problems for everyone who stores valuable data as 0s and 1s (yes, the states, businesses, fed's, you, your mom and your kids).

1. Digital formats. It's silly to use anything that isn't a "defacto" standard. .jpg works. .mp3 works. Because of the availability of filters, .doc works (except for that temporary silliness with word 95 a while back), though I'd say "here-here!" to MS opening this up to everyone to use.

2. Media. There will be more loss of valuable 0s and 1s because of media incompatibilties and/or inabilities to retrieve from archaic medias than because of digital formats. Someone will always be bright enough to write filter software for archaic file formats. But no one will make hardware to support archaic storage media. This was a problem in analog days, remains so today, and will continue to be so in the future.

mark d.

[kquickkquick]

Solved

Hardware Emulation is the key to solving the hardware obsolescence problem. Hardware emulation is easy and very effective as is evidenced by orgs like the Amiga users group.
A technology for storing binary files on microfilm that was recently developed and patented by ACS, Inc. solves the media obsolescense problem for 500+ years.
This has been solved.