Businesses are being forced to ask hard questions about the current and future value of the data they hold and will need help from record-keeping experts about what to keep and what to discard -- as well as how to make it accessible and useful to the business.
That was the message from David Moldrich, knowledge systems manager at Australian brewer Fosters, and others at an international gathering of record-keeping experts in Wellington last week.
Organisations confronting a growing mountain of data will no longer be able to take the easy option of keeping it all because Moore's Law is "no longer in play" and storage will not continue getting cheaper, says Canadian recordkeeping specialist Brian Thurgood."
Moldrich and Thurgood, together with Hans Hofman, senior advisor on "digital longevity" at the Dutch national archive, were part of an international gathering of record-keeping experts in Wellington last week to discuss and push forward international standards. The International Standards Organisation (ISO) was a sponsor of the conference, hosted by Archives NZ.
Standards are evolving slowly in the international recordkeeping community. Yet new technologies keep coming.
"We are used to being at the back of the wave; now we have to sit on top of it," says Hofman.
The record-keeper needs assurance that what is valuable in the long term will still be accessible in the future and can be appropriately migrated and repurposed to deal with new business needs and styles and new recording and access technologies.
The world of reliable recordkeeping is battered by a storm of change, say the experts. Organisations are changing, there is increasing connectivity allowing more collaborative ways of working through the web, says Hofman.
"That has an impact on the way information is created, used and managed."
Useful records are being created outside the realm of structured data by blogs and wikis and opened to access by social bookmarking, tagging and syndication, says Thurgood. Record-keeping specialists have to become involved in the way this information is distributed and reused.
For the user, syndication is about bringing together information from a number of sources, but for the provider it's a matter of regulating and administering outward transmission.
"Most organisations are not prepared for the task of sending an ongoing information stream out to their public," Thurgood says.
A growing number of users are also going mobile, and this presents its own problems of maintaining consistent and up-to-date versions of records available to each appliance.
Then, of course, there is the problem of data longevity, which is dependent on three factors: understanding the format in which the data and associated metadata are stored, keeping it on a durable storage medium and having the hardware and software to read it.
Technology vendors have more understanding of the problem today and are veering towards sharable standards, but "no-one has put their hand in their pocket and said 'we will make sure that this particular disk drive will be available for the next hundred years'," says Moldrich.
Open source isn't the answer. An open standard is what really matters; so you know what you've got and it's properly documented to give you a starting point for accessing the data.