Waikato District Health Board expects to recover all but a small percentage of the data it lost because of a disk failure in late October.
The Hamilton-based organisation is also reviewing its IT operations and already has begun to make changes, says CIO Alan Grainer. The failure affected countless emails and personal work files of its 690 staff, including doctors, but the board says no patient records were affected.
Waikato DHB has spent at least $60,000 trying to retrieve the “unstructured” information and has hired experts from the United States, who are expected to complete their work this month.
Jan Adams, the DHB’s general manager, is leading an incident management team of IT staff and the DHB is also conducting a Serious Event Review as to how the failure occurred.
Ontrack, the American computer forensic specialists the DHB has employed to recover the data lost from the file and print storage area network (SAN), managed to recover 80% of the lost data within 24 hours.
“They have analysed the tapes and realised they need to write a special program to get the information out. They are working with one of their engineers in the UK. The two tapes that were sent to Auckland have now also been sent to the US,” Adams said in an internal staff memo dated November 19.
Alan Grainer says other data has since been restored and around two-thirds of what remains lost should eventually be recovered, leaving a small amount of data lost forever.
The failure occurred on the DHB’s existing Novell Netware applications running on IBM hardware, which were already due for replacement.
Grainer says the DHB had a “well established” back-up regime using Legato. This stores seven weeks of tape cycles, which are then sent offsite to secure storage.
“There was a problem with that. We had a problem with some of the writing to tape,” he says.
The DHB, Grainer says, is doing a “full route-cause analysis” and while he didn’t want to pre-empt its findings, there were “a line-up of issues, no defining issue” that caused the failure.
Reports are due for completion by Christmas.
Grainer says overall the DHB stores terabytes of data, and the failure was at the gigabyte level. The board has learnt it must be more rigorous about how and where it stores its data, he says. The board will be looking at its point of failure in any business case for replacement systems, Grainer says.