A data centre rival has little sympathy for EDS’s internal “power outage” and backup failure that interrupted the overnight settlement processes of the country’s banks last week.
EDS, which handles interbank financial transactions, says the power was out for half an hour last Wednesday night after its electricity supply system and backup failed, delaying processing until the following morning. ANZ took 24 hours longer than the country’s other trading banks to recover from the problem and is reviewing the manual process responsible for the delay.
The EDS disruption stemmed from a power failure at the Mt Wellington data centre of computer services company EDS, which handles interbank transaction processing. The 30-minute outage affected payments for customers of all the trading banks on Thursday.
Unisys data centre manager Shane Gaskin is critical of the situation that apparently occurred at EDS. “From the power point of view, reliability should not be too difficult to maintain.”
If a company has functioning back-up generators that can run long enough to tide the data centre over any interruption, and a means of “holding the power” — an uninterruptible power supply, and correct configuration to supply appropriate power to all components — he would not expect power to ever be interrupted to a large data centre. “All these precautions are pretty standard these days,” he says.
“Having said that, if everything fails — your main supply and coincidentally your backup at the same time — you may just not be able to maintain continuity.”
In the relatively short time Unisys has been running data centres in New Zealand such a total outage has not happened, he says.
However, Steve Matheson, technical director at payroll system specialist Datacom, says a large data centre is “still a very challenging environment” from the point of view of continuity of power supply to all components.
“There are still single points of failure, and as availability requirements increase it becomes more difficult to satisfy them.” Some failures are still “extremely difficult” to anticipate and to trace when they happen, he says.
“It’s an electrical engineering issue; a cable may get too hot, a switch may melt. Whatever you do, once in a while these things happen.”
EDS, meanwhile, is scrutinising its systems after the outage.
The company says systems were progressively restored overnight, with all interbank transactions completed by 7.30am on Thursday.
EDS spokeswoman Helen Morgan-Banda describes the event as “very rare”, never having previously occurred in her two years at the company.
EDS has an alternative power supply and is investigating why this back-up system did not swing into operation when banking operations ceased at 9.07pm.
“We are doing a root cause analysis in conjunction with our clients,” says Morgan-Banda.
She says power was restored by 9.41pm and processing systems were then progressively restarted.
Most banks had caught up with a backlog of payments resulting from the computer glitch by midday on Thursday, but ANZ only did so on Friday.
“We’re trying to ascertain what the processing problem was,” says ANZ marketing head Mark Thomas.
Thomas says the bank thought it would take until midday Thursday to complete manual checking of payment data but the process ended up taking longer "perhaps because we have more customers".
The bank says it will waive fees and interest charges incurred by thousands of customers affected by the problem.
Thomas says a “standard operating process review” takes place after such occurrences.
“No one is terribly happy about it but it was a situation out of our control.”
ASB’s chief manager of electronic banking, Matthew Bartlett, says the EDS failure caused “some minor delay” in processing some transactions between it and other banks, but systems returned to normal by 10am.
BNZ spokewoman Sue Mills also reported “processing problems” but says online banking and ATMs were unaffected.
Major banking customers appear to have been largely unaffected by EDS’s delay.
A spokesman for the Pacific Retail Group was initially unaware of the breakdown, but later confirmed the group, which owns Noel Leeming, Bond & Bond and
Computer City, had no direct credits going out on or automatic payments made on the 12th.
Barry Hellberg of the Retail Merchants Association knew of no trader contacting his organisation about the outage. “If it was a matter of life or death, someone would have rung us,” he says.
NZ Bankers Association chief executive Errol Lizamore says banks and organisations have back-up systems in place, otherwise the problem would not have been remedied so soon.
Investigations will help avoid further failures, he says. Systems were to be fully updated by midday Thursday and systems operating normally Thursday night.