Failing gracefully - Unisys datacentre outage

A decision was made not to switch customers to disaster recovery

Just how badly Auckland’s power outage last month affected data processing may never be known but it took out Unisys’s datacentre at Penrose for at least an hour at a time when the centre was in the middle of a major upgrade. A decision was made not to switch customers to disaster recovery after discussions between their CEOs and Unisys managing director Brett Hodgson.

Unisys was running a standby generator from a third party because it was in the middle of commissioning a new, $700,000 generator that will boost the datacentre’s capacity from 800 kilowatts to 1,600 kilowatts. A decision had been made a year ago to double the size of the datacentre’s processing capacity.

Hodgson says a decision was made early to refuel the standby generator because of past experience of demands on diesel during power cuts. During the refuelling operation, the fuel became contaminated with water, taking the generator down for an hour.

“We don’t know what happened,” he says. “We’ve called for two reports but we may never know.

“A number of customers were already out[because of the power cut]. We took people down in a controlled fashion.

“A number of CEOs spoke to me about enacting disaster recovery but we decided not to because we thought it would be quicker to get the situation stabilised.”

That said, the incident points out how essential it is to have disaster recovery, he says. “We haven’t had an outage in nine years and we don’t expect to go down again in the next ten years.”

Unisys will keep the 800 kilowatt environment to give it redundant capacity of up to 2,400 kilowatts.

Hodgson says it was a big learning incident for Unisys, particularly in how to manage communications in the event of such a failure. Telecommunications were affected badly and there was overload demand on cellphones. “We’ve tightened that up.”

He says a number of Auckland datacentres had issues, either directly or through third parties.

He emphasises that Unisys’ fuel tank was not contaminated. “We haven’t breached any service level agreements as far as we are aware.”

Unisys has around 20 customers using its Penrose datacentre.