On the morning of Tuesday, October 30, 2001, as far as Telecom New Zealand was concerned, Wellington was struck by a severe earthquake.
Buildings around Jervois Quay were damaged, it was difficult for people to move around the city, two major telephone exchanges were taken out and tremors continued to shake the windy city.
Telecom worked from its crisis response centre in Courtney Place as if there were no phones working there. More than 100 staff from its networks, investor, government and media relations divisions took part, plus suppliers such as EDS and those who provide radio communications.
There was role-playing, too. A staff member acted as Communications Minister Paul Swain, who had to be assured that the telecommunications systems would soon be up and running.
Telecom spokesman Andrew Bristol played himself advising a fictional media on what was going on.
“You really do learn when you are put into that situation,” he says. “The network staff had to get the network running. Could they rob Peter to pay Paul, and get stuff down from Hamilton?”
Telecom believes that if a major earthquake hit the capital, “it is highly unlikely that the network would be totally affected”. The network control centre in Hamilton would, if necessary, send down mobile exchanges to augment those already in place in Wellington, Bristol says.
That afternoon in October Telecom pretended it was three days later and that there were still problems. Staff were reminded with the help of a video that an earthquake’s effects on a city would last weeks rather than days.
Bristol says Telecom NZ faces many recovery issues. This month, services were hit by contractors in Northland cutting a fibre-optic cable. And on September 11 (New York time), Bristol says he was woken up at 4am so he could tell New Zealand that because Telecom had phone links through the World Trade Centre, the line to London was “compromised”. But, he says, within hours the traffic was re-routed and normal services resumed.
Citing “commercial sensitivity” from preventing him revealing much detail, Bristol assures customers that the telco has everything in order should an emergency arise. Data is backed up daily, Telecom uses offsite storage centres, follows “good procedures”, has a committed disaster recovery team, knows who to contact and what to tell them to do, such as work from home.
“Telecom is in a position to do whatever is required as soon as possible,” he says. “Each business unit is required to have effective management plans and test them in real time.”
Telecom’s ISP, Xtra, says it is equally prepared, with risk management strategies and disaster recovery procedures that are reviewed twice a year.
Xtra CIO Shane Ohlin says these were tested greatly in the 1998 Auckland power blackout when all its 160 central-city staff were relocated, mostly to work from home. The power crisis did not affect other ISPs as much as most were based out of the central business district, but Xtra customers were not affected, he says.
Ohlin says Xtra’s policies cover what is needed by the business, what third parties should do, how it declares a disaster, what its recovery process is and how to meet a timeline of necessary actions. This includes technical issues around data storage, retrieval processes, platform and network recovery plans and a disaster recovery and test plan.
Xtra says as the largest portal in New Zealand, its plans cover cyber-terrorism and include a “comprehensive set of protection”.
“We have a classic approach — a multitiered architecture of firewalls and intrusion detectors that you would expect in a large company, plus a dedicated security team of four, plus a manager,” he says. Xtra uses Telecom’s own data storage centre facilities, with its system split between two separate buildings. “We make full use of backup, managed through EDS. If we had to, we could also move to a separate building,” says Ohlin.
In the event of disaster, staff can easily be contacted as key staff carry pagers or cellphones. A system of delegated authority means staff can usually make decisions without contacting their bosses, he says. Staff can work from home, with the ISP providing facilities such as web access. “The point is, we focus on what are the key priorities, protecting against the major types of events and what is common and likely,” says Ohlin.
Xtra mainly uses Sun hardware, EMC for storage area networks, Veritas files and volume manager, Oracle databases and Storagetek backup silos.
To date Xtra has only suffered a few “systems” issues, with only the 1998 power blackout activating its disaster recovery plan, Ohlin says.
Taking back control
The Auckland blackout also saw staff relocation at Auckland Regional Council. About 100 went to work out of Ericsson Stadium and 150 worked from home as the council proceeded through its annual planning cycle.
Some four or five years ago, says ARC CIO Tony Darby, a council financial system was cared for in an outsourcing arrangement but the outsourcer did not back up the information properly, so the council lost six days of financial transaction processing. The outsourcer had to pay for the back-data capture and the system to be operational again. Six months later the work was brought in-house.
“Now we are responsible for our own mistakes,” he says.
Darby says recovery systems in place are being reviewed at the moment. There are currently three levels of recovery procedure. It checks weekly for normal file recovery, once or twice a year for recovering data from databases offsite, and annually for out-of-action systems and buildings.
Every day the ARC puts its backup tapes in a fireproof safe and each week they go to an undisclosed offsite storage space. The council’s 30 internal servers are also stored on special plates to protect them in an earthquake. Darby says the council uses Compaq servers, Toshiba notebooks and other “mainstream” brands so equipment can easily be replaced. This also helps should relocation be necessary.
Key employees can be traced — generally cellphone and phone details are programmed on to phones — and phone trees and other communication strategies will ensure staff are told. Vendor details are held on hard copy and electronic format, Darby says. Staff have autonomy to make decisions and working from home is also possible. Many staff do so already, so the ARC has developed web-based front ends and things like access to financial systems, Darby says.
The council could survive for a short time using manual systems, he says, and as long as the public were told of events he believes perceptions of the council would not change.
The Auckland blackout also tested online travel agent Travel.co.nz, who held a similar “dry run” last year, with staff working from home.
Chief executive Greg Southcombe is confident the business could cope easily in a crisis because the business is largely “virtual”, with many functions taking place away from its Emily Place, Auckland, headquarters. “Our website is hosted in Brisbane; we are protected from physical attack. There is nothing hosted in the building,” he says.
In addition, some 80 of its 130 staff already work from home; Southcombe is based in Hawke’s Bay.
Southcombe says his firm has an alternative site that could be moved into quickly. Backup tapes are removed from the premises daily and this backup and storage work is outsourced. The only possible threat to company systems, he says, would be internal — a problem for any business.
Staff can be tracked down through mobile phones, he says, and they would have autonomy to make decisions.
Even if the headquarters were knocked out entirely, “there would not be much change” as work would take place from home as largely happens already.
“We are like the internet. Our infrastructure is supported all over the place. There is not single area or place that could be knocked out that would then knock out the business. There would need to be four or five knockouts. That’s an invaluable strategy,” Southcombe says.
Bank on it
Telecom and IT services company EDS handles interbank financial transaction within New Zealand.
Last September and October EDS suffered power outages and backup failures, hitting Eftpos, ATM services and overnight banking settlement processing. EDS was criticised for having insufficient procedures and later blamed problems occurring in the data management centre during maintenance work. It was such a rare event, EDS said, that it was unlikely the company change procedures following it.
Spokeswoman Helen Morgan-Banda says EDS naturally has disaster recovery policies but cites commercial confidentiality in not revealing them. EDS protection systems are “robust”, she says, exceeding the committed minimum levels agreed with customers.
Business premises can be changed, she says, and this is tested regularly. Staff can easily be contacted by mobile phones and systems are in place to ensure decisions are made. EDS also has the technology for people to work from home. Being a multinational business, Morgan-Banda adds, means other EDS centres could step in until the damaged centre was restored.