“How will you get your data back after the "insert catastrophe here” was the title of a Gartner white paper in February 2001. Seven months later, the September 11 terrorist attacks filled in that blank, stunning the US and forcing businesses to consider disaster recovery on a previously unimagined scale.By 9am that morning, as hundreds of thousands of workers passed through lower Manhattan on their way to their jobs, smoke was billowing over the city from the first hijacked plane’s crash into the World Trade Center north tower. Within hours, the entire complex was in ruins, throwing Manhattan’s transportation and communications systems into turmoil and the nation into shock over the vicious attacks on New York and Washington.
“There was a sudden awareness: we could have not just a building, but a regional catastrophe,” says analyst Dianne McAdam, with market researcher Illuminata in New Hampshire.
“What happened with September 11 in Manhattan, it was a loss of the phone lines, the data lines, the transportation, of parts of the entire communications infrastructure.”
Even a building-wide catastrophe was more than CareerEngine Network was prepared to handle.
The small, three-year-old web developer was located on the 21st floor of the World Trade Center’s south tower, which was hit by a hijacked plane soon after the crash into the north tower. Several of the company’s dozen employees would normally have been at work by the time the attacks began, but a series of minor emergencies and delays — a sick child, a broken train, an apartment problem — conspired to keep the staff out of the office and unscathed. CareerEngine’s infrastructure, however, was demolished.
“We lost all our information about our customers. In some cases, we didn’t even know who owed us money and who didn’t. We had to contact all of our clients and have them work with us to recreate our records,” says George Benoit, the company’s chairman and chief executive.
CareerEngine creates customised career-information and job-listings websites for web publishers. Those websites, hosted remotely, remained live without interruption. But the company’s internal data was wiped out.
“We had been making backups. And then we left them in the same place, in a different area,” Benoit says. “We figured maybe there would be a fire, something like that we’d have to deal with. We never thought the building would fall down.”
The dramatic events of last September 11 galvanised some planners into action in a way natural, more routine calamities like earthquakes and storms have not. US companies are starting to take disaster recovery as seriously as their European counterparts, says Ken Boyd, director of advanced disaster recovery solutions at IBM.
The 1993 trade centre bombing, coupled with last year’s attacks, created an environment here that resembles what European companies have faced for years from terrorist threats, he says.
“Before September 11, we had a few disaster recovery planning calls a week,” he says. “Now we are getting a few calls a day.”
That has also been the case at data replication vendor NSI Software in New Jersey. “The visibility of data protection and availability has been raised to an all-time high,” says Bob Guilbert, head of marketing. NSI has been expanding the staff of a services unit it launched in October, an offering that quickly attracted several dozen customers seeking help constructing a business continuity plan.
Businesses are increasingly considering the possibility of a disaster that paralyses a large geographic area, say analysts and vendors.
“One of the lessons that customers learned from September 11 in one sense is that it was a regional disaster — not just a floor or a building, but all of the infrastructure around the World Trade Center,” Boyd says.
The scope of the disaster undermined some companies’ disaster recovery plans, as workers could not reach buildings to handle manual tasks and organising phone calls could not be made. Consequently, IBM now stresses to its customers the importance of automating data backup, Boyd says.
“You have to ask how automatic is your disaster recovery,” he says. “How much are you depending on people to make decisions? Because you may not have a scenario that lets that happen.”
Since the attacks, IBM has set up a technical design council that examines the disaster recovery proposals the company makes to its clients, checking that they will work in real-world conditions. IBM engineers examine the bandwidth connecting primary and recovery sites, for example, and evaluate hardware products to ensure those will work together as well in practice as they do in theory.
Customers are also extending the distances between data centres, realising that a backup in New Jersey for a network in Manhattan could be vulnerable, Illuminata’s McAdam says.
In the year since the attacks, customers in the New York metropolitan area have been adding “heavy lifting” data centres not in New York, New Jersey or Connecticut as an extra safeguard, says David Palermo, marketing head at Pennsylvania company SunGard Data Systems, one of the industry’s major providers of disaster recovery and business continuity protection services.
The severity of September 11 prompted some businesses to escalate plans to implement or improve their protection plans, but as is always the case, action lagged behind talk, Palermo says.
“It’s the same thing as with Hurricane Floyd, with earthquakes, with any other disaster. It has the ability to make people say, ‘Wow, we’ve been thinking about this forever.’ But after any of those events, many more businesses think about it than actually end up doing something about it,” he says.
For large enterprises, particularly those in heavily regulated industries, such as the financial services firms that fill lower Manhattan, fairly extensive emergency planning is a routine part and cost of doing business. Small and medium-sized businesses are less likely to have plans in place, but even those that do face daunting obstacles in recovering from a catastrophe. Gartner estimates that two out of five companies that experiences a disaster go out of business within five years.
Having a complete plan for both data backup and an off-site emergency work location for its employees allowed the New York Shipping Association (NYSA) to be fully operational two days after the September 11 attacks destroyed its World Trade Center office. The 70-year-old organisation, which co-ordinates cargo operations in the New York and New Jersey ports, drafted detailed plans after scrambling to recover from the 1993 World Trade Center bombing, says NYSA president Frank McDonough.
A SunGard customer, NYSA evacuated all of its employees safely on September 11 and relocated just under 100 of them to several temporary SunGard facilities in the region. Within the day, its staff had fully wired temporary workspaces, and data retrieval efforts were under way.
“I don’t think there’s a darn thing that we would have done differently — there’s no second guessing,” McDonough says.