No one likes to go through a disaster recovery exercise, but the experience of New York-based financial institution Lehman Brothers’ data centre was worse than most. The company suffered the effects of the disaster that has gone down in history like no other in modern times – the terrorist attack of September 11, 2001.
Lehman Brothers had an office in the World Trade Center, and one employee there died in the building’s collapse. Lehman’s headquarters and main data centre were in the adjacent World Financial Center and the data facility was totally destroyed.
However, with the help of software from EMC and Citrix, the backup data centre in Jersey City, New Jersey, picked up the load. The database was replicated and the main client portal, the chief entry point for clients and employees, was up in 30 minutes.
Benjamin Wu, a member of the Lehman Brothers portal architecture team, visited Auckland and Wellington last week to relate the company’s experience of the recovery and to pass on some lessons.
“Why did it take us 30 minutes to come back up and not one minute? That was the human factor; we wanted to make sure our people were safe first”, says Wu.
He had a narrow escape himself, being on a train from his base in Jersey City to visit the WTC office when the first hijacked airliner hit.
Lehman Brothers was better prepared than many companies, but still drew some lessons from the incident. “Whatever disaster recovery plans you have, make sure they are self-running,” Wu says.
In the immediate aftermath of the attack the area was under martial law and no one was allowed in. Companies which depended on staff to be physically present to operate disaster recovery suffered worse than those whose procedures were automatic.
The disaster led to a higher awareness of the need for business continuity planning and execution, and although business continuity plans worked smoothly with the group then assigned to it, Lehman Brothers has now appointed a single head of business continuity, former e-commerce head Bridget O’Connor.
“Another big lesson is to collect details on all your employees, including where to contact their families, and keep them up-to-date,” Wu says. The Lehman portal now regularly “nags” users every 45 days to re-read their personal details and make any necessary changes.
The key employees necessary to a business continuity plan should be identified, so the plan proceeds smoothly, he says.
The Citrix Nfuse portal at Lehman Brothers was able to put the basic trading and office application out over the web after September 11, so employees could continue to work from their homes and other sites. This was done through Tocket, a custom Lehman application implementing a private “tunnel” with SSL over the internet. Tocket had reached prototyping stage with a group of pilot users just before the disaster.
The application is similar to Citrix’s own since-released Secure Gateway, and made contributions to its development, Wu says.
The seminars in Auckland and Wellington attracted between 250 and 300 people, demonstrating an awareness of the need for disaster recovery and business continuity procedures which has been heightened, even in New Zealand, since September 11.
These factors are “always in the top five priorities for IT managers, but they never quite make it to number one”, says EMC business continuity marketing manager Michael Cunningham, who accompanied Wu on the tour.
Other staff from Lehman Brothers addressed earlier seminars in six Australian centres with EMC and Citrix personnel.