As Mardi Gras was celebrated in still-recovering New Orleans earlier this month, more than a dozen IT managers in Louisiana and Mississippi were rushing to build more resilient systems, improve their companies’ communications capabilities and apply the many lessons they learned from Hurricane Katrina.
There’s a particularly strong push to increase data backup capabilities, especially via online replication to other sites, and to set up wireless and satellite communications systems, the IT managers say. The work is being done with a sense of urgency because of worries about this year’s hurricane season, even while many employers struggle to find IT staffers to replace those who left the region and haven’t returned.
“I have a great concern for this coming hurricane season,” says James Mehaffey, manager of the business continuity programme at Blue Cross and Blue Shield of Louisiana, a healthcare provider. “In the Gulf Coast our infrastructure is severely weakened.”
Many of the people leading the efforts to reinforce IT systems and networks have already been through a lot, both professionally and personally.
For instance, the home of David Scripter, regional IT manager of URS’ New Orleans operations, was located near a breached canal in the city. “I lost my house and pretty much everything I own,” he says, adding that at least he, his wife and their three young children were all safe.
From a business standpoint, the engineering services firm’s offices were inaccessible for about a month after Katrina struck, causing devastating flooding the following day. Because systems were out of reach, URS had to buy new hardware, especially PCs for its end-users, Scripter says. The company set up remote offices in Baton Rouge, Louisiana, as well as in Houston and Austin, Texas, and it took the firm two weeks to restore its IT operations after the hurricane.
Scripter is now back in New Orleans and working to make sure the company can recover from another catastrophic storm within five days. He says his staff have installed a SAN (storage-area network) in New Orleans and are investigating how to mirror the data on the SAN to an off-site location in another state. URS is also installing a system that will replicate separate Oracle databases used in 3D engineering applications to remote locations. “We’re trying to get a lot of this stuff in place by June 1.”
Larry Mayo, vice president of IT at Keesler Federal Credit Union in Biloxi, Mississippi, is working to the same deadline.
Keesler’s headquarters building lost its roof and most of its windows to Katrina, but the credit union’s datacentre was spared because it was located between two cement floors in the middle of the building, Mayo says. A generator worked throughout the seven days that the company was without electricity and Mayo took steps to activate Keesler’s disaster recovery site in Arizona.
However, Keesler decided not to go live with the DR site because of the widespread failure of voice and data networks in the Gulf Coast region. “Without the communications leg of it, no matter which datacentre we worked out of, we were going to be dead in the water,” he says.
As a result, the credit union is installing satellite communications links as network backups at its 12 locations in the US and three offices in the UK, he says. The installations have been completed at three facilities and the remainder are due to be finished by June 1. By then, Keesler also expects to have deployed a converged voice and data network that will replace an aging Cisco data backbone and a PBX phone system. The converged network, which was expedited after Katrina, will be able to work over the satellite links, Mayo says.
Even at a company like Blue Cross and Blue Shield of Louisiana, which had invested heavily in business continuity capabilities, Katrina managed to expose issues. For instance, the health insurer used diverse routing techniques on its networks but wasn’t aware that at one point they all went to a common physical site in New Orleans, Mehaffey says.
At the University of New Orleans, IT workers weren’t able to begin returning to the campus until mid-November, says Jim Burgard, assistant vice chancellor for university computing and communications. And it wasn’t until the start of January that all of the employees who remained on the IT staff after the storm were back on campus.
Even now, the university is still running its mission-critical applications at Louisiana State University (LSU) in Baton Rouge on 25 servers — some taken from its own datacentre and some donated by LSU. But the New Orleans school is installing an uninterruptible power supply and a new generator powered by natural gas at its own facilities, Burgard says. When that work is completed in about six to eight weeks, the applications at LSU will be shifted back to the campus in New Orleans.
Even so, the university plans to leave the servers in place in Baton Rouge and mirror data to them so that the LSU location can function as a recovery site if another disaster occurs. That strategy is also geared toward eliminating some of the problems the university had retrieving its backup tapes after Katrina, he says. Although the school didn’t lose any data, it had to wait three weeks to access the tapes, which were stored on the sixth floor of a building in downtown New Orleans.
Meanwhile, Burgard is still trying to replace eight IT workers — nearly a fifth of his 45-person staff. A recent advertisement for a senior networking position that before the storm would likely have garnered about 20 CVs netted just four, he says. In addition, he received only two CVs for a senior database administrator job.
Jan Rideout, CIO at defence contractor Northrop Grumman’s Ship Systems unit, lost her Mississippi home to Katrina. She said that if there’s a bright side to the storm for her it’s that the disaster prompted the Ship Systems operation — which has facilities in New Orleans — to accelerate a plan to replace networking cables with wireless technology.
Disaster recovery plans have to be flexible and must take an organisation’s ability to respond into account, Rideout says. For instance, many plans assume that employees will be available after a disaster occurs, she says. However, when the air conditioners in Northrop Grumman’s datacentre stopped working after Katrina, there weren’t any employees on hand who knew how to gracefully shut down the servers, she says.
That points to the challenges that Rideout and other Gulf Coast IT managers are facing as they scramble to shore up their systems, networks and data protection mechanisms so their companies can withstand another major storm.
“There is not ever going to be the perfect disaster recovery plan,” Rideout says. “You can’t write the perfect script and the next time will be different. And I don’t think we can forecast and predict every combination.”