- As floods from Tropical Storm Allison inundated the Houston area, corporate IT departments were sent scrambling to restore computer systems and communications in affected buildings downtown.
Now, as the clean-up continues, it's apparent that companies with detailed emergency plans fared better than those without. But even those disaster plans were not always enough.
"The storm and flooding was probably more extensive than we've ever had in Houston," says Tim Tindle, a senior vice president of strategic planning at Solid Systems, a Houston-managed hosting provider and systems integration firm.
The city's downtown and medical center areas were hit hardest, according to Tindle.
"Companies that were prepared fared pretty well for the most part," he says. But it was the little details that often caused unforeseen problems: Although many companies had emergency power generators on hand, many of those generators were left wet and useless because they were located in basements, which were among the first areas to flood.
Some companies had installed generators on upper floors to protect them from flooding. But that led to a different problem, Tindle says. "Generators have run out of fuel because [the businesses] hadn't contracted with anyone for refueling," he explains.
A lot of computer and data communications equipment is located in the basements of many of the affected corporate buildings, and it was severely damaged by the drenching rains, according to Tindle. "Those buildings got hit, and it really took down a ton of data communications infrastructure," he says.
At the Texas Medical Center campus, on the southwest edge of downtown, even the best-laid disaster plans yielded to problems, so those plans will need to be improved before the next emergency, says Dr Ken Mattox, chief of staff at Ben Taub Hospital, one of about 54 medical institutions on the campus.
One of the major lessons learned as the hospitals lost power and shut down was that IT staff members should have been included as integral members of the medical center's emergency command centre, Mattox says.
"If I knew this was coming all over again, I would get a bunch of wire-pullers and an IT person in the command center very early," he says. "Once the hardware starts going kaflooey, it takes a little time to get it back up to speed."
Another shortcoming revealed during the storm is that hospitals across the city have different data file formats, making it often impossible to transfer electronic patient records to another institution still up and running, Mattox says. "It would be very nice if I could dump their medical records into our system," he says.
At the Harris County government offices, five of 26 buildings were knocked completely offline due to the flood waters, which ravaged underground utility and communications tunnels, destroying wiring, phone cables and other systems.
"We had the perfect storm in the tunnel systems," says Steven Jennings, the executive director of the county's central technology center. What was learned, he says, is that the traditional method of locating data centres in the bowels of buildings is dangerous because it can leave critical systems vulnerable.
Instead, data centres and communications equipment hubs need to be located on upper floors, isolated from flooding and other problems. Future government buildings, including several on architectural drawing boards right now, will incorporate such changes, Jennings says.
"You train for scenarios and what you find out is that Mother Nature has a curve ball," he says.
Tom Smith, a senior vice president of IT at the Houston headquarters of trash hauler Waste Management, says that although his company's experienced no flooding, some areas did lose electricity for about 48 hours, and backup generators had to be put into use.
The darkened hallways and stairways taught IT staffers a simple lesson, he says.
"We should probably have more flashlights on hand," Smith says. "It's the little things you don't think much about."