FRAMINGHAM (09/18/2003) - IT managers along the mid-Atlantic coast were busy this week double-checking telecommunication lines and reviewing disaster recovery plans in preparation for Hurricane Isabel, which could cause billions of dollars in damage by Friday. Most in danger are telecommunications and other technology infrastructures in both the private and government sectors.
The U.S. Census Bureau this week calculated that nearly 50 million people could be affected by the hurricane. As of 11 a.m. EST Thursday, the hurricane was making landfall near Cape Hatteras, N.C., with strong winds, heavy rain and flooding expected as far north as New Jersey and Pennsylvania through Friday.
Mark Pennington, information systems director for the Virginia Department of Emergency Management in Richmond, said telecommunication lines are of particular concern for state and private entities because a wet summer has left many trees susceptible to strong winds. "It doesn't take much to put a single node out of business. There's a large amount of concern with trees still in soggy ground, and with high winds coming in, we're worried about them doing damage to telecom lines," he said.
Pennington said before the storm hit that he had reviewed his agency's business recovery plans to ensure they're up to date, and he replicated all servers to hot sites "so if we do get hit hard, at least our information is moved off-site." He has also checked backup generators to ensure that they're running and fuel tanks are full.
The Virginia Department of Emergency Management, the state's counterpart to the Federal Emergency Management Agency, also began archiving disk hard drives as well as tapes in its underground bunker operations center for disaster recovery purposes, Pennington said. "We ended up with servers with hot-swappable SCSI drives, and it occurred to us that it would be a pretty decent way to back up data. It's a lot easier to restore from disk than tape backup. And with the SCSI drive, we can pretty much plug it in and go."
But for keeping operations running during a disaster, Pennington and others say low tech may be key. For example, he's stocked up on boxes of pencils and legal pads.
John Griffin, vice president for business continuity and emergency preparedness at Verizon Communications Inc., said that among the top tips the company is offering customers is not to rely on cordless phones for communications during the hurricane but instead to distribute phones that can work during a power outage. He also pointed out that laptop computers will also be able to function over Digital Subscriber Line and dial-up lines during an outage.
Griffin said there are more than 300 telecommunications switching stations in the path of Hurricane Isabel, an area that he's assuming will lose power.
This morning, Verizon activated its internal command and control infrastructure plan, which places staff on alert in corporate, business unit and regional control centers for monitoring and responding to any emergencies. "We're checking and double-checking communications links. There's a lot of preparation for event management and recovery," Griffin said. "When the event occurs, we have to already know who's going to do what, and how you'll monitor operations."
Verizon's corporate center is located in New York, but it has corporate backup facilities in Texas as well as state and regional redundant backup centers in Maryland, Virginia and Pennsylvania, Griffin said.
Earlier this month, New York-based American International Group Inc. (AIG) announced the opening of a new 200,000-square-foot data center facility in Fort Worth, Texas, which will provide business continuity and disaster recovery capabilities to AIG member companies. The centers have 150TB of storage capacity and asynchronously mirror all changes to production data.
Mark Popolano, CIO at AIG, said the hurricane may offer the first real-world test for restoring data to the company's New Jersey data center from the backup site in Texas.
"In the event of an emergency, we can flip our centers over and bring up our applications in the second facility," he said. "We also have people identified that in worst-case scenarios can be put up in hotels. But we have people both in Fort Worth as well as in New Jersey that can respond to these emergencies."
Bob Scarlata, CEO and president of the March Group LLLP, knows about the need for disaster recovery plans. Since 1989, his private investment banking firm in St. Croix has survived hurricanes Lenny, Georges and Hugo, which devastated the island with up to 205-mph winds for up to 12 hours.
"In the last storm I was in, some people didn't get phone service back for a couple of months. Most commercial operations had to wait at least a couple of days for it. You're talking significant losses," he said.
Scarlata believes it's crucial for a business to continue operating -- not only for profitability, but because work can represent a place of stability for workers who otherwise would be stranded at home without communications. Over the past decade, Scarlata has installed washers and dryers, refrigerators and vending machines, a water filtration system and additional backup generators to ensure that his business will continue running after a hurricane strikes.
"That's a psychological boost for your employees," he said.
About six months ago, Scarlata installed a broadband satellite communications system as a backup network for Internet connectivity. The service, provided by San Diego-based Tachyon Inc., uses a power generator and a satellite dish to provide T1-like speeds.
Scarlata said he will allow the dish to remain operational in winds of up to 75 mph, at which point he brings the dish inside until the storm subsides. The system allows his company to continue transmitting trading information to partner companies throughout the U.S.
"We're an entirely Internet-based company, so as long as we're online, we're in business," he said.