AT&T's frame relay network experienced a major outage on Tuesday due to failed network nodes, AT&T CEO Michael Armstrong said yesterday.
"We had an outage in our frame relay network which started around 3pm yesterday," Armstrong said during a press conference. "We basically let our customers down," Armstrong said, adding that he apologises to "each and every one of them."
AT&T's frame relay network services customers are businesses which transmit large amounts of data, such as banks and their automated teller machines, or stores which receive credit-card payment authorisation via the network.
AT&T said the outage was initiated by a faulty connection between two frame-relay switches which spiraled across the network, interrupting service for what AT&T would only characterize as "in the thousands" of customers.
The company's voice and wireless networks were not affected by the break in service, officials said.
AT&T officials said that service had been 96% restored by midday Tuesday but did not have an estimate as to when the restoration would be complete. Charges for frame relay service customers will be suspended for customers until AT&T has identified, isolated and fixed the problem, officials said.
Analysts said that the fact that the outage happened to highly reliable AT&T underscores both the vulnerability of data networks and the need for network redundancy.
"It doesn't take much for a problem to cascade down a network," said Jeffrey Kagan, president of Kagan Telecom Associates in Marietta, Georgia. "Because you can't have something that's flawless ... you've got to have backup and you've got to have redundancy both on the part of the carriers and the customers."
Analysts also said that AT&T was handling the problem appropriately.
"What else can you do other than fix the problem, apologise and offer credits?," said Dan Taylor, senior analyst at the Aberdeen Group Inc. in Boston.
"It's nice to see them putting their money where their mouth is," Kagan said. "They're embarrassed. They're standing behind their service and there's not really a whole lot more you can ask, other than flawless service."
And flawless service will continue to be elusive, even for the most reliable carriers, analysts said.
"This is not the last time we will see these kinds of problems," Kagan said. "I guarantee this will happen again, whether it's with AT&T or another carrier."
AT&T can be reached at http://www.att.com/.