A robust mobile network requires not just good technology, but an architecture designed to mitigate any shortcoming that technology may have and sound operational processes.
Vodafone's general manager for network services, Mike Davies, says all three elements must be in place to avoid potential service disruptions.
In the wake of Telecom's major mobile network outage, Davies told Computerworld all network components have limitations and strengths. Limitations need to be mitigated in the design of the network and these are then supported by operational processes that protect the network from failure.
"Some elements have huge fault tolerance built into them," Davies says, citing the DX200 switches the company uses from Nokia. He says, touching wood, that Vodafone has never had a complete failure of one of these units.
"They are incredibly robust and reliable," he says.
However, other units, such as radio network controllers (RNCs) do not have that level of built in redundancy.
That's why Vodafone has six of these and uses one, connected to a small number of cell sites in Auckland, as a test platform for software upgrades before these are rolled out, one at a time, to the other five.
"We test the software to death in a test environment before putting it out," he says.
Software upgrades and changes are rolled out on the Auckland RNC and then left to operate for two weeks, before being deployed further.
But beneath all that, and possibly even more important, is Vodafone's 2G network. If the 3G network were to fail, phones would automatically switch over to 2G and voice and text services would continue. Only mobile data would be significantly disrupted.
Telco company 2degrees could have similar backup, having launched a 2G network first and following that with a 3G network. However, Telecom's legacy network is not compatible with most of the handsets now running on its new XT network. So a failure means a loss not just of data, but of voice and texting as well.
Davies says the weakness of RNCs has to be compensated for through having several of them, so any failure affects a smaller number of customers. The six 3G RNCs are complemented by 24 2G units.
Vodafone also deploys routers on the network in groups, with six interconnecting, to ensure further redundancy in that part of the network. Most routers have two power cables, but they also work in pairs in case of failures.
"We spend a lot of time discussing 'what-if' scenarios," he says. "The operational process is what is often overlooked. Often the machines will give notice of trouble through alarming and monitoring."
That's when process kicks in to ensure remedial action is taken.
When it comes to the possibility of a complete network failure, Davies is loathe to use a word like "never".
"It's extraordinarily unlikely and would require a combination of events," he says.
Davies compares building a network to painting the Harbour Bridge. "You never stop."