Twenty-four-hour banking presents the industry with the problem of how and when to back up the rivers of transactional data that flow through their systems. This feature looks at how banks ensure customers never suffer rejection online or at an ATM, and never see their backup sleight of hand.
Aucklanders and Northlanders wanting to withdraw cash from ATMs one afternoon a few weeks back found they could not.
The outage affected National Bank and a few others from the Queen City northwards. Its cause was unclear, though internal emails between National Bank and Telecom apparently referred to “faults with a switch”.
Last year, many overnight banking settlements, such as direct debit payments, were hit on two separate occasions when EDS, which handles many processes for the clearing banks, suffered human error on one occasion and faulty installation of equipment with the other.
The rise of 24/7 banking and internet access to accounts has increased the pressure on core banking systems, but the reality is, unless an internal breakdown has an external effect, customers aren't likely to hear about it.
Twenty-four hour banking, says ASB technology and operations general manager Clayton Wakefield, has shifted the goalposts, creating a need for extra backup.
“They were configured to be more available for business hours rather than every hour," he says. "It presents some architectural challenges. We have to design something so you cannot it switch off."
WestpacTrust CIO Bob Hennessy says the bank realised some time ago that to achieve the high levels of uptime demanded by its customers it had to enhance its backup strategy, so it focused on employing high availability systems.
BNZ IT manager Peter Fletcher wouldn't say if 24/7 service presented any extra problems, but noted that “many of our systems have been operating 24/7 for a number of years”.
The Taranaki Savings Bank (TSB) says the rare moments its ATMs are offline mean withdrawals are still possible but account enquiries are not.
David Tripe, director of the Massey University Centre for Banking Studies, says he is not aware of any problems with any of our banks’ backup systems.
“If there are any, they are keeping quiet about it,” Tripe says. He says it's one thing banks do better than other businesses. "It’s something they focus on getting right.”
But Tripe does have one continuity concern fuelled by the predominantly Australian ownership of many of our high street banks.
“If you are running from an Australian mainframe, you have a little exposure. It opens them to risk if the [transtasman telephone] link goes down,” he claims.
ANZ and BNZ perform some processing in Australia, which they say presents no problems. WestpacTrust and ASB perform it in New Zealand, with ASB recently opening a new backup operation in Albany, Auckland.
The new state-owned Kiwibank said it was “too busy” to comment on its backup arrangements, but Tripe says despite it using building society rather than banking software, the fledgling financier is still doing “a reasonable job”.
The pattern of success
Common patterns emerge in the technology banks use to make sure they can recover if systems go down. All claim to have a range of backup systems (such as mirroring disks at other locations), they ensure their software is well-tested, operate several processing centres and employ uninterruptable power supply systems (UPS). All claim to typically achieve near 100% reliability.
The Bank of New Zealand, which has four main processing centres -- two in Melbourne handle the production and disaster back-end systems while the ones in Auckland and Wellington can back up the other -- says it has “multiple backup systems from instant production remote disk mirroring to a hot disaster site through to offsite backups to removable backup media”. The bank employs redundant infrastructure at local sites.
BNZ's Fletcher says such data replication and mirroring, plus local recovery and contingency plans avoid outages and defects. “All mission-critical and customer interfacing systems have backup processes and supporting infrastructure,” says Fletcher. Each of the four sites provides dual power feeds, multiple UPS units and emergency generators, which are regularly tested and maintained. Business units perform regular business continuity testing.
The bank declines to identify the products or brands used, other than to say they are “the lead bunch in their category”. Service level agreements are reviewed as new technology is introduced.
WestpacTrust's mainframe systems are backed up by FDR/ABR, a system for its two major data centres that uses disks and high-speed cartridges.
The backup -- daily incrementals and weekly full ones -- is, however, regarded as a last resort, says Hennessy, for the sturdy high availability technology the bank relies on. “By this, we mean that backups are great, but they can only take you back to a certain point in time and there is therefore the threat that some data could be ‘lost'. With high availability our systems are mirrored and therefore a transaction being processed on our primary system is replicated immediately at our disaster recovery site."
Again, software is “strictly” tested and the bank employs backup power supplies. “All significant computer systems, from the data centres to LAN server rooms, have UPS installed. The larger the environment, the more complex the UPS systems. Most backups -- especially for critical systems -- are dual copied and stored at separate physical locations,” Hennessy says.
“This means that the failure of one machine or site does not impact on the service we provide as the alternate [data recovery] machine or site is immediately available and has the exact same data captured. We also back up our ‘high availability’ environments as an extra precaution and also to deal with the situation where a corrupt change could affect both sites, primary and mirrored,” he says.
WestpacTrust says 24/7 service presents no problems bar the “not cheap” cost of the technologies.
“The main considerations for people are data loss -- how much data you could potentially lose if you had to restore to a previous backup -- and recover time -- how long your systems are down while you recover from backup. These need to be weighed up against the business risk inherent in the loss of any of your computer systems and solutions constructed accordingly,” Hennessy says.
TSB managing director Kevin Rimmington says round-the-clock banking means no downtime for testing, unlike when the bank ran systems 20 hours a day, but the TSB believes its technologies can cope.
“The system is divided within the computer into segments so the system tests within the computer can still be undertaken while the online system remains operational,” Rimmington says.
The bank mirrors information between mainframes in different cities and employs a range of backup systems.
Much of the bank's backup employs Unisys technology. Back-up generators ensure power is available in the event of a blackout and software is regularly tested, but Rimmington says the equipment the bank uses is so reliable the TSB achieves 99% reliability.
“Offline systems” operate for ATMs when the computer or communications system is not available. Rimmington claims this is for less than one per cent of the time, and simply means the system allows withdrawals but not balance enquiries.
For the ASB, Wakefield confirms the use of multiple sites, multiple suppliers and multiple application systems, so if one system fails another can stand in for it. The ASB also tests its software and uses UPS power backup systems.
While the ASB acknowledges a few “incidents”, Wakefield says there has been nothing that has affected customer service levels.
The National Bank failed to respond to enquiries.
ANZ eyes sleeker, easier, backup
We will keep banks going: EDS