EDS is confident its two failures last year to keep the banking settlements system working will never happen again.
Mike Morris, the service executive for the core infrastructure in the region, says both failures were “extremely unusual and had different causes”.
One was “human error that occurred during routine maintenance” and another was due to installation issues with a particular piece of equipment, he says.
“When these types of outages occur, we do an extensive root-cause analysis to identify why the outage happened and, most importantly, to ensure that it does not happen again,’ he says.
In the installation issue in Auckland, specialists from EDS and the supplier were called out to identify and remedy the issues. “Since we carried out the remedial work, we have carried out monthly full tests of generators and cut-over of UPS systems -- all have which have been successful,” Morris says.
This banking industry “interchange”, notes EDS, is not a full 24/7 facility, having defined activity times and downtimes during which backups are easily made. However, EDS does provide “true" 24/7 services for other clients like the New Zealand Immigration Service.
Morris says a number of backups are provided for EDS clients depending on their needs and contracts. For the banking settlements, EDS has a two-site operation based in Auckland and Wellington, with full processing, storage and networking capacity at both sides.
“One site acts as the primary switch. The other site receives real-time transactional data, allowing it to trail the primary and be switched rapidly to primary mode,” says Morris. Defects are avoided through vigorous testing and change management.
“A full replica of production primary and backup system processing is maintained on a separate development system. Changes are only made after regression and control testing,” he says.
Operational defects are detected through a large number of control checks built into the applications themselves, he says. In particular, duplicate or missing transaction files are tracked automatically and don't rely on human checking. Where defects are detected, reports allow quick targeting of the problem, allowing resolution within the "very tight timeframes" available for the interchange.
“With regard to outages, we have the dual site real-time data as above, but also ensure that each site has internal redundancy to reduce the chance of outage,” Morris says.
Redundancy includes dual UPS on the primary site that can operate on diesel generators indefinitely, main power supplies from two different substations, RAID-1 disk storage and "network front-end processors with dual engines and automatic switching". Critical data is backed up offsite.
Further backup, says EDS, comes from using IBM S/390 systems, whose mean time between failure is several years.
EDS has processing centres in Auckland, Hamilton and Wellington with back-up systems in each centre. Real time data is also sent to a back-up site with equivalent redundancy and EDS regularly tests them, including for disaster recovery.