Software testing is widely regarded as the second-class citizen of computing in New Zealand, with inadequate resourcing and funding. Yet the risk of getting the software code wrong can increase the cost of a project by as much as 50 percent.
A bug picked up at the coding stage in development may cost as little as $25; if it proliferates through to post release, that cost may jump to $30,000, says Garth Hamilton, founder and CEO of Assurity Consulting. His company has just launched a platform for testing as a service, based on Hewlett-Packard’s software application lifecycle management suite of products.
Last month in Wellington Assurity held its second annual test managers’ forum, targeted at managers of test teams of 10 or more people. The 25 attendees included representatives from banks, retail, manufacturing and distribution, telecommunications, transportation and the IT industries.
The company was launched four years ago and has subsequently grown to 55 full-time staff.
One of several surveys from the forum revealed an extraordinary difference in perceptions on test expenditure. More than 70 percent of test managers thought it too low; just 8 percent of project managers felt the same; that question didn’t register with the business. A large majority of project managers felt expenditure was about right or somewhat too high; 85 percent of businesses felt the same – mainly too high.
Hamilton says comparative surveys from overseas show New Zealand is well behind Europe in testing and a little behind Australia, but significantly up locally on recent years.
Research company Ovum predicts the worldwide market for software and systems testing services will reach more than NZ$100 billion in 2013 and will grow at faster rates than most other IT services. In a 2008 report, Gartner estimated that savings from rigorous testing disciplines had ranged from 20 percent to 60 percent.
Resourcing was a major issue highlighted at the forum. Hamilton says in-house test teams are quite well regarded. “Nobody likes offshoring. Not many have adopted the cloud, but see the benefits.
“The clear message was that contractors are out of favour. Test managers would rather outsource. If you employ, say, five contractors, they all have different methodologies, then take the IP with them.”
He says those attending the forum felt that no one gave them enough equipment to test properly.
SAP testing has become a special case. “Every part of SAP is quite unique,” Hamilton says. “It’s becoming a specialty field in its own right.”
Then there is the perennial question of automation, to which people are gradually moving.
“Everyone wants it, but it is expensive and people are not sure how to pay for it,” Hamilton says.
Attendees at the testing forum were asked if, within the past two years, they could recall any situations where a lack of testing or poor testing has cost the organisation real money. Twenty percent said once; 27 percent said two or three times; 13 percent said several times.
Bearing in mind that these were representatives of banks and major government departments, those numbers are significant.
The attendees were also asked if, after testing, their products still suffered from post-release problems in the first few months: 62 percent said occasionally; 19 percent said frequently; 13 percent said always.
At another briefing this month, industry veteran consultant Colin Boswell was clear that automated testing is not the silver bullet.
The briefing was organised by Wellington company Palantir System, which represents Grid-Tools in New Zealand and Australia. Grid-Tools’ Datamaker is a tool that allows testers to subset, anonymise and create rich test data for development and testing.
“Comprehensive test automation is neither widespread nor particularly successful,” Boswell says.
“But, automating test data generation and test management can provide a partial answer.
“If done properly, it can help to minimise the variability of results, speed up the testing process, increase test coverage and generate greater confidence in the quality of the software being tested.”
He says, though, that inadequate testing can increase the cost of a project by up to 50 percent.
Boswell highlighted the need to protect personally identifiable information by rigorous data masking, or obfuscation.
“No individual’s personal or financial information should be available in test data for development, testing, training or to production staff,” he says.
“Release of personally identifiable information may have legal implications, and can embarrass stakeholders.”
Boswell describes software testing as an art that is nowhere near maturity.
“Complete testing is not feasible and at some point software testing has to be stopped and product has to be shipped.
“Successful testing requires both manual and automated testing. Manual testing is best in those areas that need spontaneity and creativity; automated testing leads itself to explicit and repetitive testing and to scenario, performance, load and stress testing.
“The key is knowing when and how to deploy automated testing and automatic test data generation.”
Gartner estimates that organisations are estimated to spend between 25 percent and 50 percent of their project life cycles on testing.