If a project runs over time and over budget, then commitment to software testing is the first thing to go by the board. That's according to the chief executive of one vendor who doesn't want to be named, as he deals largely with government departments.
"There is a distinct lack of interest in testing in the public sector," he claims. "There's far more interest in the private sector.
"It is evident those organisations that have moved to Agile development factor testing into each iteration. Therefore, if things get tight [in time], they can be rapidly addressed, compared to trying to shave three or four months off at the end of a big project [when testing is cut].
It used to be quite common in government projects for the budget to be broken down into percentages for individual components such as testing. But methodologies have changed to a large degree. There's Agile and there's iterative, which encompasses such technologies as .NET and Java.
"Testing is an area of IT that has never been seen as sexy," the chief executive says. "It is about a group of people trying to break something and, as such, is almost considered as a negative force in systems analysis.
"It has become the poor cousin. But without it, all the good things will be undone if the reputation of what has been built suffers from poor quality control."
He claims that in the public sector there are not always the people available who have the wherewithal to drive testing. Budgets don't necessarily cater for large numbers of people who may not be employed 100 percent of the time after projects finish.
He wonders whether offering testing as a shared service might be a way around this.
Computerworld asked several government departments what percentage of project budgets they generally allocated to testing. The answers vary, considerably in some cases.
The Ministry of Social Development commented that it was not possible to provide a precise percentage set aside for software testing, due to the number of variables that must be considered when comparing IT projects.
"Aside from the size and nature of projects, there are also different types of testing, such as unit testing within the software development phase and then regression, integration, user acceptance and performance testing.
"Taking that into account, the ministry will generally dedicate between 15 and 20 percent of a software project's budget to testing. There may, on occasion, be individual projects that fall outside that range due to their specific characteristics," a spokesperson said.
The Ministry of Health says that in 2010-2011 it spent between 7.5 percent and 8.5 percent of IT project budgets on testing.
That is similar to Inland Revenue's testing costs, which in the year to June 30, 2011, were approximately nine percent of total project costs.
A spokesman says the testing costs include employee, contractor and overtime effort.
Total project costs included all capital and operating costs incurred by the project.
Ministry of Education CIO Leanne Gibson says keeping track of the testing effort as a percentage of overall development spend is an important metric, because it provides an indicator on effectiveness and efficiency in the application lifecycle management.
"Key areas of this cover project leadership, development effort, business analysis, testing (unit, release, integrations, UAT) and architecture/design," she says.
"Testing will vary, depending on the complexity and risk of the project. As an overall percentage, the range for test expenditure tends to be between 24 percent and 34 percent, based on an application lifecycle management approach. This includes costs arising from project management, business analysis, architecture/design, testing, development and other related project costs."
Accident Compensation Corporation acting CIO Jane Hardy says the nature, extent and cost of software testing depends on the specific project being carried out and can vary greatly between projects.
"Without investing the considerable time that would be required to analyse a sufficient number of completed projects to establish a statistically accurate average figure, it is difficult to provide a generic indication of software testing costs as a percentage of ACC's overall IT project costs," she says.
Hardy did however provide some insight into the level of testing carried out by ACC's internal test team.
She says that this year, for continuous improvement releases (minor changes to existing systems), around six percent of the costs have been for testing;
However, generally speaking testing costs would not be expected to fall below 10 percent for a major project, excluding third-party test costs as part of the development cycle. "If these costs were included, we would estimate an average total test cost of 15 to 19 percent."
What is software testing?
Several software testers who spoke to Computerworld during research for this article provided the following insights into testing methodologies.
The definition of software testing is that it is the process of analysing a software item to detect the differences between existing and required conditions (that is, bugs), and to evaluate the features of the software item. Fundamental test processes are: test planning and control, analysis and design, implementation and execution, evaluating exit criteria and reporting, and test closure activities.
A Waterfall process means the test team is often independent of the development team. Tests are performed on artefacts produced in the previous phase and are rarely automated during the testing window.
An Iterative process, means a full testing cycle is performed within each iteration, and formal acceptance occurs during the transition phase.
Testing within an Agile project means the tester is part of the development team, and tests are automated in conjunction with software development. The purpose of testing on an Agile project is to prevent defects, not to discover them.