Users begin to demand software usability tests

The Boeing Co. is changing the way it buys software and is making a product's usability -- the ease with which end users can be trained on and operate the product -- a fundamental purchasing criterion. It's a move the aerospace giant sees as an essential means of controlling IT costs.

"We simply can't afford to pay for products that cost us a lot of overhead anymore," said Keith Butler, a technical fellow at Boeing's Phantom Works research and development arm. When thousands of end users are involved, design flaws can cost millions of dollars in lost time and productivity, he said.

What's helping Boeing change its purchasing approach is the recent development of a standard for comparing product usability that was spearheaded by the National Institute of Standards and Technology (US).

Called the Common Industry Format for Usability Test Reports, the standard outlines a format for reporting test conditions and results and gives user companies enough information about a test to replicate it. It's a means for objectively evaluating software, say its backers.

Next month, NIST intends to seek international standards recognition. The standard has already received American National Standards Institute certification.

CIF's success as a purchasing tool depends on whether other companies follow Boeing and make usability a "peer," as Boeing officials put it, of such traditional purchasing criteria as a product's functionality, price and system requirements. If that happens, users say, the standard could have a far-reaching effect in improving the usability of software.

"The real value of CIF, quite honestly, is that if vendors know we are expecting it, meaning large software purchasers, they will focus their attention on usability and hopefully make their products better before they ever come out the door," said Jack Means, superintendent of usability at State Farm Insurance Cos..

Boeing played a lead role in the development of CIF after its experience and internal studies showed that usability played a significant role in total cost of ownership. In one pilot of the CIF standard on a widely deployed productivity application, the Chicago-based company said improved product usability had a cost benefit of about US$45 million.

Butler said it's much better to have vendors refine an interface design "than to have thousands of end users doing it involuntarily on top of their jobs and then just feeling frustrated."

Spotting Problems Early

Doug Francisco, director of IS architecture at Boeing's commercial airplane division, maintains that CIF will improve the ability of the IT department to spot problems before a product is rolled out to employees. The company has looked at usability in purchasing, "but sometimes we wouldn't discover the inefficiencies of a software product until we brought it in-house," he said.

Microsoft, in its capacity as a CIF development participant, has incorporated the usability testing it conducted on its Windows XP, Windows ME and Windows 2000 operating systems into the CIF format, said Kent Sullivan, Microsoft's usability lead for the Windows client.

Sullivan said Microsoft is prepared to use CIF but noted that its adoption will depend on customer demand. Microsoft typically doesn't receive questions about usability from customers, so when users do ask about it, he said, "it indicates that they are ahead of the curve a little bit."

In the past year, interest in CIF has grown from about 50 firms taking part in the NIST effort to more than 150, including PeopleSoft Inc., Oracle Corp. and Eastman Kodak Co.

The CIF format will also be adapted for hardware testing, said Emile Morse, who heads the effort for NIST. Morse said she believes CIF makes it possible for vendors and users to discuss usability as a science rather than marketing hype. "I think CIF gives a lot of credibility to the practice of usability," she said.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about Boeing AustraliaEastman KodakKodakMicrosoftOraclePeopleSoft

Show Comments