The bugs, beauty and benefits of beta testing

Don't volunteer to be a beta site unless you're prepared to commit to it, say those who have done it

In 15 years at Case Western Reserve University in Cleveland, Ohio, systems engineer Jim Nauer has been involved in six major beta tests. In each case, Nauer was motivated by the failure of existing technology to meet Case’s technology or business needs.

For example, in the mid-1990s, the university beta-tested a native version of Novell NetWare that ran on Apple’s PowerPC platform. Nauer was excited about the product’s potential.

“It meant guaranteed hardware compatibility and plug-and-play device drivers and, in those days, there was quite a bit of voodoo involved in getting Ethernet and SCSI cards configured correctly,” he says. “You didn’t just pull [the network operating system] out of the box and click through a wizard to install it.”

The beta product worked beautifully, he says. Unfortunately, Apple decided not to develop it commercially. “It would have been really neat if they’d finished debugging it and it solved some of our issues,” he says. “But it didn’t come to pass.”

As Nauer has found — during the Apple beta and his other more successful tests — there are numerous benefits to being a beta tester. The most important is the ability to get your hands on cutting-edge technology to gain a competitive advantage.

However, be prepared to deal with the frustrations, the resource demands and the possible disappointments that accompany the experience. Not only does beta testing stretch the resources of IT staff members, but it also can try their patience, as they are, after all, working with buggy products. Vendor approaches to beta-testing can range from very organised to haphazard, and your own testing and bug-reporting process has to be rigorous.

As Nauer found with the Apple beta, there’s no guarantee the product will ever hit the market. Still, Nauer says Case Western came out a winner. “It enabled us to get our hands on NetWare 4.1 code before we had the PC native version,” he says. “We got knowledge and experience we were able to apply elsewhere.”

In fact, other beta testers report that, even when things go wrong, they still feel the pros outweigh the cons. Reasons include early access to new technology that can solve long-standing problems, the opportunity to influence product development and a direct line to code engineers.

“You’re doing it because you have a vision; you want to beat the guy down the street and improve your revenue quarter over quarter,’’ says Louis Barton, executive vice president at Cullen/Frost Bankers in San Antonio.

Secrets to successful beta-testing

The secrets? Don’t expect beta-testing to be a free ride, and be prepared to jump into it with both feet, says Mark Moroses, senior director of technical services at Maimonides Medical Centre in New York.

Moroses has been involved with several beta tests and, while he says his team is very accomplished at the process, it’s not something he’d turn into a hobby. “We only go down the [beta test] road if we believe the technology is superior,” he says. “If it’s a tie, we prefer to work with an established platform with [fewer] bugs and support issues.”

Moroses counts just one time that a beta test came close to being a failure. He was asked to test a thin-client version of a product he was already using in-house. Unfortunately, the beta version wasn’t ready for prime time.

“As our debugging process went on, it became apparent that their approach needed to be completely redone,” he says. His team pulled the plug on the beta effort and submitted feedback to the vendor. But Moroses still contends the experience wasn’t a waste of time because it gave the team deeper insight into the technology’s capabilities.

To survive a beta test without the negatives outweighing the positives, Moroses says, your IT organisation has to start with several characteristics, including a strong problem-solving and partnership mentality. In fact, his group is often approached by vendors to do beta tests because of those traits, he says.

“They’re looking for certain types of folks,” he says. “Our engineers and people in the IT department don’t say, ‘I have to call the vendor’ when something doesn’t work — we spend time finding the core of the problem.”

But, if you don’t manage that effort, you can drown in it, Moroses says. “You have to stay cognisant of where your entire portfolio of projects and upgrades are and what people’s workloads are,” he says. One management tip that he offers is to conduct just one beta test at a time. “If we’re at the end of one, we might start up another, but we wouldn’t do three or four at a time,” he says.

It’s also important to establish upfront what you expect to gain. For instance, Moroses insists on getting the names of the software developers he’ll be working with, not just a support phone number. He demands 24-hour turnaround time on issues that arise when the tested product goes into production, as well as senior support personnel on-site.

“You have to be careful whom you partner with,” he says. “Not all vendors will voluntarily suggest bypassing the normal support centre.”

Moroses also negotiates a price break on the product or better software maintenance terms. “We tie it to an economic benefit,” he says. “We end up making the product better anyway, so we might as well get a financial reward for it.”

Beta-testing can be a drain

The resource demands of beta testing made Barton think twice before agreeing to test the Cognos 8 business-intelligence system. “It was a balance between the commitment of resources we’d have to apply and what we’d get back,” he says. “If you’re going to do it right, you have to really do it and put your best people on the project.”

Barton is not alone. “Resource constraints are the number one challenge for our customers,” says Faiyaz Shahpurwala, vice president of the datacentre practices group at Cisco Systems, who says tests can average out to be a three- to four-month effort. “There are times when we’re not able to complete a test plan on time, and we have to keep pushing on our side.”

However, because Barton’s US$10 billion (NZ$15.8 billion) financial services conglomerate was making full use of other Cognos products, and he planned to eventually move to the new business-intelligence system, he decided beta-testing would be a good way to prepare for the implementation.

Barton says Cognos was an “extremely organised and thorough” partner in the beta test, which is not always the case. “It wasn’t, ‘Here’s some software you can install so we can get some feedback,’” he says. “There was a plan, a structure, a schedule, names of people to support us, good communication and objectives, which was a new experience for us.”

Preparing for the beta test required several phone conferences, web conferences and email communications, after which a few Cognos engineers came on-site at Frost for three weeks. The entire project lasted three months and required the full-time effort of two to three IT staff, plus some support staff and business users.

But, no matter how organised Cognos was, Barton says the effort would not have been successful if his own team didn’t have a structured approach to testing, complete with test-control forms that recorded the objective and expectations of each test, and documented what happened during testing. “When you get an error, you have to make sure you can document it and communicate about it thoroughly enough to the vendor so they can fix it,” he says.

While Barton says the beta test prepared his team to implement Cognos 8, there were some downsides to the experience. “Software always has bugs, but beta software will have more bugs,” he says. The time involved in reporting on those bugs, waiting for a fix and then testing it again can be a source of frustration. “It’s a drain on resources, especially when everyone already has a full-time schedule and full workloads,” he says.

The testing period is intense. “Testing is an art,” he says. “Many conversions fail because of insufficient testing or the wrong types of tests, which takes planning and discipline, and procedures and documentation.”

But the benefits outweigh the risks

If you plan right, Barton says, the downsides are offset by the rewards, one of which is a head start into new technology, which can mean a competitive advantage. “The hands-on personal training reduces the learning curve for support staff, business users and everyone else involved,” he says. “It’s hard to quantify, but they’re real when you start using the product.”

John Denardo, CTO for Eagle County, Colorado, agrees. As an early adopter of storage technology from LeftHand Networks, Denardo has beta-tested new products and configurations for the vendor.

“I’d be sending my guys out to train otherwise anyway, and this way, we get more detail,” he says. “Having access to the engineers working on the code is important.”

Case Western’s Nauer says beta tests give the university a big leg-up in terms of its deployment plans. For instance, it has been able to make an early move into storage virtualisation by beta-testing Cisco’s MDS Series switch in 2002. The university already had a small storage-area network, but it wanted to migrate to iSCSI technology, which MDS enables. “Cisco brought to the table things we didn’t even know existed,” Nauer says, such as intelligent applications at the switch level. “We don’t need it today, but it’s absolutely strategic for us to move in that direction.”

Beta tests can give companies a chance to influence product development. “You’re providing feedback that will help them make money, so they’ll listen and respect your input versus if you went blindly through your sales rep,” Barton says. This depends, of course, on the tester’s ability to truly enter into a partnership with the vendor.

Getting leading technology on-site is also a boon for recruitment and retention efforts. “If your tech guys can put on their resumé that they’ve worked with cutting-edge technology they’re less likely to seek other employment,” Moroses says.

Don’t forget the financial incentives. A couple of years ago, Michael Fine, director of client services for Centre Code, helped beta-test a high-end security product which ended up providing the user with US$20,000 worth of product advice for free.

But, while lower software costs can be a benefit, experienced users warn that it can’t be a driving factor. “You can’t go into beta-testing because it’s a good way to get some freebies — that’s the wrong motivation,” Nauer says.

It’s better to approach it as a business engagement. “If you treat a beta like it’s outside of your core mission you’ll be disappointed. But, if you treat it like any other business project, you’ll get a good result,” Barton says.

Advice for beta testers

Beta tests are not something to enter into lightly. You may need to put lower priority projects on hold or delay other projects by a few days or even weeks. “It’s a strain because you still have to get the other work out, so you’re working lunches and overtime,” Barton says. So, if the beta test is low-reward or other high-visibility projects are going on, or you don’t have a test environment prepared, don’t do it, says Barton.

Negotiate everything upfront

If you want the names of support people or other terms of agreement, make sure you do it upfront and in writing. “You can’t just pretend your previous agreements don’t exist anymore,” Barton says. “It has to be done legally and in an organised manner.” And, while financial incentives can be issued, be sure you get a commitment before you go into testing, when you still have leverage, he says.

Apply yourself

When you’re applying to be a beta tester, fill out the form completely, says Michael Fine, director of client services for Centre Code. “We get forms where they don’t give data on what equipment they have to test with,” he says. “And people who can’t spell are less inclined to be chosen.” Abide by the confidentiality rules, he says, and don’t expect a miracle. “Don’t set overly high expectations for the product before you get it,” he says. “This is beta; it’s being tested. Go in with an open mind.”

Join the newsletter!

Error: Please check your email address.

Tags Development IDtestingbeta

More about AppleCiscoCiscoCognosCognosLeftHand NetworksNovell

Show Comments
[]