Validity of NT testing questioned

A month ago, the Mindcraft performance-testing lab touted a benchmark that showed Windows NT outperforming Linux. Charges of jury-rigging followed when it was revealed that the test was commissioned by Microsoft and conducted in Microsoft's laboratories with experts and resources devoted to tuning the NT portions of the test. Now the test is to be run again - but Microsoft says Windows 9x and not Windows NT clients must be used.

A month ago, the Mindcraft performance-testing lab touted a benchmark that showed Windows NT outperforming Linux.

Charges of jury-rigging followed, however, when it was revealed that the test was commissioned by Microsoft and conducted in Microsoft's laboratories with experts and resources devoted to tuning the NT portions of the test -- but with no effort to equally tune the performance of Linux.

As a result, earlier this month Mindcraft invited the Linux community to participate in yet another benchmark test.

However, Microsoft's specification that Windows 9x clients must be used, rather than Windows NT clients, raises questions about the retest's fairness.

The more or less obvious conclusion was that Microsoft performed this test internally, and liked the results when using 9x clients, but not the results with NT clients. At first glance, the client type should not matter much because the comparison is of server-side Server Message Block performance.

Different clients have different performance characteristics. NT's robust, multithreaded architecture should allow benchmarking applications to open more connections at once, as well as see greater throughput to each connection. Presumably, that is where Microsoft is concerned about comparison with Samba, which adds native NT file and print services to Linux.

The Netbench test Mindcraft uses is suspect from the start, however, and is more of a "speeds and feeds" test than a real world performance test.

Look for InfoWorld's file and print benchmarks of NT 4.0 vs. Linux in next week's issue.

Additional reporting by InfoWorld Seattle Bureau Chief Bob Trott

Join the newsletter!

Error: Please check your email address.
Show Comments
[]