Large questions have appeared over the accuracy of two recent reports comparing the relative costs and benefits of the Linux and Windows operating systems in which Windows was painted as being superior to its open-source rival.
The reports, Forrester Research's "Is Linux more Secure than Windows?" and a Yankee Group survey on the relative costs of running the two operating systems, were both issued over the past week.
The security study -- whose raw data was vetted by Linux distributors Debian, Mandrakesoft SA, Red Hat and Suse Linux -- found that on average, Microsoft patched flaws faster than Linux vendors. The Yankee Group survey reported that, except for small businesses with customised vertical applications, companies deploying Windows enjoyed a lower cost of ownership than those with Linux.
But the Linux distributors involved in the Forrester study today issued a joint statement calling the study's conclusions inaccurate. And The Yankee Group's methodology has been called in question, with critics arguing it could not have possibly delivered objective results.
Yankee doodle dandy
Yankee Group's survey, it turns out, was based on the responses given by companies that had been selected from a mailing list devoted to Windows issues. The survey was funded and carried out by Sunbelt Software, a vendor of Windows utilities, which publicised the survey through a mailing list called W2Knews, which bills itself as "The world's first and largest e-zine designed for NT/2000 System Admins and Power Users".
Sunbelt itself clearly identified the survey as being aimed at Windows system administrators. In the 16 February edition of W2Knews, which launched the survey, the company said it and Yankee Group were "surveying Windows Sites" to see how they were "responding to the Linux phenomenon and the TCO question".
The survey was carried out via an online form, which contained no controls and so was open to manipulation. Yankee Group supplemented the raw figures with in-depth executive interviews taken from the list of survey respondents, who were all subscribers to W2Knews.
As such, the survey can only be said to be representative of system administrators that already use Windows, rather than sysadmins in general. In the executive report, its author Laura Didio wrote that "a significant Linux deployment or total switch from Windows to Linux, would be three to four times more expensive and take three times as long to deploy as an upgrade from one version of Windows to newer Windows releases."
However, Linux supporters say that such a claim knowingly gives only part of the picture in order to build the notion that Windows is cheaper than its open-source alternative. The survey failed to consider other important factors in switching operating systems, such as the freedom of choice that Linux makes available, since companies can easily change vendors and support contractors. These benefits are more readily recognized by CIOs and IT directors, said Red Hat's European marketing director Paul Salazar.
Salazar also claimed the Windows-to-Linux focus was not representative, claiming that Red Hat (which controls about 70 percent of the Linux market) would rarely pitch Linux as a cheaper alternative to Windows servers. Instead, he said, the major opportunity for Linux is the huge installed base of Unix servers. In this case, Linux costs less, runs on cheaper hardware and is more compatible that both Unix and Windows. "With Windows it's never a night and day comparison," he added.
The Yankee survey is just the latest to compare the TCO (total cost of ownership) of Windows and Linux, but is the first (unlike those from Jupiter Research Inc., Forrester and IDC) that have not been requested and funded by Microsoft.
Forrester's security study is a somewhat different matter: the research firm was eager to distance itself from the furore surrounding earlier publication of its Microsoft-funded research, which led Forrester to bar companies from publicising research they themselves had backed.
The company allowed Linux distributors to scrutinise its raw data, a database of all the security vulnerabilities for Linux and Windows over the course of a year, and made the data publicly available.
As a result of this collaboration, Linux vendors accept that the raw data is correct, but in a public statement this week they said Forrester's analysis had led to "erroneous conclusions".
The report compares the "days of risk", calculated as the number of days between the disclosure of an operating system vulnerability and the release of a vendor patch, for Windows and several Linux distributions. Microsoft took on average 25 days to release a patch; Red Hat and Debian 57, SUSE 74 and MandrakeSoft 82, Forrester said.
Such figures are flawed however, the Linux distributors claim, because they use a straight average and take no account of how significant the security holes are. As such, obscure, low-risk problems that don't need immediate fixing are treated the same as highly critical flaws. "Our users will know that for critical flaws we can respond within hours," a statement by the vendors said. "This prioritisation means that lower-severity issues will often be delayed to let the more important issues get resolved first. The average erroneously treats all vulnerabilities as equal, regardless of the risk they pose."
Forrester analyst Laura Koetzle, who authored the report, said she had considered giving critical vulnerabilities extra weight in the average, but decided against it. "I considered responsiveness, or days to fix, relative severity and thoroughness separately, partly because I wanted the scoring to be exceedingly easy to understand and transparent for the readers," she said. Readers were free to analyze the raw data in this way.
The report distinguishes high-risk from lower-risk vulnerabilities, but the distinction was not included in the key average figures. The Linux vendors also criticised the report's definition of high-risk vulnerabilities, arguing that it included numerous routine bugs.
"This is one of the worst cases of doublespeak out there," Red Hat's Salazar said. "It's exceedingly difficult to peel through those statistics." The important thing, he said, was to make sure customers were able to have secure systems, and Red Hat was succeeding at that. "From our point of view, there's no crisis," he said.
OSIA: biased reporting = unfair business assumptions
Open Source Industry Australia (OSIA) also hit out against so-called “independent” research into Linux software, identifying both the Yankee report as well as a recent Microsoft commissioned IDC study into the cost-effectiveness of Microsoft versus free and open source (FOSS) software, as examples of “substantially biased” reporting.
In a press statement issued this week, OSIA called for industry analysts to challenge the findings presented by both research firms, claiming their biased results prevent Australian governments and enterprises from making accurate decisions about the “efficacy of open source to solve their business needs”.
OSIA spokesperson Con Zymaris said both the IDC and Yankee Group reports were based on “inequitable and unfair assumptions, criteria or scenario choices and sample-data statistical processes”.
“We recommend to all organisations considering migrating to, or adopting Linux and open source software, with analyst reports hand, that you thoroughly research any claims made by these analysts with respect to the attributes of Linux and open source,” he said.
For example: “while Win2Knews and Sunbelt software may be a great resource for Windows users, they would obviously be a poor pool from which to draw any unbiased opinion on Linux and open source software”, Zymaris said.
He added Linux and open source “can and does compete strongly, but only when contesting on a level playing field”.