Rootkits: the next big enterprise threat?

Late at night, a system administrator performed a routine check of a crashed server, one of 48 systems comprising a major online infrastructure that generated about $US4 million per month in revenue. He was a bit surprised that the system had gone down, as it had been humming for months without any indication of being prone to crashing.

The check uncovered three encrypted files. The administrator called on information security analysts Mandiant to examine them.

What Mandiant found was that an unauthorised kernel modification had caused the system to become unstable, and that the modification had compromised the system's security as well. To determine the extent of the breach, each of the 48 servers needed to be taken offline, booted in a controlled environment, and analysed for three to five hours each.

About half had the crack installed, forcing the company to assume that all credit card information had been compromised. What had first seemed routine resulted in a financial nightmare -- one that many companies are leaving themselves exposed to, unaware of the increasing pervasiveness of rootkits.

Every organisation is aware of the importance of securing core systems, networks, and end-user equipment in an increasingly mobile and malware-saturated world. But what most may not realise is the growing threat of malicious software intended to keep its presence hidden from administrators and traditional antivirus software.

Termed after early Unix packages designed to replace commands that would otherwise alert admins to the presence of intruders who had "root" or admin access to systems, rootkits are on the rise among those seeking to steal corporate and personal information for financial gain.

Rootkits alone, of course, are not inherently malicious. But when packaged with malware, they can facilitate deeply compromising security breaches undetected, especially as they become increasingly popular for attacks on non-Unix systems, specifically Windows.

And with Forrester Research recently estimating that security breaches cost companies between $90 and $305 for each record lost, who can afford to turn a blind eye to what may invisibly be leaching sensitive data from their network?

The rise of rootkits Rootkits date back to the earliest years of the Internet, when crackers created cloaked variants of Unix commands to ensure their deeds on compromised systems would go undetected. A concern mainly of system administrators for Net-connected Unix systems, rootkits remained relatively low-profile for many years, until Sony BMG Music Entertainment's Windows (digital rights management) boondoggle of 2005.

In an attempt to enforce copyright protection, Sony BMG developed a rootkit that surreptitiously installed XCP (Extended Copy Protection) or MediaMax CD-3 software when music CDs were played on a PC. Poorly designed, the software opened holes in the Windows OS, facilitating infection by viruses and causing other system problems.

Mark Russinovich, now a technical fellow at Microsoft, discovered the rootkit's behaviour, which he then announced on his blog. The resulting furore and further illustrations of the fallout of the rootkit led Sony BMG to recall the CDs and issue a removal program.

Unfortunately, the removal program was equally poorly designed, leading to additional privacy and security concerns, as documented by Russinovich.

This incident awoke two groups to the potency of Windows rootkits: crackers and professional criminals who break into computers on the one side, and the companies who create software to protect systems on the other. Already entrenched in a high-stakes battle over malware, the two camps now had a new, potentially more damaging front on which to contend.

The Computer Economics 2005 Malware Report, the organisation's latest, put the cost of malware in 2005 at $US14.2 billion. The ability of malware authors to hide their scripts from antivirus software's capability of automatically detecting, protecting, and eradicating most malware would only serve to escalate the stakes, especially as malware authors' motivation "continued to shift from a general desire to inflict damage to an intent to gain financially, through theft of personal information such as credit card data or by gaining access to financial accounts", according to the survey.

The greater emphasis on mobility in the enterprise has certainly contributed to the increasing likelihood of infection with cloaked malware. So too are the various unpatched security holes in Microsoft Windows and related products, which provide access for automated rootkit installation.

The proliferation of rootkits -- which are used to cloak files on disks, system hooks, and processes running on systems -- is alarming, as spyware developers and malware authors are creating bot networks that use rootkits to evade detection, hiding not only the malware but also what information is being obtained.

Some of the more sophisticated rootkits even modify and corrupt Windows APIs. (For more detailed information on rootkits, visit or read Greg Hoglund and Jamie Butler's Rootkits: Subverting the Windows Kernel.)

Part of what's fuelling the proliferation of rootkits is the ease with which they can be implemented.

"It has definitely ramped up over the last year and a half to two years," says Butler, principal software engineer at Mandiant. "It has become very easy for malware authors to cut and paste these technologies into their code set to maintain a presence on the machine."

For the time being, malware rootkit use remains crude. "Many of the attacks are unsophisticated," Butler says. "We're not seeing leading-edge rootkit technologies." But the dynamics of intrusion and response that are the hallmarks of the security industry are fast pushing the use of rootkits in innovative directions.

The front lines of rootkit defence Rootkits employ a variety of methodologies to conceal themselves. Some overwrite kernel structures to replace the hooks normally used by Windows commands. Others create files within the file system that are effectively invisible. Still others capture hooks in Windows commands to corrupt their outputs.

Many hook into addresses used for kernel services, changing the address of the table entry so the rootkit gets called before the real Windows system call is performed. Extensive details on current approaches to concealment are available at and other Internet sites.

One recent methodology posted on involves loading a drive in place of the Windows null.sys dummy driver. The same post outlines three other methods for hiding drivers and offers the code for null.sys replacement.

In terms of defending against infection, Microsoft Windows Vista 64-bit resource protection and Software Restriction Policies in Windows XP provide some assurance, but developers of rogue software have proven their ability to find new ways to hide code on compromised machines. In fact, the rootkit front is fast transforming into an arms race, with each side innovating in response to developments the other camp pushes forward.

Keeping on top of the latest modes of prevention is essential, especially if you are responsible for a fleet of computers running any variations of Microsoft Windows.

As for the big security players, a number are appropriating the traditional approach to viruses, using signature-based searches to track down known rootkits and applying related fixes. Two of the major vendors, Symantec and Trend Micro, however, are taking unique tacks in combating rootkits.

Symantec is leveraging mapping technology to discover rootkits on compromised systems. Oliver Friedrichs, director of emerging technologies for security response at Symantec, believes rootkit eradication requires a stable, reliable design that minimises false positives and mitigates system instability during rootkit removal.

To make good on this mission, Symantec has employed the expertise and technology brought on board during the Veritas acquisition. Using VxMS (Veritas Mapping Service), Symantec's Norton Internet Security 2007 maps data on the hard drive, compares it with the Windows file structure, and isolates any discovered mismatches in an effort to repair potential problems. In effect, VxMS enables Norton to compare file systems with the raw data on the disk. Differences are immediately suspect.

For example, say Windows Explorer shows five files in a directory, whereas VxMS shows 10. Clearly, the additional five files are cloaked. Norton sends the suspicious files to Symantec for analysis, eradication occurs during reboot, and the discovered rogue is removed from other systems worldwide as a result.

Trend Micro takes a different approach. Using experience gained in its security labs, the company developed a complete library -- the RCM (Rootkit Common Module) -- to replace the Windows APIs, says Geoff Grindrod, solution product manager at Trend Micro. According to Grindrod, the library includes double encryption to avoid spoofing, and its proxy for API calls is constructed as a special kernel module.

With the RCM, the system sees hidden processes, hidden registry keys, and hidden files. As the RCM has matured, it has been integrated into more and more Trend products and is now a core component of anti-spyware and other Trend Micro products, Grindrod says.

Discovering rootkits, however, is only half the battle, as excising them can result in its own set of problems; "Rootkits are so imbedded in the operating system," Mandiant's Butler says.

"Plus, we're seeing firmware attacks and survivable rootkits installing themselves in the BIOS. Removing rootkits can also make the system unstable while it's running."

Admins should be aware of the implications of rootkit removal before lunging headlong into the endeavour, says Ron O'Brien, senior security analyst at Sophos, one of the first security vendors to offer a rootkit removal tool.

"Rootkits are not 'bad', but they have developed a reputation for being bad," O'Brien says. "They are really just a form of hidden files" that may have legitimate uses. Ripping rootkits out before establishing their purpose can prove detrimental to overall system health, he adds.

Coping with an evolving threat Despite advances in prevention and removal, Steve Manzuik, senior manager of security engineering and research at Juniper, sees no end in sight to the rootkit threat. In fact, Manzuik believes that, Joanna Rutkowska's work on the Windows kernel, and Microsoft's resource protections for 64-bit Windows Vista, are "making it more difficult for both attackers and vendors".

Manzuik sees that current approaches to rootkit discovery and removal are beginning to fail despite improvements in Windows security. Factor in the lag time before Vista protections are widely deployed, and you have a perfect breeding ground for rootkit innovation. For example, Manzuik points out that some rootkits can now bypass the security sandbox. They detect they are in the sandbox and lay low, effectively tricking the system into thinking they are legitimate apps.

Mandiant's Butler, however, believes that Vista protections will have an impact. Not only will the protections make it more difficult for rootkit authors to break in, Butler says, but it will also require "another separate effort to conceal themselves and maintain their presence".

Manzuik and Butler do, however, agree on the importance of strict user access policies. Both view rootkits as further evidence against giving users admin-level access to systems -- especially at smaller organisations, where the practice is often promoted as a cost-cutting necessity.

"The culture in smaller companies is that they will only call the IT guys if they can't figure it out themselves, which leads to most users having admin rights on machines," Manzuik says. Any organisation employing this policy -- regardless of its size -- will be compromised, Manzuik says.

Because of this, Manzuik believes policy should figure foremost as a means for protecting systems against rootkits: "Without buying special technology, [most organisations] can deal with the majority of the threats with proper security policy and management."

That said, recent attention paid to rootkits has resulted in a raft of discovery and removal tools, both free and host-based, including IceSword, RootkitRevealer, F-Secure's Blacklight, and Sophos Anti-Rootkit.

Over time, these functions will be integrated into enterprise-grade antivirus and host-based security solutions. In the meantime, however, most organisations remain unprepared -- all the more troubling, given that opportunism is pushing rootkit know-how deeper underground, out of the IT community spotlight.

In the past, innovations in the art of hiding rootkits was shared in newsgroups and posted to community Web sites. The financial upside of having rootkit knowledge, however, is changing that, Mandiant's Butler says. Those who uncover new approaches may take their discovery to a security company as their calling card to obtain a job.

More disturbing, however, is the amount of money malware authors are willing to pay for new techniques. And with both sides of the divide doling out cash for the latest innovations, rootkit development is clearly becoming a lucrative pursuit -- one that leaves most organisations in the lurch, unaware of what's coming.

To reduce the probability and impact of rootkit infection, organisations should take the following proactive steps:

1. Do not ignore the threat and do not rely entirely on deployed antivirus or host security systems.

2. Develop and implement a plan to analyse the current state of all systems.

3. Establish proactive procedures for maintaining an expanding defense against rootkit installation attempts, including policies and end-user communication.

4. Create a plan to analyse any infections that occur.

Kevin Mandia, president and CEO of Mandiant, notes two essential capabilities for discovering rootkits in the enterprise: "the ability -- tools and technology -- to detect the rootkit's network traffic via network security monitoring; and the ability to perform a sophisticated host-based console review, [making sure you're] able to conclude that the host-based review did not identify the process that is generating the suspicious network traffic."

For organisations looking for added protection against rootkits, enlisting the assistance of security experts is a worthwhile idea. Mandiant, for one, provides incident-response software and professional services, enabling organisations to tap experts when developing risk-mitigation strategies and when responding to incidents to determine what data was lost and how the attack entered and evolved.

Unfortunately, too many organisations will wait until they have lost data and have exposed themselves to great financial harm before taking steps. Don't be one of them.

[sidebar] McAfee puts Rootkit Detective on the beat

By Matt Hines

McAfee has begun offering a new application called Rootkit Detective, designed to detect and remove dangerous rootkit attacks.

The software will also help endusers ward off the threats, as well as funnel new intelligence into the company's ongoing research operations.

Following in the footsteps of SiteAdvisor -- the free Web site security program acquired by McAfee in April 2006 that warns users about potentially dangerous sites and search results -- company officials said that the new tool will be offered at no charge from its Web site via download, with benefits for both end users and its researchers.

The freeware program promises the ability to find and remove so-called rootkits -- self-cloaking malware attacks that install themselves as kernel modules or drivers and are most often used to hide other types of threats such as keyword-logging programs -- and send data about the attacks that are discovered back to McAfee.

As greater numbers of PC users have employed more sophisticated anti-malware tools in recent years, hackers have rushed to adopt the rootkit model as a means for circumventing antivirus systems and keeping their attacks hidden on people's computers.

According to the most recent estimates released by McAfee, more than 7325 new rootkit variants were discovered in the first half of 2007, a dramatic twice the number of rootkits the company's researchers uncovered during all of 2006.

Rootkit Detective specifically promises to find hidden kernel processes and registry entries, as well as remove them when a user reboots their system. The tool also claims the ability to test the integrity of a PC's kernel memory and track any modifications that might also highlight rootkit activity.

As part of a beta program, Rootkit Detective -- which was developed within McAfee's Avert Labs -- has already been downloaded by more than 110,000 users, including businesses and consumers, company officials said.

"Dealing with rootkits will always be an arms race; the whole process is a game of challenge-and-response between the hackers and security community, and as the authors have advanced the complexity of their attacks, we need to continually update our own technologies to keep up," said Joe Telafici, vice president of operations at McAfee Avert Labs. "We started putting rootkit detectors into our products in 2006, and this is the next stage in advancing our detection technologies."

While most rootkit-fighting programs use what Telafici labelled a "tainted view" approach to finding the attacks -- that is, comparing results of system calls to the kernel to look for potential issues -- Rootkit Detector uses a variety of means to find hidden processes and registry keys that might evade such tactics, he said.

The approach is also particularly effective at helping McAfee find new rootkit variants, based on the detailed manner in which it monitors a machine's kernel and memory, according to the researcher.

Telafici goes as far as to claim that Rootkit Detector can find and remove every known rootkit reported to its researchers thus far.

"The bad guys are spending a lot of time trying to hide their work from simpler tools, but we can still see these programs making their calls, and we've already used the tool to find several new variations that we weren't previously aware of," he said.

"We use a variety of means to detect processes, files and registry keys that might otherwise remain hidden, and to bypass cloaking techniques employed by the rootkit authors."

In passing out Rootkit Detective to consumers and businesses free of charge, McAfee is hoping that, as with SiteAdvisor, people will actively use the application to submit virus samples to Avert Labs.

After analysing any new attacks, McAfee will create a signature for any rootkits it tracks and channel that information into its other client security products.

"Gathering information I this manner is a very effective way for us to get a handle on threats we haven't seen before, and it should get new kits flowing in that we can begin researching to adapt to throughout our product lines," Telafici said. "It's great to be able to offer something valuable for end users that can really help protect them, while allowing us to find new attacks and develop technologies to address for our customers."

The Rootkit Detector launch underscores recent efforts by antivirus providers to launch technologies aimed at fighting the most complex, cutting-edge attacks being aimed at users by hackers.

Rival Symantec introduced a beta version of its Norton AntiBot program, which is designed to thwart the growing problem of PC-hijacking botnet attacks. However, unlike McAfee's latest offering, AntiBot is a for-pay product that will retail to consumers for less than $US30.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about AvertBillionBMG Lab TechFirst SecurityForrester ResearchF-SecureHISJuniper NetworksMcAfee AustraliaMicrosoftNortonPLUSSecurity SystemsSonySophosSymantecTrend Micro AustraliaVeritasVeritasVIA

Show Comments