Rootkits get better at hiding

A new Trojan horse is so good at hiding itself that some security researchers claim a new chapter has begun in their battle against malicious-code authors.

The new pest, dubbed "Rustock" by Symantec and "Mailbot.AZ" by F-Secure, uses "rootkit" techniques crafted to avoid the detection technology used by security software, Symantec and F-Secure said in recent analyses.

"It can be considered the first born of the next generation of rootkits," Elia Florio, a security response engineer at Symantec, wrote in a blog late last month. "Rustock.A consists of a mix of old techniques and new ideas that when combined make a malware that is stealthy enough to remain undetected by many rootkit detectors commonly used."

Rootkits are considered an emerging threat. They are used to make system changes to hide software, which may be malicious. In the case of Rustock or Mailbot.AZ, rootkit technology was used to hide a Trojan horse that opens a backdoor on an infected system, putting it at the beck and call of an attacker, according to Symantec.

In their continuing race with security software makers, the creators of this latest rootkit appear to have looked closely at the inner workings of detection tools before crafting their malicious code, said Craig Schmugar, virus research manager at McAfee, which calls the pest "PWS-JM."

"Security companies are trying to stay one step ahead of the bad guys, but the bad guys already have the technology that is available from the security vendors," he said. "A number of techniques have been combined to really strengthen and harden this particular threat. They have done a pretty good job at closing all the doors."

The mixture of cloaking methods makes Rustock "totally invisible on a compromised computer when installed," including on a PC running an early release of Windows Vista, Symantec's Florio wrote. "We consider it to be an advanced example of stealth by design malicious code."

To avoid detection, Rustock runs no system processes, but runs its code inside a driver and kernel threads, Florio wrote. It also uses alternate data streams instead of hidden files and avoids using application programming interfaces (APIs). Today's detection tools look for system processes, hidden files and hooks into APIs, according to Florio's post.

Additionally, Rustock defeats rootkit detectors' checks for the integrity of some kernel structures and the detectors' efforts to detect hidden drivers, Florio wrote. Furthermore the SYS driver the rootkit uses is polymorphic and changes its code from sample to sample, according to the blog posting.

Still, chances of people being attacked by this rootkit and its malicious Trojan horse payload are slim, experts said. "People are blogging about it not because it is highly prevalent, but because of the challenges it poses to existing rootkit detection tools," Schmugar said. Symantec and F-Secure also both state the threat is not widespread.

F-Secure updated its BlackLight rootkit detection tool that can detect current versions of the pest, the company said in a blog. Symantec and McAfee are still working on tools to detect and remove rootkits from computers.