Defense in Depth

In February of 2005, a Miami man sued Bank of America for not adequately protecting him against a $90,000 fraudulent wire transfer to the Parex Bank in Latvia. Joe Lopez was the first online user to sue his financial institution for not protecting his assets from a computer hacker.

Lopez, owner of a computer and copier supply business, accused Bank of America of negligence and breach of contract for not alerting him in advance to the existence of a piece of malware known as "Coreflood" prior to April 6, 2004, when the alleged theft took place.

Shortly after the wire transfer occurred, a sum of $20,000 was withdrawn from Parex by unknown individuals, according to the complaint filed in court. The remaining $70,000 was, however, frozen by Latvian banking authorities. Bank of America has since settled this case; neither side has revealed the terms.

"I had probably heard the news about Joe Lopez, but (until recently), I hadn't thought twice about the whole Coreflood episode of a few years ago," admitted Joe Stewart, director of Malware Research at SecureWorks, when I spoke to him at last summer's Black Hat conference in Las Vegas.

In particular, Stewart recalled hearing that the U.S. Secret Service had found evidence of Aflood or Coreflood on the Lopez computer.

"The Secret Service actually named Coreflood. That was very surprising. Normally, we don't get the final tally. We don't know who's account got stolen. It's very unusual to actually have a victim that is public, and everybody knows exactly what (was) taken."

Unlike a lot of bots and botnets, most of which exist primarily to relay spam, Stewart said Coreflood has a different agenda: "Its goal is to steal the data directly from users." The much more popular Storm botnet, he said, is more of a nuisance. "Coreflood has a real financial impact for people like Joe Lopez."

Who's behind Coreflood? Stewart declines to say, but in an interview in The New York Times, he suggested that the gang responsible was based somewhere in Russia. He would not tell me the name of the group because of ongoing criminal investigations.

In this video, Stewart talks about what first drew him to study the Coreflood botnet.

When Stewart heard about Lopez, he renewed his research on the Coreflood. With the help of Spamhaus, an antispam organization, Stewart and SecureWorks were able to gain cooperation from a Wisconsin-based provider of one of the command and control centers for the botnet. What he found was not only the bot's source code but also 50 gigabytes of compressed data, searchable in a MySQL database.

Within that database were 378,758 unique bot IDs over a 16-month period. There, for everyone to see, was the time-stamped life cycle--from infection to removal--of each compromised computer. Stewart found the average to be about 66 days.

The graph shows how one state policy agency was infected with Coreflood from April 2007 through January 2008.

(Credit: SecureWorks)

Apparently, Coreflood would enter a network via a drive-by browser exploit, download a copy of the installer, then run PcExec, a legitimate Windows administration tool available from Microsoft.

"It could happen to anybody," Stewart said, "any user who happened to go to the wrong site." If the user also happened to be on the corporate network when that happens, the bot is then able to take advantage of that structure and is able to be a threat to everyone on that network.

"So it's not so much a targeted attack," Stewart said. "But I think they have intentionally set a trap for the domain administrator and are leveraging that in order to have access to the entire company."

Later, the criminal gang responsible for the attack can find out which company it has infected by looking into the registry of the infected computer. "They pull out of the registry a separate request to say who is the registered owner the Windows license. They ship that information back up to the botnet controller."

Just looking at that one C&C server in Wisconsin, Stewart estimates that the gang responsible has infected more than 35,000 domains. It may sell those Web mail accounts to a spammer, because spammers love Web mail accounts. But over the years, Coreflood seems to have targeted only banks. Stewart knows this from the forensic evidence he's collected.

In this video, Stewart talks about digital forensics and what it can tell us about botnets such as Coreflood.

Within the 50GB file, Stewart was able to discern how the thieves culled the data. He said they run a test script against that data that will log via a proxy into the bank using the credentials captured, say, by a keylogging application. The Coreflood script will then capture the HTML data on the post-log-in page.

In most cases, that page also contains the account's bank balance. This is so that after running the test, the hackers have a picture of what the highest dollar amounts are, he said.

"I don't know whether they steal from all of them. We don't have access to the accounts; the bank is not going to tell us how much was stolen out of any given account," he said. "We're not going to get that information, but we know they're actively logging and checking accounts to collect the balance data. The only reason (the script) can see that data is to target the biggest accounts first."

Coreflood does not take a screenshot, Stewart said, but rather scrapes the text out of the HTML. "When they run these tools, it leaves a log file behind, and all the post log-in (data)...are saved in that directory. So we have all of the account balances. So we can parse out what everyone's balance is and see actually how much (the thieves) had access to at any one institution."

In this video, Stewart talks about why Coreflood has been around since 2001, yet hardly anyone has been talking about it.

The problem is that Coreflood has been around since 2001.

"It's unique in that's been around for so long," Stewart said. Moreover, it's unusual that it seems to have been maintained by the same group, "not something that's been sold to another group," as is the case with some botnets.

The way it's managed to evade detection, Stewart said, is that it hasn't really crept high on anyone's list of botnets. "It's not on anyone's radar." Yet it's managed to seriously impact some enterprises that use Windows domains. In companies that have been hit, every employee is potentially sending everything they do back to these guys in Russia.

"To me, (Coreflood) is far more insidious because it doesn't get the attention," said Stewart. Unlike Storm, Coreflood is not constantly in your face. "You're not seeing new social-engineering campaigns every week, not seeing a new news article about it every week talking about all the great innovations the peer-to-peer thing has now. It's been quiet, and just does a few things, and tries not to garner any attention."

So the story of Lopez is significant. It's a tangible event about how online criminals are actually affecting people. It illustrates how much money got taken from an actual bank account, and the real impact on the victim's life. Unfortunately, there are many more botnets--and many more victims to talk about.

What if you wanted to build your own botnet to act as a spam relay or to launch a denial-of-service attack against an organization or a country? "It's actually a lot of work," says Joe Stewart, director of malware research at SecureWorks.

I had a chance to talk with Stewart at this year's Black Hat security conference in Las Vegas where, in a talk, he provided insight into the inner workings of one botnet, the Storm worm botnet. Using unpackers, debuggers, and decompilers, Stewart was able to dissect the rogue network and learn how it works and why Storm remains so resilient when other botnets simply fail over time.

Joe Stewart of SecureWorks at Black Hat Las Vegas 2008

(Credit: Robert Vamosi / CNET)

Botnets, whose combined computing power can equal that of a large supercomputer, are organic, yet they only evolve when they need to, such as after they've been discovered and shut down, Stewart said. But he said anyone wanting to copy a successful botnet like Storm would simply be wasting their time. While all the coding tricks used to make Storm successful are available on the Internet, it's combining them that's the trick.

"How you are going to make all that work for your specific needs? It's pretty complex," he said. "The person who developed Storm did it over a long period of time. They didn't start out with the peer-to-peer program (as used today); they started out with something much simpler. They then made small modifications. A lot of hours have been put into it."

Storm's structure
A basic botnet would includes a Command and Control (C&C) server contacted to thousands of compromised desktop computers worldwide. Were that always the case, botnets could be taken down quickly by simply finding and shutting down the C&C server. Storm's approach is more nuanced and layered. Top level is a Command & Control server running Apache (presumably somewhere in Russia). Next level is a server running a Nginx 0.5.17 proxy; this server is designed to hide the Apache machine from view. At the third level are a couple more Nginx 0.5.17 proxies used to hide the master Nginx 0.5.17 proxy from view. Sitting at the fourth level are public nodes that act as reverse proxies leading back to the controller and perform as fast-flux name servers. Fast flux means that a hard-coded URL can be sent out with the bot code, but where that URL resolves changes.

The final level is composed of thousands of compromised computers worldwide. Stewart says that Storm starts out infecting a computer with a dropper. Right now the preferred infection process is via an e-mail link, but this might change to a peer-to-peer process. However infected, the initial click by the end user installs a rootkit which, in turn, reaches out to the EXE file from a fourth-level supernode. Once infected, the compromised computer and supernode trade the infected desktop's IP information. This information is sent to a third-level supernode proxy as pert of its mapping operation. At the third level it is also compressed and encoded for obfuscation, then sent on to the second level proxy, and finally to the top level server.

Overnet/eDonkey
At the second and third levels, the Nginx proxies listen for Overnet/eDonkey peer-to-peer Internet traffic. Overnet/eDonkey was a popular peer-to-peer network application until it was shut down by the Recording Industry Association of America. While the service is gone, the code still exists. What botnet operators like most is Overnet/eDonkey's distributed nature; it lacks a central peer list. Thus, each of the nodes keeps only a small list of neighboring peers.

This decentralized network is what Stewart and many other experts say is the key to Storm's resilience.

And it almost proved to be Storm's undoing. Overnet/eDonkey is still used for file-sharing, so in Storm's view there is a lot of bogus traffic out there. To better distinguish its traffic from other traffic, Stewart says Storm uses the Kadamlia distributed hash table (DHT) and its C&C servers listen only for predictable MD4 hashes. Those hashes are derived from a simple checksum algorithm that includes IP address and the port used. Authentication is accomplished through a 4-byte challenge and response.

The predictable hashes also have a positive effect for researchers, says Stewart: If a given peer doesn't know the location of the specific node you're searching for, the known peer will provide you with a list of peers closest to what you asked for. And, because the Overnet/eDonkey supernode peers all broadcast their presence, Stewart and other researchers can walk all the nodes in a network to get a fairly accurate count of the botnet's size.

Not perfect
Lately, though, Storm has been evolving yet again. This time it's isolating its network further from the general Internet traffic by encrypting packets using an embedded key and simple XOR. It also has been changing its initial infection packing or compression process. The outer layers change every 10 minutes, while the interior bot code changes packing more on the order of once a month. Neither the packing nor the encryption have so far proven defeating to security researchers.

However, one downside to encryption is that Storm's handlers could now segment parts of their network--that is, they could rent or sell off pieces of the botnet to others. Although speculation around segmentation has been widespread, Stewart says he has not observed it.

In addition to Stewart's research, see Brandon Enright's report for another detailed look at the structure of this venerable botnet.

Spammers will do just about anything to get their e-mail through corporate and desktop filters. According to MessageLabs, they're now using Google Docs, a perfectly legitimate way to publish to the Web. Only what they're publishing is the same old wares--this time, it's enhancement pills. This week I talked with Matt Sergeant, senior anti-spam technologist with MessageLabs, who told me how they they've tracking one Google Doc since May 8, 2008.

Later in the conversation, Sergeant talks about the resurgence of Storm. Only a few weeks ago, MessageLabs reported a notable decrease in computers infected with the Storm botnet.

Below is a transcript of part of my interview. The entire podcast can be heard here.

Matt Sergeant: What's happening with Google Docs is that Google Docs is a way to publish your documents online. So, for example, word processing documents and spreadsheets and so on, and much like if you were using Microsoft Word you can embed links within those documents. What this does for the spammers is it allows them to effectively publish online a Web page on hosting sites such as Google that has all the bandwidth in the world for hosting it, and it's also a Web site that is never going to get blacklisted by anyone because nobody would be stupid enough to blacklist Google. So in effect, for the spammers this is a human shield effect. They can host their information and links online on a very stable source of bandwidth and links, and not worry ever about it being taken down or blacklisted.

Me: When did you first see this happening?

Sergeant: The first one that we saw, which showed on our radar in extremely small numbers clearly as a test by the spammers, was on May the 8th. So I guess that's about two weeks ago now.

Me: Have you contacted Google?

Sergeant: We've contacted Google, and also there's a link at the bottom of each one of the documents that Google publishes online that says, "Report this as spam." We clicked that link and I imagine anyone else who got the e-mail clicked that link as well. Unfortunately, Google has proved themselves to be quite slow at tackling this kind of abuse. Weeks later this document is still available online despite the reporting as spam.

Me: When you say that Google has a history of this can you site another example in recent memory where they've been slow to act on spam like this?

Sergeant: Generally, yeah there's a couple of different issues that we see in spam with Google. The first and very obvious one is spam directly from Gmail accounts, often that's the Nigerian spammers who are sending out these offers of millions of dollars where there is in fact no money. By most people's standards, Google tends to be quite slow at shutting down those accounts, whether it be an account that's actually an e-mail or just a drop box account for people to reply to. So those accounts seem to stay active for longer than if they were being hosted somewhere else for example. The other thing we see with Google is redirector links, so they have these links on their Web site which allow anyone or just about, but obviously mostly the spammers to have a link that looks like it's going directly to Google, but in fact after you've visited Google it redirects you to the actual spammers Web site. These redirectors are quite common on loads and loads of Web sites out there, but obviously again they're gaining advantage from Google of all the bandwidth and unblock ability of the Google Web site.

Me: So give me an example of what we would see if we went to the spammers website, what sort of, where is it being hawked or Malware being served up.

Sergeant: In the example that we saw on May the 8th it was a very simple pills scam or a pills Web site. So the e-mail came in with a link to Google Docs and very little of a text in the e-mail itself. They're very hard to block because there was very little to go on regarding the contents of it. When you went to the Google Docs Web site you saw much more information about the pills available for sale and the prices and so on, and almost every bit of text within that was a link which took you to the spammers drop Web site, which is where you would actually go if you wanted to purchase some of those pills.

While Operation CyberStorm is intended to improve our ability to defend against a foreign cyberattack, the Air Force is talking openly about our ability to launch a preemptive attack in cyberspace.

In the May 2008 issue of Armed Forces Journal, Col. Charles W. Williamson III wrote that "America needs a network that can project power by building an af.mil robot network (botnet) that can direct such massive amounts of traffic to target computers that they can no longer communicate and become no more useful to our adversaries than hunks of metal and plastic. America needs the ability to carpet bomb in cyberspace to create the deterrent we lack."

He argues, "The time for fortresses on the Internet also has passed, even though America has not recognized it. Now, the only consequence for an adversary who intrudes into or attacks our networks is to get kicked out--if we can find him and if he has not installed a hidden back door. That is not enough."

He concludes: "While America must harden itself in cyberspace, we cannot afford to let adversaries maneuver in that domain uncontested. The af.mil botnet brings the capability to help defeat an enemy attack or hit him before he hits our shores."

"Although it's hard to prove it," said Yuval Ben-Itzhak, CTO at Finjin, "I believe the cyberspace is already in use by various governments for intelligence purposes. The disclosure that the Air Force plans to have offensive cybertools should not surprise us since many systems rely on the Internet to operate/communicate." He added that someone will also need to make sure these systems can be protected when needed.

That's a sentiment echoed by Dancho Danchev, who offers some insight on ZDNet. Among his observations is that these systems can be spoofed or otherwise fooled. For example, attacks against the U.S. may appear to originate in a country that the enemy wants us to DDoS (perhaps for them).

Over on F-Secure, a poll of readers worldwide showed on Thursday that nearly 70 percent of the respondents feel the U.S. should not build its own offensive botnet.

On Thursday, MessageLabs reported in its April Intelligence Report a marked decrease in the number of malware links connected to the Storm botnet. "It's not too often that a security company says that things are getting better," said Mark Sunner, Chief Security Analyst.

At its peak, Sunner said, the Storm botnet resided upon one million computers worldwide. That number has since come down to between 85,000 IP addresses at the end of April. He said that over the last eighteen months Storm has been constant, and never decreased according to MessageLabs research. "Other security companies have reported decreases in the past," he said because of different methods of studying the botnet, "but this is first decrease we've seen."

He credited the most recent patches from Microsoft with the decline. He said that in the weeks following the most recent Patch Tuesday there was a sharp drop off.

Given that the creators of Storm managed and maintained a constant flood of variations for more than one year, it's a little odd that they would just take their money and walk away. Sunner said that they are seeing an increase in Srizbi, named for the one of the Web sites from which is downloaded. A Trojan, Srizbi uses rootkit technology to hide on an infected machine but, like Storm, it is also known to relay spam.

On Wednesday, the SANS Internet Storm Center and others published details about the massive SQL-based Web attack that occurred over the weekend. The attack, says SANS, is similar to a smaller SQL-injection attack seen in November. At least 70,000 sites were compromised in a short period of time, leading some to speculate this was an automated attack.

From logs files, the attack code appears to exploit a variety of SQL injection vulnerabilities existing on Web sites using Microsoft SQL or Microsoft IIS. On the vulnerable sites, malicious JavaScript is injected into all variable character fields and text fields in the SQL database such that when visitors hit the site, their browsers, if vulnerable, are then redirected to another domain--in this case, us8010.com.

Roger Thompson, chief research officer at Grisoft, identified one of the exploits served at the malicious server as taking advantage of MS06-014, a Microsoft Data Access Components vulnerability that Microsoft patched in September 2006. He also noted that "this domain uc8010(dot)com was registered just a few days ago (Dec 28), and yet, at one point Google showed script injections pointing to it were showing up on over 70k domains." Yet by January 5, most of these domains had already been cleaned.

What's interesting about this attack, aside from its automation, is that the SQL injection script is given in terms of a CAST statement, code that converts one data type to another. Ryan Barnett has provided a decoded version of this attack.

Barnett suggests that to protect against this attack a Web site should be front-ended by an Apache proxy and then back-ended by ISS or MS-SQL. SANS says other methods, such as blocking CAST statements, would also be effective.

Last week, the FBI announced the end of the second phase of Operation Bot Roast, an ongoing investigation into botnets, and the criminal activity associated with them. I recently asked Dr. Jose Nazario of Arbor Networks where in the world the bot herders, the people who control the botnets, might be. Here are some excerpts:

We see a few major groups. We see Americans and Western Europeans often interested in using the botnet to make money either directly or indirectly by selling services, or stealing information from those botnets to sell and use credit card information bank information, etc.

There are some botnets out of South America, but mostly South America seems dominated by the Brazilian, what folks used to call the banker Trojan, the browser helper object that steals information right out of the browser from banks from online banking or e-commerce transactions. Some of the more high-profile botnets we've dubbed TeamUSA and Peruvian Power. These have been long running and relatively successful. But they're not exactly household names.

The botnet community is also taking off in the Russian language part of the Internet. Lately I've been watching a lot of DDoS attacks come out of Russia, commanded by Russians. Possibly for pay, as retribution, or as punishment to those who try an stop some of the other illegal activities, such as fraud and theft.

I have been tracking lately Russian DDoS bot code run by different groups. The code itself is bought and shared between them. One of the big ones is a code base called Black Energy. The author is a Russian language speaker who offers his help files and other things in the Russian language and sells it on the Russian language forums anywhere from $40 on up. Black Energy is strictly a DDoS botnet

We have watched some botnets from China but I don't see a whole lot of botnet activity coming out of there.

You can read more of Nazario's comments in this Security Watch column. And you hear more of my interview with Dr. Nazario in this Security Bites podcast.