Securing Microsoft: A long road

(continued from previous page)

"That was a very painful experience," said Snyder, who at the time was part of Microsoft's security outreach team, but has since left Microsoft and now serves as "chief security something-or-other" for Mozilla. "It was pretty intense."

Slammer was followed by Blaster and others. Snyder recalls a sense of dread that permeated the team during 2002 and 2003.

"It kind of seemed at that point like it would never end," Snyder said.

But things did shift. Mass mailers gave way to the rise of botnets--networks of computers taken over by hackers for the purpose of sending spam, harvesting credit card information or clicking on online ads. Widespread attacks fell out of favor with criminals who found there was more money to be made from targeted attacks. The moves forced Microsoft, again, to shift its approach as security threats no longer merely opened customers to the prospect of headaches and lost productivity, but also to financial loss.

As threats became less a random crisis and more a fact of life, Nash realized the security team needed its own space. "It used to be we'd take over a conference room and people would say, 'We need the conference room,' and we'd say 'No, we need the conference room.'"

Although communications is an important part of emergency response, Nash decided it was necessary to build two "war rooms," allowing the engineering team to brainstorm separately from the employees who were communicating with customers and the media. The Microsoft Security Response Center was completed in June 2005. A door connects the two rooms, making it easier for people in both areas to get together when need be.

Microsoft also began to realize that it simply couldn't afford for everyone in the company to learn security lessons the hard way. It needed more people to get exposure to the threats that were out there.

The need for more dialogue with the security community prompted Snyder to suggest the idea for Blue Hat, an internal Microsoft conference where hackers would present in front of the company's engineers. The idea was controversial at first, with not everyone thinking it was such a good idea to put Microsoft's engineers face to face with the folks they blamed for many of their headaches.

Among those initially opposed was then-chief of Windows Jim Allchin, who didn't like the idea of having to sit face to face with the people who poked holes in the products his team created. Nash recalls Allchin saying to him, "Let me get this straight, you want the people hacking us and telling us the problems in our system and you want me to listen to them." Yes, Nash said, that's exactly what we want you to do.

Uncertainty among outside researchers
The outsider researchers, too, were skeptical about Microsoft's motives and commitments.

But, it proved to be a hit in both camps. Blue Hat, which was first held in March 2005, is now a twice yearly event, with the most recent one taking place over a two-day period in September. As usual, Microsoft's engineers were confronted by hackers showing a range of techniques that can be used to attack Microsoft's products. In perhaps its most confrontational invitation of the year, Microsoft invited the team from WabiSabiLabi, a group that operates an auction site where people can bid on vulnerabilities, much like collectors bid for trinkets on eBay. The enterprise is happy to sell to vendors who can patch the hole, or to people who might have other purposes in mind.

Although the presence of WabiSabiLabi's Roberto Preatoni at Blue Hat in September was unnerving, Cushman says that it's important for Microsoft's engineers to understand the current threats. (In November, Preatoni was arrested in Italy in connection with a spying investigation at Telecom Italia, where he previously worked.)

"I don't want every team to have to learn those painful lessons first-hand, but yet I want each of them to get that visceral understanding of how important this is," Cushman said. And there's nothing like having a hacker come in, he said, before correcting himself, "having a security researcher come in and demonstrate vulnerabilities in your product to bring that lesson home."

More recently, the company has started an exercise called "Defend the Flag," in which IT pros and security newbies get a day of training on setting up a Windows network before having to build and protect one themselves.

"If the network you've set up and are defending (gets) compromised because of misconfiguration or some vulnerability, you are going to remember that," Cushman said.

While many of the lessons surround ways that Microsoft needed to do more or move faster, one of the strongest lessons for Shostack was a story he heard at a previous Blue Hat about the need to proceed with caution. At the event, a colleague talked about Microsoft trying to prevent bitmap art exploits by more narrowly defining what could be in such a file. In trying to shore up security, Microsoft had also broken some files, which meant that companies that used a bit-mapped logo in their invoices couldn't print bills.

"Once a system administrator has gone though that experience, they become much more hesitant to patch," Shostack said. "It's very important when we build an update that it won't break anything."

Stories

Day 1: From pain to progress
Remond's security practices have been transformed since threats like Slammer and Blaster first wormed their way onto the scene.

Day 2: Inviting the hackers inside
Aiming to be more open, company reaches out to the security research community it once kept at a distance.

Day 3: Emerging security threats
Forget widespread worms. Nowadays, limited-scale threats like targeted e-mail attacks are causing the most concern.

Beyond Binary Blog

Day 1: Inside the war room
After years of having to scramble whenever an outbreak hit, Microsoft builds adjoining situation rooms to coordinate its response efforts.

Day 2: Off to the Limo Races
In what might seem an unlikely pairing, Microsoft employees and security researchers team up to go on a scavenger hunt through Seattle.

Day 3: Meet the bug hunters
One talks a mile a minute, another dresses like a bug. Meet some of the people who have helped lead a massive culture change at the company.

Related special reports

Microsoft's lessons from the desktop

Microsoft meets the hackers

'MSBlast' echoes across the Net

Related stories

Microsoft gathers hackers in Redmond

Microsoft puts key security under Windows umbrella

Microsoft gets good reception at Black Hat

Gates: End to passwords in sight

Bug hunters, software firms in uneasy alliance

Microsoft wants to meet more hackers

Is there method in Microsoft's security buys?

Microsoft's blast from the past

Gates: Security is top priority

Photo galleries

Inside the war room
Painful episodes lead to the creation of a security response center, where teams take on the task of hunting bugs and keeping customers informed.December 3, 2007

The bug hunters
Just who are the people charged with the task of keeping code secure at Microsoft? They're risk takers, whether donning silly costumes or swimming with sharks. December 5, 2007

Credits

Editors: Anne Dujmovic, Mike Ricciuti
Design: Andrew Ballagh
Production: Kendra Dodds