Securing Microsoft: A long road

(continued from previous page)

Although those early IE flaws awakened Microsoft to the dangers posed by the scale of the Internet, it took several more waves of attacks to fully form the company's security strategy.

The arrival of Melissa, on July 23, 1999, knocked down one of the core pillars of Internet security at the time: by avoiding e-mail from unknown senders, one could avoid most attacks.

"They broke the trust between the user and his address book," Stathakopoulos said of the worm's authors.

Mass mailers like Melissa and I Love You were largely annoyances, though many companies had their e-mail systems overwhelmed by the sheer number of messages being sent by the viruses. But the threat became stronger as mass mailers started carrying payloads designed to attack, a period Stathakopoulos calls the era of "weaponized" vulnerabilities.

Two major attacks, Code Red and Nimda, hit in mid-2001, striking Microsoft's corporate customers hard and becoming a major headache for not only the security team, but also for the company's top brass.

In the wake of Code Red and Nimda, Gartner issued a report saying companies should "immediately" consider moving away from Microsoft's Internet Information Server product and over to rivals. That was another painful lesson, Cushman said. "Every single person on the IIS team took it personally that there was an outbreak." Cushman said the team felt the report was misreported, but it also led the unit to take new actions, such as bringing in Microsoft's top security experts to help train the members in writing better code, followed by a "bug bash" aimed at rooting out bad programming from the product.

In late 2001, Gates began drafting Microsoft's response, in what ultimately became his now infamous January 2002 Trustworthy Computing memo.

"When we face a choice between adding features and resolving security issues, we need to choose security," Gates wrote in his missive to employees. "Our products should emphasize security right out of the box."

But not everyone took the Microsoft chairman at his word.

"At the time I thought it was a PR initiative," said Adam Shostack, who was then working for Zero-Knowledge Systems in Montreal and is now a senior program manager at Microsoft, working on the company's secure development approach. Shostack said he changed his mind in the ensuing months as Microsoft followed up Gates' words with action.

Microsoft stopped virtually all Windows development work, and for a month all of its engineers focused on security-related work.

It wasn't a demonstration of rigorous coding practices nearly as much as it was a show of brute force designed to attack the problem at its source.

"It was 'take all the engineers and have them each go review code,'" Thomlinson said. "It was kind of the infancy of security engineering."

Even so, there was still a culture inside the company that attempted to play down the bugs to the outside world.

"We used to get the reports and say, 'That's not a security bug,'" Stathakopoulos said.

But when Nash was appointed to head up the security team in late 2001, he came in with a different approach: fess up and tell the world about potential security problems. "He said, 'No, you've got to be transparent (with the outside world)," Stathakopoulos said, recalling that his team looked at Nash as if he were insane.

"People already think our products are bad, and if we start talking about those issues more and more, people will think we are horrible," Stathakopoulos said he argued at the time. But Nash persisted, arguing that the company might initially take some added lumps, but over time the company would come to be respected.

Looking beyond the software industry
In building Microsoft's security response apparatus, Microsoft had to look beyond the software industry. "No one had had to figure this out before us," Nash said. One of the companies that Microsoft used as a guide was chemical maker DuPont. While not an exact parallel, Microsoft studied how DuPont reacted to train derailments.

Among the lessons it learned was the fact that emergencies occur at all hours, so Microsoft needed to be staffed more often. "It wasn't quite banking hours, but it wasn't 24 by 7," Nash recalls of the system in place at the time.

Katie Moussouris, who worked for AtStake for a number of years before joining Microsoft, said she recalls a slow but noticeable shift in Microsoft's attitudes and practices.

"You could almost see the aircraft carrier turning," she said. "It took a lot of miles and a lot of time, but now it's got the power of the aircraft carrier behind it," said Moussouris, a security strategist for the Security Engineering and Communications Group.

While the effort would eventually pay dividends, it wasn't enough to head off the era of big worms that kicked off with Slammer in January 2003.

Stathakopoulos recalls getting a call at 3 a.m. from Symantec's Vincent Weafer, saying that a known bug in SQL Server had been exploited. A bit groggy as he answered the phone, Stathakopoulos recalls thinking that the company had patched the flaw months earlier and that there was nothing more that Microsoft could do. He headed back to bed. About 20 minutes later, he got a call from his boss, Nash. Stathakopoulos was told he had better do something.

Window Snyder remembers being in a meeting the next Saturday morning when Stathakopoulos pointed to her and motioned for her to leave the room. The two headed straight to another conference room--one full of people "with fire coming out of their ears."

Next page: The beginning of Blue Hat

Stories

Day 1: From pain to progress
Remond's security practices have been transformed since threats like Slammer and Blaster first wormed their way onto the scene.

Day 2: Inviting the hackers inside
Aiming to be more open, company reaches out to the security research community it once kept at a distance.

Day 3: Emerging security threats
Forget widespread worms. Nowadays, limited-scale threats like targeted e-mail attacks are causing the most concern.

Beyond Binary Blog

Day 1: Inside the war room
After years of having to scramble whenever an outbreak hit, Microsoft builds adjoining situation rooms to coordinate its response efforts.

Day 2: Off to the Limo Races
In what might seem an unlikely pairing, Microsoft employees and security researchers team up to go on a scavenger hunt through Seattle.

Day 3: Meet the bug hunters
One talks a mile a minute, another dresses like a bug. Meet some of the people who have helped lead a massive culture change at the company.

Related special reports

Microsoft's lessons from the desktop

Microsoft meets the hackers

'MSBlast' echoes across the Net

Related stories

Microsoft gathers hackers in Redmond

Microsoft puts key security under Windows umbrella

Microsoft gets good reception at Black Hat

Gates: End to passwords in sight

Bug hunters, software firms in uneasy alliance

Microsoft wants to meet more hackers

Is there method in Microsoft's security buys?

Microsoft's blast from the past

Gates: Security is top priority

Photo galleries

Inside the war room
Painful episodes lead to the creation of a security response center, where teams take on the task of hunting bugs and keeping customers informed.December 3, 2007

The bug hunters
Just who are the people charged with the task of keeping code secure at Microsoft? They're risk takers, whether donning silly costumes or swimming with sharks. December 5, 2007

Credits

Editors: Anne Dujmovic, Mike Ricciuti
Design: Andrew Ballagh
Production: Kendra Dodds