Securing Microsoft: A long road

At software giant, pain gives rise to progress

TalkBack E-mail del.icio.us Digg this

Editors' note: This is part 1 in a series examining how Microsoft's security strategy has evolved over the past decade.

REDMOND, Wash.--With a measure of pain, Matt Thomlinson recalls the summer of 2003.

"I remember buses pulling up to the Microsoft campus to shuttle engineers away from their day jobs to go work the phones down at (product support)," said Thomlinson, who heads Microsoft's security engineering efforts. "That was just heartbreaking."

The Blaster worm had just hit, swamping Microsoft's support lines with calls from angry customers.

Andrew Cushman, director of the Microsoft Security Response Center, remembers standing in Muck boots and installing a catch basin in his front yard when he got a call from an account manager. It was just days after September 11, 2001, and one of Microsoft's largest customers had just been hit with what turned out to be the Nimda worm.

George Stathakopoulos, Cushman's boss, still hasn't seen the end of the movie Master and Commander. In spring 2004, he was sitting on his couch watching the film when he got the call that Sasser had hit.

Indeed, much of Microsoft's current security practices can be traced to painful lessons learned during the past decade by people whose job it is to secure Microsoft's products.

Because of the experience of Mike Nash, a vice president at Microsoft, the company finally instituted calling trees as a way to quickly reach people in an emergency. When the Slammer worm hit in January 2003, Nash had to work feverishly to track down the vice president of SQL Server, Gordon Mangione, eventually locating him at his sister's wedding in Canada. (Slammer used Microsoft's SQL Server database to propagate a denial-of-service attack.) Nash first heard reports of Slammer on the local news radio station at 6 a.m. At first, he thought he was dreaming. But as the report played a second time, he knew it was real and headed into work. "I was the second one there," Nash recalls.

Slammer also taught the company that it was not enough to have a patch; the patch had to be easy enough to deploy so that most customers would do so, lessening the chances that outbreaks would propagate so quickly. And it was Blaster that taught the company that it wasn't enough to patch a single flaw; it needed a systematic process for catching whole classes of vulnerabilities, a realization that paved the way for Microsoft's current approach, known as the Security Development Lifecycle, or SDL.

"We've put a lot of our best people in these areas," Microsoft Chairman Bill Gates said in an interview with CNET News.com. "Still tons to be done, but you know, we've definitely made five years of progress in the last five years."

Much of the reason for that traumatic on-the-job training can be traced to Microsoft's decade-long evolution in how it and its employees deal with security. Until 1997, security was seen mainly as a set of features that the company bolted onto its software long after product design and development. The idea of securing code as it was being developed had not been considered.

IE flaws send Microsoft scrambling
That all began to change in March 1997, when the first significant flaws were discovered in Internet Explorer. Researchers at Worcester Polytechnic Institute found a vulnerability in browser shortcuts known as .LNK files. Even as Microsoft was scrambling to deal with the problem, word of the flaw hit cable television news. A few hours later, researchers at the University of Maryland found a second problem and reported it to Microsoft.

Simultaneously, the IE team, which Stathakopoulos was part of, was in the process of moving into a new building. The timing couldn't have been worse: most of their equipment was in boxes. Someone had to run to a store to buy a power supply for one of the team's laptops--the power cords had been packed away--before the battery went dead. Jason Garms, now a senior director for technical strategy, wrote the company's first security bulletin in a Windows' Notepad file and then copied it to a floppy so it could be distributed to customers.

At the time, the company didn't even have a system in place where outsiders could report security bugs directly to Microsoft engineers. The IE flaw came to light because someone had called Microsoft's support line and the matter had gradually escalated.

"We said 'This has to stop,'" Stathakopoulos recalls thinking of the disjointed system at the time. "It's not working for us."

In the aftermath of that bug, Microsoft created the Microsoft Security Response Team as well as a separate Internet Explorer security group. The company also created an e-mail address where outsiders could report potential issues.

The Microsoft Security Response Team was made up of volunteers--employees who had other day jobs, but were interested in helping out when there was a security problem.

Next page: The era of big worms

Stories

Day 1: From pain to progress
Remond's security practices have been transformed since threats like Slammer and Blaster first wormed their way onto the scene.

Day 2: Inviting the hackers inside
Aiming to be more open, company reaches out to the security research community it once kept at a distance.

Day 3: Emerging security threats
Forget widespread worms. Nowadays, limited-scale threats like targeted e-mail attacks are causing the most concern.

Beyond Binary Blog

Day 1: Inside the war room
After years of having to scramble whenever an outbreak hit, Microsoft builds adjoining situation rooms to coordinate its response efforts.

Day 2: Off to the Limo Races
In what might seem an unlikely pairing, Microsoft employees and security researchers team up to go on a scavenger hunt through Seattle.

Day 3: Meet the bug hunters
One talks a mile a minute, another dresses like a bug. Meet some of the people who have helped lead a massive culture change at the company.

Related special reports

Microsoft's lessons from the desktop

Microsoft meets the hackers

'MSBlast' echoes across the Net

Related stories

Microsoft gathers hackers in Redmond

Microsoft puts key security under Windows umbrella

Microsoft gets good reception at Black Hat

Gates: End to passwords in sight

Bug hunters, software firms in uneasy alliance

Microsoft wants to meet more hackers

Is there method in Microsoft's security buys?

Microsoft's blast from the past

Gates: Security is top priority

Photo galleries

Inside the war room
Painful episodes lead to the creation of a security response center, where teams take on the task of hunting bugs and keeping customers informed.December 3, 2007

The bug hunters
Just who are the people charged with the task of keeping code secure at Microsoft? They're risk takers, whether donning silly costumes or swimming with sharks. December 5, 2007

Credits

Editors: Anne Dujmovic, Mike Ricciuti
Design: Andrew Ballagh
Production: Kendra Dodds