Perspective: When security researchers become the problem

There's a myth about security researchers that goes like this: Vendors are made up of indifferent slugs who wouldn't fix security vulnerabilities quickly--if at all--if it weren't for noble security researchers using the threat of public disclosure to force them to act.

The reality is that most vendors are trying to do better in vulnerability handling. Most don't need threats to do so, and some researchers have become the problem.

In so stating, I thank those researchers who are genuinely motivated by the public good, most of whom never get the headlines of their more notorious brethren. I also acknowledge that the vendor community needs to improve the quality of commercial software so we have far fewer vulnerabilities.

Here's a rundown of some of the notions that play into this myth about how security researchers interact with software makers.

1. You should be able to fix this in two days
Some researchers think they can push vendors to work faster by threatening to "tell all," and that if vendors really tried, they could meet the researchers' arbitrary 5-day, 15-day or 30-day "fix window."

In reality, many of the best researchers aren't the ones you hear a lot about, because discretion is their stock in trade.

In reality, when a researcher reports a vulnerability, the fix might be a two-line code change and take 20 minutes to do. However, getting the fix in customers' hands often takes weeks. Remediation may require the vendor to analyze whether the bug is specific to a particular version/platform or all versions/all platforms or analyze whether related code has a similar problem (to fix the problem everywhere). Vendors may also need to provide fixes on multiple versions/platforms or bundle multiple security fixes together to minimize patching costs to customers, not to mention various testing on the products shipped to ensure the fix does not break anything else.

As an example, Oracle has done 78 fixes for a single vulnerability, which took more than five days to complete. We also release bundled fixes quarterly on dates tied to financial reporting calendars (e.g., many customers will not touch their production systems during quarter-end).

A two-line code change can take five minutes, but getting a fix into customers' hands in such a way that they will apply it takes way more than a few minutes.

2. The more notorious I am, the more business I will get
Many researchers think that the more vulnerabilities they disclose publicly, the more vendors will hire them as consultants. Some engage in explicit threats ("Pay me $X or I sell this to iDefense") or implicit threats ("Fix it in the next three weeks because I am giving a paper at Black Hat").

Not all researchers are noble-minded, and not all vendors are indifferent slugs.

In reality, many of the best researchers aren't the ones you hear a lot about, because discretion is their stock in trade. They are often far better than the "look what I did" researchers who run to the press with their latest vulnerability pronouncements. The circumspect researchers are the only ones we hire and the only ones we recommend to our customers.

Also, notoriety can backfire: I've known customers to terminate contracts with researchers for releasing exploit code. Researchers, you might get applause from hackers when you show off at Black Hat, but businesses will not pay you to slit their throats. With knowledge comes responsibility.

3. I should always get credit for vulnerabilities I find
Most vendors credit researchers who report vulnerabilities so that researchers will continue to work with them. Also, saying "Thank you for working with us" is just good manners. The myth is that researchers are always entitled to credit.

In reality, when a researcher puts customers at risk by releasing exploit code for a vulnerability before the vendor has had a chance to fix it, it's ridiculous to expect the vendor to say, "Thank you for putting our customers at risk." I've never had a customer ask us for exploit code or exploit details, though they do want enough information to do a risk assessment.

In some cases, vendors may actually be giving more credit than the researcher deserves. For example, Oracle finds more than 75 percent of significant security vulnerabilities in-house. Yet if a researcher finds an issue that we already found internally but may not have completed the fixes for, we typically still give that person credit, anyway.

The reality is that not all researchers are noble-minded, and not all vendors are indifferent slugs. The other reality is that the highest purpose of everybody in this game should be protecting customers who use these products from harm.

Biography
Mary Ann Davidson is the chief security officer at Oracle, responsible for security evaluations, assessments and incident handling.

More Perspectives

[fergdawg]

Okay, a couple of days is fine, but over 700 days?!?

I agree with most of this, but this seems to me more like a knee-jerk reaction to the fact that it took Oracle over 700 days to provide patches to known security vulnerabilities. That is just a tad ridiculous.

See also:

http://www.eweek.com/article2/0,1759,1838810,00.asp


- ferg

[dargon19888]

Puleeze give us a break...

Seems you're trying to lash out at those who report problems with your product.

Lets face it. Yes it sometimes takes time to fix a bug. But the reality is that security has always been a secondary thought. Getting a product to market quickly has always been at the forefront of Oracle, IBM and Microsoft, damn the code or product defects!

Why don't you take the time performing better code reviews or testing prior to shipping a product? Or maybe you should ONSHORE the work and hire a better quality developer? The majority of your security bugs occur because of sloppy work and a lack of adherence to following "best practices".

Oracle CSO has little room to talk

I find it amazing that the CSO of Oracle would knock security researchers. Given Oracles continued penchant for security problems with their products, while providing in some cases very slow response times for fixes, and the fact that they require users to *PAY* for access to security patches through their maintanence program (unlike every other major vendor including Microsoft..)there appears to be a significant hidden agenda within this assumingly otherwise credible security professional.

[C.Schroeder]

On the contrary...

I don't know what part of the computing field you work in, but from where I stand paying maintenance fees is accepted as a normal business practice. Paying maintenance is typically enforced through the use of license keys (e.g. FlexLM). Purchase price of software of this caliber is typically greater than the computer hardware it runs on.

Microsoft has been the exception, not the rule, when it comes to charging customers for patches. Haven't you noticed that since Microsoft no longer has explosive sales growth for Windows and Office they are now trying to figure out how to gracefully move their customers away from free maintenance? The alternative, of course, is (forced) upgrades, where the customer must repurchase the latest version of the product.

In a saturated market, maintenance fees are a legitimate method for a software vendor to continue receiving income, enabling them to continue maintaining and evolving their product.

Spin, n.

What a wonderful example of spin. My kudos to the author.

The demolishing of hyperbolic strawman ('indifferent slugs', 'two day fix'), and the choice to spend most of the article caricaturing security researchers Oracle doesn't like ('2. The more notorious I am, the more business I will get', '3. I should always get credit for vulnerabilities I find') are pure propaganda gold!

Nicely done!

Anyone from the real world out there

(don't bother flaming, how about some constructive talk?)

How many of you have worked in a large software company with a huge code base, which has evolved over 20+ years that supports multiple platforms and versions?

Assuming that it does take real time to fix bugs and do all the cross and backporting.

Sssuming that Oracle is lazy in fixing them.

Assuming that these risks have to be evaluated and prioritized. That's business, good or bad, plain and simple.

Assuming that most of these vulnerabilities have never, ever been exploited (here come the flames). Who'd like to, even anonymously, admit that someone exploited a flaw in their Oracle Db? Who even runs an Oracle Db that is available on the net? Some SQL*Plus ASP? or are we talking an inside job, where it's easier to just read the sticky note with the DBA password on it, but I digress.

Who benefits from publishing exploits?
Not Oracle.
Not customers.
"Security Researchers" (look at me!)

I agree

I also think they need to stop publishing the names of the security researchers. . . These guys get off on seeing their names next to the flaws they discover, makes their Mothers real proud. Why give them the pleasure.

[wbenton]

After fact article about Michael Lynn?

I take it this article was written as an "after the fact" article about Michael Lynn's decision to go public.

Researchers have the responsibility to disclose security bugs to the respective parties, but likewise... those respective parties also have a responsibility to provide a patch.

If somebody claims they'll go forward in x days if a patch is not released is one thing, but in the case of Michael Lynn with his Cisco bug release... Cisco had over 4 months to provide a patch.

Trying to get money to not publicly disclose is one thing, but not disclosing what should have already been patched is whole different ball game.

In the case of Michael Lynn, I believe he did right, but for somebody who says they'll go public in 2 days for a patch that only takes 5 minutes... now that would fall into the category of this article.

Patches often times require 1-2 weeks of testing to ensure the patch doesn't cause ill effects. Thus a minimum of 2 weeks should be considered.

But likewise, if a vulnerability exists... the purchaser of that vulnerable equipment wants to know about the bug ASAP, so there comes a trade off.

Bugs should be made public as they're identified... with a possible 1-2 week disclosure delay period. But longer than that... and it's not safe.

The problem is not the end-user... but the vendor's product whom the end-user purchased. As such, the vendor should/MUST take responsibility for released software/firmware and offer patches as quickly as possible.

In Cisco's case... they had over 4 months to fix the bug but didn't. As for the why's... only Cisco can tell you, but 4 months is too long to wait and thus going public in Michael Lynn's case was the right thing to do.

Walt

[ciropabon]

I have to confess Oracle has a point..

I started by not liking Oracle's representative attitude. When you are so big, why waste your time on little guys that do not have as much fame as Oracle's chief security guy (or girl)?

Then, I did a little homework and I found that Oracle's record regarding security holes and unpatched flaws is excellent (www.secunia.com).

After a quick search I could find only a couple of holes and both are from July 2005. Even venerable Oracle 7 has only ONE flaw and it is patched.

Really impressive. I am an Oracle DBA since version 6 and this was ages ago (when I was young and beautiful), and I really had not the slightest idea of this particular Oracle strength.

But (yes, there is a but) when you try to measure public opinion about hackers, please, take a look at IE's record of years and years of unpatched holes.

It is hard to understand Oracle's management. Are you not touting everywhere and right now that you are virtually "hacker proof"? And it seems to me that you are almost there. Maybe Oracle should start Hacker University as a marketing tool against MS. And, please, do not mention my name in your next strategy meeting... :-)

Half the story

There's a whole other side to all of this that Mary so blatantly fails to understand. She complains about security researchers as a whole, but then points to the actions of those who publicize their findings. Anyone who's seen someone reverse engineer a patch (or done it themselves) can tell you that in most cases a security patch fixes more problems than they tell you about. The truth of the matter is that not all researchers ever make their findings public. I (together with one other person) have found significant holes in a VPN product years ago, but given the situation at the time we decided together that releasing information about it to the public would be a horrendous idea. The vendor fixed the issues and mandated the upgrade, we verified that the problems were fixed, and there was no need to make it public. But you know what? It took us less than 2 days to find the flaws, and they were wide enough to drive a bus through. Any bad guy worth his salt (or for that matter, enough salt to cover an order of fries) would have found them also, no problem. And if the good guys didn't look first, eventually a bad guy would have.
Which brings me to our immediate situation with Mr. Lynn. Mary seems to be upset that Michael Lynn has given this information to the bad guys now, and that he should have kept it to himself as my colleague and I did. Mary, you IDIOT...he first got the information off of a CHINESE WEB SITE! Think about that one for a second. No, really! Let it sink in! This was the diametric opposite of my situation. This wasn't like finding out that there's a way to kill the president and yelling it in public...this was a little more like finding someone's notes about HOW to kill the president and going to the news because nobody else would listen. Either get with the program or get a decent amount of information before you actually write an article under the guise of having the slightest idea what you're talking about. This level of blatant disregard for/disinterest in the facts is shameful, most of all for someone trusted with the security of the second-largest software company in the world. If I were your boss, I'd be calling you out on the carpet with the serious consideration of replacing you.

But Oracle you haven't fixed dozens of your holes yet.

This is sad. Mary gets a whole spin article and I don't see any articles by David or Cesar.

I am a security researcher. I have published a small fraction of the holes I have found in commercial software you may be using now. Many of the holes are still not fixed.

I publish what I research on my own time. I am proud of the work that takes nights and weekends to complete. I do not publish what clients pay me to find, unless they okay it. The #1 reason a client will okay publication is to get an arrogant ISV to fix their security defects.

After playing nice with a**hats all week long, people talking to me like Mary make me want to Zero-Day everything.

1. Oracle products are *still* full of security holes, whether you like it or not.

2. Going to Secunia or any bugtracking list won't help you understand the issues. Looking at the code will. Many of these are simple to find, simple to exploit. And still disasterous.

3. Several of the security patches Oracle has concocted do not fix the security flaws. The patches simply do not work, or provide a myopic plug to a macro hole.

4. Mary hasn't a clue, and clearly is not being held accountable for her negligence. Were she "security manager" I'd recommend laying blame with senior management. Oracle's security negligence is clearly an organizational issue as much as a technical issue. Mary is, however, CSO. As an officer of the company, she should be held accountable for her egregarious failings.

5. This isn't a "security researcher problem". Security Researchers don't write low-quality software for Oracle. They don't sell it to Oracle's clients. They aren't responsible for all of our personal data stored in still-insecure Oracle products.

6. Let's face it, most of you haven't a clue how to find or exploit these flaws. I don't see posts anywhere from people whom have a clue arguing that security researchers are the problem. I don't see anyone who has a clue defending the vendor(s).

The people defending Oracle are a biased-sample:
-CSO/CISSP types sucking up to business management
-Non-technologists
-Oracle certified DBAs
-Developers who do not want to take responsibility for the security flaws in their code, or have a chip on their shoulder for being burned before.

7. Can anyone actually make an accurate statement about what Michael Lynn released at BlackHat? He didn't release new vulnerabilities. He demonstrated that Cisco was dishonest about the scope and exploitability of existing security defects...for years.

There was nothing for Cisco to "patch" but intellectual integrity, and as a researcher who has had the unfortunate pleasure of dealing with Cisco before, let us just say that I sympathize.

7. Sales and marketing runs the world. That's all Mary is in this.

Opinions on parade.


We help you for free. We do not have to. Sure we may benefit from publication. What of it?

I saved the a nice pink copy of the Financial Times where you run the ad repeatedly in London claiming Oracle is "Unbreakable".

I'll post a scan of that with a few dozen unfixed holes for your clients if you like,

TDD

[dsorak]

Myth #4: It is always easy to report your finding to a vendor.

I cannot begin to count the times I have reported a problem or vulnerability to a vendor's technical support line only to have the monkey on the other end of the line start reciting scripted responses. (My apologies to the monkeys - you are only doing what you've been told.)

Some of the time I can actually get them to escalate the problem, but the effort of attempting to convince them that I have really found something is exhausting. Not only that, much of the time the report seems to fall into a black hole. The vendor never acknowleges that it actually IS a problem and no followup contact is ever made.