Microsoft isn't too happy with the results of a recent test that found fault with its antivirus software.
For the second time in a row, the company's Security Essentials failed to win certification from AV-Test, a German-based testing lab that evaluates the efficacy of antivirus products. Out of 25 programs tested, only three failed to gain AV-Test's thumb's up for certification.
Microsoft's Forefront Endpoint Protection, which is geared toward corporate customers, also failed to gain certification.
Microsoft responded to the test via a blog posted yesterday, challenging its findings.
"Our review showed that 0.0033 percent of our Microsoft Security Essentials and Microsoft Forefront Endpoint Protection customers were impacted by malware samples not detected during the test," Joe Blackbird, program manager for Microsoft's Malware Protection Center, said. "In addition, 94 percent of the malware samples not detected during the test didn't impact our customers."
Blackbird cited three different examples in which he believes AV-Test missed the mark:
AV-Test reports on samples hit/missed by category. We report (and prioritize our work) based on customer impact.
AV-Test's test results indicate that our products detected 72 percent of all "0-day malware" using a sample size of 100 pieces of malware. We know from telemetry from hundreds of millions of systems around the world that 99.997 percent of our customers hit with any 0-day did not encounter the malware samples tested in this test.
AV-Test's test results indicate that our products missed 9 percent of "recent malware" using a sample size of 216,000 pieces of malware. We know from telemetry that 94 percent of these missed malware samples were never encountered by any of our customers.
Microsoft cut AV-Test a bit of slack by saying it's difficult for independent groups to devise tests that can mimic the real-world conditions of virus attacks. At a security conference last year, AV-Test itself admitted to certain flaws in the methods used by current independent tests. But it still stood behind its approach.
Security Essentials has been on a bumpy ride with AV-Test as of late.
In an e-mail to CNET, AV-Test CEO Andreas Marx said that Microsoft brought up some good points in its blog and that the group has been discussing these items with the company and other antivirus vendors. According to Marx, the issues cited by Microsoft aren't specific to testing but rather to the prevalence of certain viruses and their impact on actual users. "Depending on what you count, you will get different results," he said.
The CEO also explained the approach used by AV-Test.
"As of today, every two seconds we see three new malware samples, which are summing up to a few million samples per month," he said. "Instead of looking at millions of samples, our focus is on the unique families. Out of every family, we select recent samples in order to use them in our tests. So the impact of these samples is indeed low, however, the impact of the malware family is considerably high. We favor the family-based approach over the sample-based one because of today's malware situation."
The testing is also dependent on the amount of data AV-Test receives, both from antivirus vendors and users.
"To create meaningful data, we are already using telemetry data from a wide range of antivirus companies and users of these products," Marx said. "Our tests can only be as good as the prevalence data we're getting in a timely manner."