New comScore figures suggest fewer people believing comScore
comScore has done a wonderful job. Of marketing comScore results.
If the Internet abacus company sees its readings suggest a significant conclusion, it releases the information in an interesting and digestible form.
However, I understand that both comScore and its frats-in-stats at Nielsen Online are having their audits audited by the Interactive Advertising Bureau after mlb.com declared that Nielsen Online's score for its site of 6 million was a "conScore." The real figure, according to mlb.com, was actually 19 million. (the results of the audit's audit are due at the end of this year.)
I try to leave discussions of numbers to intelligent people.
But there seems to be a big difference between 6 million and 19 million.
As I was thinking about this, a book wafted beneath my nose that tended to crystallize some human instincts about facts, something that numbers purport to be.
It's called "True Enough: Learning to Live in A Post-Fact Society" by Farhad Manjoo.
(Credit:
misocrazy)
Mr. Manjoo performs an enjoyable analysis of some recent political controversies, such as the allegations that the elections of 2000 and 2004 were stolen by devious and surprisingly organized Republicans. (His conclusions seem to suggest that Mr. Gore was hard done by, Mr. Kerry was not.)
The book is at its strongest in describing just how deeply most human beings want to find information that most closely confirms their own prejudices. And how they shut out information that counters those prejudices.
What prejudices do research companies have? Is it, perhaps, important for them to have their research come up with newsworthy results? Are their methodologies actually primed to achieve that?
There are allegations that comScore's and Nielsen Online's figures tend to discriminate against, for example, foreigners and MacOlytes.
Why would the research companies allow for this sort of speculation?
Why would they allow for the perception that someone on a Mac in Krakow, Poland, is nothing more than a hanging chad?
According to the Interactive Advertising Bureau's CEO, Randall Rothenberg, these companies are "still relying on panels, a media-measurement technique invented for the radio industry exactly seven decades ago, to quantify the Internet".
Wait a minute, they're using panels? Does everyone know about this? Do the people who use their numbers know about this?
For so many people in the advertising business and beyond, who have their prejudices too, it is the headline that matters. They present in headlines. They talk about themselves in headlines. They need news.
Being 21st Century humans whose budgets are shrinking, attention spans are short and careers even shorter, they sometimes eschew analysis for today's news currency, the soundbite.
comScore and Nielsen Online are in the business of creating some very soundbiting headlines indeed. (FACEBOOK OVERTAKES MYSPACE!!! OHMIGOD!!! I NEED TO WRITE A SONG ABOUT THIS!!!)
Which leads me to the headline of this post.
I have no reason to believe that the folks at comScore and Nielsen Online are anything other than well-meaning, dedicated but imperfect professionals.
But what if the conclusion of the IAB audit is that the figures from companies such as these have been wildly inaccurate?
What would their PR people do with that?
Would they publicize these findings, as a declaration that they need to work harder, to find better methodologies in order to reveal more accurate truths? (Oh, there are so many inaccurate truths out there..)
Or would they decide that wouldn't be good for business?
I'm just asking.
You see, I only have a MacBook and I'm feeling ignored.
Chris Matyszczyk is an award-winning creative director who advises major corporations on content creation and marketing. He brings an irreverent, sarcastic, and sometimes ironic voice to the tech world. He is a member of the CNET Blog Network and is not an employee of CNET. 
c
Mr. Matyszczyk, You have a vivid imagination. Contrary to your claim that fewer people are believing comScore numbers, the opposite is true. As of Q1 2008, approximately 1,000 clients were purchasing comScore services, up about 40% in just over one year. The reason they choose to buy our services at such a rapid clip is that for the Internet ? as in the case of every advertising medium ever created -- third party objective estimates of audience size are demanded by the companies who pay to buy the advertising. And, in the case of the Internet it?s now been shown beyond a shadow of a doubt (http://www.comscore.com/press/release.asp?press=1389) that the numbers obtained from a site?s own server-based estimates of unique visitors ? which rely on counting cookies ? simply don?t cut the mustard. Cookie deletion rates today have climbed to the point where 30% of all Internet users delete their cookies and do so, on average, five times a month. That means a site?s server (which relies on cookie counting) will count a repeat visitor as a new ?unique? visitor every time they return to the site after they?ve deleted their cookies. This leads to an overstatement of unique visitors when using site server data of as much as 150%. And that doesn?t even include the overstatement caused by a person using different computers (e.g. a work and personal computer) to visit the same site. In that case, a site?s servers will count the one visitor as two different people. Advertisers are increasingly aware of these problems and that?s why they turn to comScore.
For example, here?s what Richard Castellini, Vice President of Consumer Marketing at CareerBuilder.com had to say:
?Panel-based measurement is needed to accurately measure the size and characteristics of a site's online audiences. The comScore cookie deletion study quantifies the magnitude of the discrepancy one can encounter using log files and highlights the significance of this issue to our industry.?
comScore doesn?t use cookies in its counting methodology. We actually observe how many times each of our panelists visits each site. We count people in our site audience statistics -- not browser, not cookies and not machines ? because marketers advertise to people and that?s what they need to have counted.
It?s abundantly clear from your comments that you have a distaste for panels. You need to remember that panels are simply samples taken from a population and which are then used to estimate the size and characteristics of the population from which they are taken. Rather than being outdated, samples are still used extensively today in fields as varied as advertising research, food production, nuclear research and medicine. If you say you don?t believe in panels in audience measurement that?s akin to saying you don?t believe in sampling theory. In that case, next time you visit your doctor and he tells you he ?wants to take a little blood?, you should instruct him to ?take it all??.
Sample populations are widely used in research. In fact, much research could not be done without them, as you point oout. Of major importance is that the sample be representative of the population you wish to extrapolate your results to, and that the sample be random. It is easy to rig a sample or use poor sampling technique and claim it is random. This is a very common practice in all areas of research and it is at best poor research and at worse flat out deception.
One of the biggest problems with the results presented by companies such as Nielsen and Comscore is the lack of transparency in their research methods and data. It is easy to get the answers you want by asking questions designed to get you the answers you want. Who will know if the process isn't transparent? Of course they will claim it is "proprietary" and, therefore, eliminate any chance of third party scrutiny. This allows them to make claims without pier review, as is vital in the scientific community. Such review allows for public debate and pier consensus.
Another issue is how the conclusions are presented. They are presented as fact, when indeed they are not. They are merely educated guesses based upon data collected through imprecise methods (even if the best available). If your methodology is faulty at any point, the results will be even further removed from relevance. Companies that offer such research services portrait themselves and their techniques as accurate and factual. They must. Would you buy Comscores services if they told you their methodology was faulty, this was a best guess, that you should be wary of any conclusions drawn from the research, and that they might not reflect reality at all?
Finally, there is ample distortion by the media in an effort to create "news-worthiness" or sensationalism, by the companies who conduct research, and by companies who purchase such research to help them make business decisions and especially to recruit investors. In short, there is powerful motivation to arrive at conclusions that best support your needs. Without outside scrutiny, there is no balance and no accountability.
This article questioned your research methods. Rather than become defensive and attack, maybe you should acknowledge the faults and explain how you attempt to overcome them. Yes, website hits are flawed, but so are panels. How do you attempt to compensate for this? Certainly not by attacking those who question you.
The scientific community operates by open debate and scrutiny. It is not always pleasant. Anyone who conducts research should be prepared to accept criticism. Granted ComScore is not a member of the scientific community, but you should also not delude yourself that you are, or try to convince others.
I tend to think that both facebook and myspace arn't going anywhere 'soon' but soon enough there going to go the way of the ".coms" of lore once advertisers realize that there are much more creative way's to get people to buy there t-shirts
Sean - cafewarrior.net
I am deeply reassured whenever I hear anyone from a research company use the phrase "beyond a shadow of a doubt."
babyspittle, thank you for phrasing some of the issues in such detail.
In a world where everyone is selling, it is surely interesting to question whether companies' methodologies affect what they ultimately feel they need to present.
I feel that beyond a shadow of a doubt.
chris
For example, in the same month earlier this year, comScore reported that Google had 29,349 unique users, while Nielsen reported 27,714. Not too far apart. But then look at something like eBay (comScore- 21,022, Nielsen- 15,523), BBC (comScore- 18,194, Nielsen- 15,807), Ask (comScore- 13,738, Nielsen- 7,495), Wikipedia (comScore- 13,180, Facebook- 8,793.) Strangest of all, Bebo, which has a very young audience, so should definitely be under reported by comScore, as it ignores everyone under 15- comScore reported 11,392 users, while Nielsen reported 4,648- a difference of 145%.
- by gfulgoni June 24, 2008 7:15 AM PDT
- somerandomnerd, you raise some interesting points. To clarify, at comScore we use 2+ domestically and 15+ internationally. Also, when comparing audiences between comScore and Nielsen it's very important to make sure one is comparing apples to apples, e.g. we include school users in our domestic numbers and they don't. We can also report global traffic and they don't.
- Reply to this comment
-
-
- by somerandomnerd June 25, 2008 1:58 AM PDT
- I should clarify that I'm quoting figures for the UK from the comScore press release (it's just as easy here to forget that there's a whole world outside our borders!), so the audiences of 2-15 year olds and school users wouldn't appear in those figures I quoted.
-
-
(9 Comments)