July 13, 2006 1:19 PM PDT

Microsoft looks to foil Web spammers

Researchers at Microsoft have developed a tool to scrub search engines of major Web sites that pollute search results and ultimately help clean the Web of spam.

Called Strider Search Defender, the tool is designed to dig out Web pages that are a front for spam Web sites, according to a paper published by Microsoft researchers on Thursday. These Web pages typically reside on blogging sites and other services that provide free Web space, the researchers said.

Spammers soil the Web with countless links to their spam fronts in order to gain a higher ranking in search engines. "By cleaning up Web search, hopefully we can discourage spammers from cluttering the Web with spam," Yi-Min Wang, principal researcher at Microsoft, said in an interview.

Microsoft's tool doesn't find spam the traditional way, by looking at the site's content. Instead, it turns the spammers' activities against them by using search engines to find links to potential spam pages. These links are often posted as comments on blogs, in online discussion forums and in guestbooks, also called "comment spam."

Search Defender starts with a list of confirmed spam Web addresses. A "Spam Hunter" part of the tool runs those addresses through search engines to find pages that link to the spam sites, using the "link:" query tag. Additional spam URLs found on those sites are, in turn, run through the Spam Hunter, resulting in a long list of potential spam sites.

Then, using another Microsoft research project, Strider URL Tracer, false positives are filtered out and a list of Web pages that redirect to spam sites is compiled. Strider URL Tracer actually visits each one of the Web addresses found by the Spam Hunter to see if it redirects to a secondary spam page.

"We use search engines to find them," Wang said. "Spammers are basically telling us: Here are my spam URLs."

Spammers use various online services to host spam fronts, including free Web hosting providers such as Tripod, Angelfire and Yahoo's Geocities, Microsoft said. Blogging services are also often abused, Google's Blogger at blogspot.com is especially popular, according to the research report.

"Our preliminary investigation shows that spam blogs hosted on blogspot.com appear to be particularly widely spammed and effective against search engines," the report said.

Microsoft's researchers are working with the MSN Search team to see how the search service could be cleaned up, Wang said. Additionally, he called on the Web community, especially the operators of blog and free hosting sites, to cooperate to combat the Web spam problem.

"In the end it is all about protecting the search engines. Because if the spam doesn't show up in any search engine result the spammer will not receive traffic," Wang said.

See more CNET content tagged:
spammer, researcher, search engine, spam, blogging


Join the conversation!
Add your comment
Tech Report
The report that you said was released by MS today
(MSR-TR-2006-97) is not new, just updated. I was first placed on the web several months ago, early April. It was discussed and link to on several blogs like this one. See the SEE ALSO link.
<a class="jive-link-external" href="http://www.resourceshelf.com/2006/04/11/new-from-microsoft-research-strider-url-tracer-with-typo-patrol/" target="_newWindow">http://www.resourceshelf.com/2006/04/11/new-from-microsoft-research-strider-url-tracer-with-typo-patrol/</a>
Posted by searchinginseattle (5 comments )
Reply Link Flag
EXCELENT> when I see 34,435,678 results I get discouraged
I sometimes want to see results of searches in other engines so I am also concerned if they might be deleting something I want to see. (like the Google lawsuit)

but this sounds like they will clean up the garbage.
Like if I search for dogs a site might say "dogs, poodles, dashunds, collies... with ads that say: if you want to see my ***** pay $29.95 per month"

a very short synopsis of the crass things I've seen anything that will clean it up will be welcome but what about the false positives? what about those sites that accidentlly get in and don't get filtered out? I think it is a very possible situation for a lawsuit that should not be thrown out.
Posted by qazwiz (208 comments )
Reply Link Flag
Good Luck Google getting this tech. All for MSN
Google's probably never going to get it's hands on this tech. M$ will keep it to try and boost hits on it's lame MSN search.

<a class="jive-link-external" href="http://www.webringamerica.com" target="_newWindow">http://www.webringamerica.com</a>
Posted by BMR777 (61 comments )
Reply Link Flag
Yes, read on Dotso.com
This article isn't new. It was linked on Dotso.com not long ago ...
Posted by JoeCrow (83 comments )
Reply Link Flag
Reverse Google Adsense ID Lookup
Web spammers SPAM for 1 purpose: to make advertising revenue. The majority of that is via Google Adsense. TagTooga.com collects adsense IDs and provides the reverse lookup both interactively and programmatically so you can see all the sites showing a particular Adsense ID. Of course, once TagTooga bans the spammer from it's site, that information is no longer available -- TagTooga.com uses the reverse lookup to automatically delete all submissions/domains/URLs with the Google ID, and to prevent future submissions using banned IDs. (see <a class="jive-link-external" href="http://geekswithblogs.net/chilkat/archive/2006/03/16/72540.aspx" target="_newWindow">http://geekswithblogs.net/chilkat/archive/2006/03/16/72540.aspx</a> and <a class="jive-link-external" href="http://www.tagtooga.com/pg/DownloadSites" target="_newWindow">http://www.tagtooga.com/pg/DownloadSites</a> for more information...)
Posted by tagtooga (2 comments )
Reply Link Flag

Join the conversation

Add your comment

The posting of advertisements, profanity, or personal attacks is prohibited. Click here to review our Terms of Use.

What's Hot



RSS Feeds

Add headlines from CNET News to your homepage or feedreader.