Yahoo BOSS + Twitter + Google App Engine = fresh news
Yahoo BOSS engineer Vik Singh
(Credit: Stephen Shankland/CNET News)Here's Web 2.0 at its finest: A Yahoo programmer has combined his own project, Yahoo BOSS (Build Your Own Search Service), with Twitter and Google App Engine to create a new way to determine what news is both new and important.
The service, called TweetNews, presents Yahoo news search results in a different way, using results from the same search on Twitter to determine what should get high placement, according to a blog posting about it by BOSS engineer Vik Singh.
BOSS supplies Yahoo search results in a form that can be repackaged, processed, and published for free, though Yahoo asks for revenue sharing for popular services.
TweetNews combines human interest, as judged by Twitter users, with a measure of authority, as judged by publications that make the cut for Yahoo News search. The application also includes an expandable "related tweets" button that supplies links to people's Twitter references to the various news stories.
"Twitter as a ranking signal for search freshness may prove to be very useful if constructed properly," Singh said in the blog.
Here's a screenshot of the search in action, using the terms "hudson plane" to illustrate the news items Twitter users find most pertinent.
Vik Singh's TweetNews application shows how Twitter can be used to find the most pertinent breaking news.
(Credit: Vik Singh/TweetNews/CNET News)The application is publicly available as a service running on Google App Engine--not the first time Singh has demonstrated BOSS ideas on his main competitor's application hosting system. Google hosts applications on App Engine for free, but only within various limits, and Singh's However, the application exceeded its quota within a few hours of his posting.
The application isn't just a novel demo, though. It's an attempt to solve a challenging problem in determining what breaking news is most pertinent to people. Here's how Singh describes the challenge:
Freshness (especially in the context of search) is a challenging problem. Traditional PageRank style algorithms don't really work here as it takes time for a fresh URL to garner enough links to beat an older high ranking URL. One approach is to use cluster sizes as a feature for measuring the popularity of a story (i.e. Google News). Although quite effective IMO this may not be fast enough all the time. For the cluster size to grow requires other sources to write about the same story. Traditional media can be slow however, especially on local topics.
I remember when I saw breaking Twitter messages describing the California Wildfires. When I searched Google/Yahoo/Microsoft right at that moment I barely got anything (< 5 results spanning 3 search results pages). I had a similar episode when I searched on the Mumbai attacks. Specifically, the Twitter messages were providing incredible focus on the important subtopics that had yet to become popular in the traditional media and news search worlds.
What I found most interesting in both of these cases was that news articles did exist on these topics, but just weren't valued highly enough yet or not focusing on the right stories (as the majority of tweets were). So why not just do that? Order these fresh news articles (which mostly provide authority and in-depth coverage) based on the number of related fresh tweets as well as show the tweets under each. That's this service.
Stephen Shankland writes about a wide range of technology and products, but has a particular focus on browsers and digital photography. He joined CNET News in 1998 and since then also has covered Google, Yahoo, servers, supercomputing, Linux and open-source software, and science. E-mail Stephen, or follow him on Twitter at http://www.twitter.com/stshank. 
a sort of mashup between all the main content providers inside a portal-like window
should be kinda fun.
- by lonestarState January 16, 2009 8:51 AM PST
- BOSS has inspired some really great mashups. BuildaSearch.com is a cool mashup, which I have learned are testing the Microsoft Live search API. Wow can't wait to see what is under the hood!
- Reply to this comment
-
(5 Comments)