Webware

When Yahoo announced BOSS (Build Your Own Search Service) in 2008, the company said it planned to make money from the service. On Wednesday, though, the company announced it's got a new way in mind: charging for high-volume use of the search data.

Yahoo will charge for use of the BOSS API (application programming interface), the service by which other Web sites can extract Yahoo's search data then repurpose it to their hearts' content, according to a blog post by Ashim Chhabra of Yahoo's Search BOSS team. Previously, the company had planned to make money from BOSS by requiring outsiders with high-traffic sites to show Yahoo search ads next to their results.

The new approach allows companies to pursue their own monetization strategies and will help make the API itself more useful by lifting constraints, Chhabra said.

"We're introducing fees for a couple of reasons. First and most importantly, we're hard at work on a number of technologies that will enhance both the functionality and performance of BOSS, and usage fees will help support this development," Chhabra said. "Second, we believe that introducing the proposed pricing structure will improve the ecosystem by optimizing capacity for our serious developers."

BOSS is one part of Yahoo's attempt to make its search more competitive with dominant rival Google, which gained market share over rivals in January, according to Nielsen Online.

One limit that's lifted will be the amount of search results that can be retrieved with one call to the BOSS API; with the fee structure, that limit goes from 50 to 1,000. Yahoo also will offer a service level agreement (SLA) so outsiders can count on BOSS working.

BOSS now can show Web sites' descriptive data spotlighted by Yahoo's SearchMonkey service.

(Credit: Yahoo)

The new fees likely will go into effect late in the second quarter, according to the BOSS fee page; those who use the service will pay on the basis of 10-cent units. For example, retrieving the first 100 results for 1,000 searches costs 10 units; developers will get 30 free credits a day, and the rate goes down during off-peak hours.

SearchMonkey injection
Yahoo also announced it's grafting some SearchMonkey technology into BOSS. SearchMonkey can gussy up certain Yahoo search results in cases when the Web sites listed describe their own data with computer-oriented descriptions called microformats such as a restaurant indicating its address. This idea, called the "semantic Web" and long under development, theoretically gives computers a better understanding of what's on Web pages.

The BOSS API now can be set so that search data it retrieves spotlights that structured data, Chhabra said.

BOSS now also shows two other elements: longer 300-character descriptions of each page in search results, up from 170 characters, and some data retrieved by Yahoo's SiteExplorer technology, which can show details such as popular pages within a particular Web site or a list of other Web sites that link to it.

Yahoo BOSS engineer Vik Singh

(Credit: Stephen Shankland/CNET News)

Here's Web 2.0 at its finest: A Yahoo programmer has combined his own project, Yahoo BOSS (Build Your Own Search Service), with Twitter and Google App Engine to create a new way to determine what news is both new and important.

The service, called TweetNews, presents Yahoo news search results in a different way, using results from the same search on Twitter to determine what should get high placement, according to a blog posting about it by BOSS engineer Vik Singh.

BOSS supplies Yahoo search results in a form that can be repackaged, processed, and published for free, though Yahoo asks for revenue sharing for popular services.

TweetNews combines human interest, as judged by Twitter users, with a measure of authority, as judged by publications that make the cut for Yahoo News search. The application also includes an expandable "related tweets" button that supplies links to people's Twitter references to the various news stories.

"Twitter as a ranking signal for search freshness may prove to be very useful if constructed properly," Singh said in the blog.

Here's a screenshot of the search in action, using the terms "hudson plane" to illustrate the news items Twitter users find most pertinent.

Vik Singh's TweetNews application shows how Twitter can be used to find the most pertinent breaking news.

(Credit: Vik Singh/TweetNews/CNET News)

The application is publicly available as a service running on Google App Engine--not the first time Singh has demonstrated BOSS ideas on his main competitor's application hosting system. Google hosts applications on App Engine for free, but only within various limits, and Singh's However, the application exceeded its quota within a few hours of his posting.

The application isn't just a novel demo, though. It's an attempt to solve a challenging problem in determining what breaking news is most pertinent to people. Here's how Singh describes the challenge:

Freshness (especially in the context of search) is a challenging problem. Traditional PageRank style algorithms don't really work here as it takes time for a fresh URL to garner enough links to beat an older high ranking URL. One approach is to use cluster sizes as a feature for measuring the popularity of a story (i.e. Google News). Although quite effective IMO this may not be fast enough all the time. For the cluster size to grow requires other sources to write about the same story. Traditional media can be slow however, especially on local topics.

I remember when I saw breaking Twitter messages describing the California Wildfires. When I searched Google/Yahoo/Microsoft right at that moment I barely got anything (< 5 results spanning 3 search results pages). I had a similar episode when I searched on the Mumbai attacks. Specifically, the Twitter messages were providing incredible focus on the important subtopics that had yet to become popular in the traditional media and news search worlds.

What I found most interesting in both of these cases was that news articles did exist on these topics, but just weren't valued highly enough yet or not focusing on the right stories (as the majority of tweets were). So why not just do that? Order these fresh news articles (which mostly provide authority and in-depth coverage) based on the number of related fresh tweets as well as show the tweets under each. That's this service.

Search queries served by Yahoo's BOSS service have been steadily growing since the program's July launch.

(Credit: Yahoo)

Yahoo's BOSS (Build Your Own Search Service), which lets others use the company's search technology, is getting some traction.

The service, announced in July, now handles 10 million queries per day, Yahoo announced Monday. And with Google still king of the heap, and Microsoft breathing down Yahoo's neck, the company needs every scrap of influence it can get in the search market.

"We believe growing to more than 100 queries a second in just over 5 months says something about the demand for an open search platform," Bill Michels of the BOSS Team said in a blog post.

"As a point of reference, the total queries from these developer-built, BOSS-powered search engines would rank ahead of the combined searches done on both Facebook and Amazon, and just behind Ask.com," Michels said, citing ComScore statistics.

BOSS queries don't count toward Yahoo's market share, but they could help the company out. That's because heavy-traffic partners using the search infrastructure must either show Yahoo search ads or sign some form of revenue-sharing partnership.

More changes will come to BOSS next year, including the revenue-sharing initiative. "Since launch, we've been focused on adding features and building up the ecosystem. We'll maintain that emphasis in 2009, as well as adding monetizing capabilities to the platform," Michels said.

With BOSS, partners get extensive leeway with Yahoo search results. They may reorder them, mix them with their own results, or filter out particular results, for example. Yahoo handles much of the heavy lifting, including crawling the Web, indexing the pages, and delivering the search results through the BOSS API (application programming interface).

Yahoo has begun offering a new variety of its BOSS (Build Your Own Search Service) called vertical lens technology that lets partners show a subset of search results relating to a particular area.

BOSS lets people repackage, reorder, blend, and otherwise change Yahoo's search results; academics and smaller sites may do so for free, but larger ones must show Yahoo search ads or sign a deal with Yahoo to share revenue. The vertical lens technology lets partners "create a truly comprehensive vertical search engine that complements their core user experience," according to Yahoo.

So far, only some partners can use the vertical lens technology. TechCrunch is one, using the vertical version of BOSS to show technology-related search results. "We're working to share the technology more openly through the BOSS API," or application programming interface, Yahoo said.

Delver, a search start-up that personalizes search results by paying attention to a person's social connections, has signed up for Yahoo's BOSS (Build Your Own Search Service).

BOSS lets larger search sites use Yahoo's search results, tailored in any way desired but stripped of any Yahoo branding, in exchange for showing Yahoo ads or sharing revenue. Delver plans to mix those results in with its own technology, which shows results drawn from publicly visible parts of social sites such as profiles, blogs, bookmarks, and videos. Smaller sites and academic projects can use BOSS for free.

"Leveraging Yahoo Search BOSS allows us to keep focusing on social-graph ranking and indexing, while providing our users with a solution that intelligently mixes social results with traditional Web results," Delver Chief Executive Liad Agmon said in a statement Tuesday.

SUNNYVALE, Calif.--It only took a few years for the science of information retrieval to move from an obscure academic niche to the secretive research departments at the heart of multibillion-dollar Internet companies.

But one of those companies, Yahoo, is trying to give a little more power back to the professors and grad students through a program called BOSS (Build Your Own Search Service). The service lets academics and start-ups build their own search sites around Yahoo's search engine for free, manipulating results however they want.

Two dozen researchers and students from Stanford, the Massachusetts Institute of Technology, Purdue, and other universities met here at Yahoo for a day in September to hear the company's BOSS pitch, show off some ideas they've had for how to use it, and try to coax Yahoo into sharing even more information through BOSS. Overall, their response to Yahoo's program was favorable.

MIT's Harr Chen would love even more data from Yahoo.

(Credit: Stephen Shankland/CNET News)

"It enables a lot of research that we wouldn't otherwise be able to do," said Harr Chen, an MIT researcher at the event.

If it works out as hoped, Yahoo will make some money out of the program: corporate users who reach large scale with BOSS will have to show Yahoo's search ads. The academic side is a step removed from direct revenue, instead giving Yahoo some prominence with potentially influential thinkers in a market Google dominates. Piquing the interest of researchers at universities with a reputation for incubating the next big ideas is smart, though, and Yahoo and Google themselves both grew out of Stanford.

And honestly, with Google hogging 63 percent of the U.S. search market to Yahoo's 19.6 percent, what does Yahoo have to lose?

"We're not a market leader," said Prabhakar Raghavan, chief strategist for Yahoo Search. "From a strategic standpoint, it does make sense to let other people innovate on top of us. If the pie grows, our share of the pie grows at the expense of somebody else."

The ultimate hope is that BOSS will mean money, too.

Yahoo has made the investment in a massive infrastructure that constantly scans and re-indexes the Web, filters out some of the dreck, interprets search queries, and provides search results in high volume in very short order. This infrastructure is prohibitively expensive for start-ups, just as it is for academic researchers, so Yahoo is letting companies use BOSS as well. Those operating on a small scale may use BOSS for free, but Yahoo requires larger efforts to either show ads or sign a custom revenue-sharing deal.

Mashing up Yahoo results
One possibility for BOSS is that Yahoo's search results can be combined with other data sets. "Other parties may have more info about their users," said BOSS engineer Vik Singh. For example, a social-networking site can track movies or the activities of friends that could be useful in shaping search results. "This is stuff we may or may not have," Singh said.

Prabhakar Raghavan, chief strategist for Yahoo Search

(Credit: Stephen Shankland/CNET News)

Chengxiang Zhai and Bin Tan of the University of Illinois at Urbana-Champaign showed one example of BOSS in action that uses this idea of modifying Yahoo's search results. Their application steered Yahoo's search engine in particular directions based on the data stored on a user's own computer.

In the example, the computer was able to discern what type of jaguar the user was more likely to be looking for--the cat, not the car, or the version of Mac OS X--based on evidence on the computer.

"We believe the client side of personalization has a few advantages over the server side," Zhai said. "It can alleviate concern over privacy and it can provide more information about user activity. And it can naturally distribute computation," so a search company's machines share work with the user's own computer.

Qualitatively different
Researchers could investigate search and related technologies such as natural-language processing (NLP) without BOSS. But with it, that research is vaulted into a different domain. It isn't just a matter of taking more time; with BOSS's vast index of the Web, the possibilities are qualitatively different.

"You gain enormously from access to the data. There are all sorts of things you can do with tons of data" that you can't with a smaller set, said Stanford's Christopher Manning.

Manning works in the active field of natural-language processing, technology that aims to let computers discern the meaning of real human speech or text and that's behind search technology from search start-up Hakia and Microsoft-acquired PowerSet. NLP benefits tremendously from having large-scale data sources, Manning said.

"To understand what words mean, you look at how they're used. We do that on a large scale, (examining) usage and context to learn about meaning," Manning said.

Please, sir, I want some more
It also was clear the researchers' appetites were whetted by BOSS. Nobody sounded ungrateful, but heck, as long as Yahoo is sharing some important data, why not share a little more?

Yahoo is headed that direction. On the research day, it opened up access to another slice of search-related "prisma" data.

Vik Singh, an engineer behind Yahoo BOSS

(Credit: Stephen Shankland/CNET News)

Prisma powers Yahoo's search assist feature that suggests searches based on what people have begun to type into the search box, which can make searching more convenient for users, but for researchers trying to build more technology atop Yahoo search results, prisma data is bigger than that. For example, it can show a search term's variations, its membership in categories such as place names, movies, and government, and the likelihood that people search for the term by itself or as part of a larger query.

"That's got a lot of potential," said Dan Ramage a natural-language processing Ph.D. candidate at Stanford. Ramage said BOSS is useful for his research, which focuses on determining the various relationships that can connect a pair of words, he said, but he'd like it better if he could get better control over the snippets of text Yahoo shows with its search results.

Yes, Yahoo will share more
Yahoo plans to release more. "Over time you'll see we'll offer a lot more ingredients, a lot more power," said Ashim Chhabra, senior product manager with the BOSS project.

Some researchers are hungry for as much as they can get. Chen, for example, hoped Yahoo could become an engine to run software supplied by researchers that plumbs its entire Web index.

"We give you a little code, you run that code on every document, then you give us a number," Chen suggested. It would be useful, for example, "to track evolution of themes and memes on the Web, different buzz trends."

Graham Mudd, product marketing manager for Yahoo search, said the idea is "not as crazy as you think," though he also gave the impression that researchers shouldn't hold their breath for that level of access. But Yahoo clearly wants to offer what he could.

When it comes to search research, "The pool of talent is divided between a half a dozen companies," Raghavan said. "We think it behooves us to open up."

A month after the launch of BOSS--an application programming interface that lets developers build a customized search engine atop Yahoo's technology--the company is showing off mashups built using the product.

Yahoo's Build Your Own Search Service interface allows Web users to build an independent search Web site, send search queries to Yahoo, and process and display the results in various formats, while boosting Yahoo's search-ad business.

So far, the ideas have been related to news and sports search, as well as general search. 4HourSearch, which CNET News touched on briefly last week, took four hours to build with a combination of BOSS and Yahoo User Interface design tools. It spits out Yahoo search results in a style reminiscent of Cuil, and cleverly lists on the front page that it "surfs enough sites."

But some of the more interesting mashups are aiming to aid niche Web users. For sports lovers, the PlayerSearch sports search engine pulls in content from a host of sources, displaying search results in categories such as podcasts, videos, national news and columns, Flickr photos, or stories from The Onion.

The mashup winner, as declared by Yahoo BOSS bloggers, was Dipity, which also showed off its meme timeline Thursday.

The site paired its timeline application with Daylife News, using the BOSS API to make NewsLine. By comparing two topics (John McCain versus Barack Obama) or simply searching for news about one topic, the search results are rendered in timeline form and as a unique perspective.

Yahoo promises that even more mashups, and perhaps new popular search sites, are on the way.

The goliaths of the Internet are dangling an ever-larger supply of bootstraps for folks who want to try new ideas for the Web.

The first case in point is Google App Engine, an infrastructure that lets people run their Web applications on Google's servers, for free up until certain limits are set. Second is Yahoo's BOSS (build your own search service) that lets people extract Yahoo search results, reorder them, and mix them with other content--also without constraint within certain limits.

A rough-and-ready search engine Vik Sighn created to show how a Python programming library used to process Yahoo's BOSS-based search results on Google App Engine.

(Credit: Vik Singh)

On Monday, Yahoo programmer Vik Singh, who has been involved in the BOSS project, released software that lets those two projects work together. Specifically, he adapted a package called the BOSS Mashup Framework (BMF), which provides some pre-written tools to let programmers more easily use Yahoo search data via the BOSS interface, so it runs on Google's App Engine.

"Running BMF on top of Google App Engine is a seemingly natural progression, and quite arguably the easiest way to deploy Boss--so I spent today porting BMF to the GAE platform," Singh said on his blog.

Those tools, called a library, are written in the Python programming language that so far is App Engine's only native language.

Singh also built an example application: the Question-Answering Service. (Don't expect infallibility, but it does answer some questions correctly.) There was a day when this sort of thing, even this imperfect, would require a lot more resources than just a few dozen lines of source code. You'd have to assemble a lot of servers to index the Internet, analyze the results, process queries, and serve up results.

Another example Singh mentioned is called 4HourSearch, so named because it took four hours for programmer Sam Pullara to whip it together, according to his blog. The search site presents a Yahoo-powered interface that mirrors that of Cuil, a loudly trumpeted would-be Google slayer.

It's a pity the National Security Agency can't talk about its computational challenges, because it's leaving a lot of the boasting rights to Google.

(Credit: Paul Ford)

In a blog posting on Friday the company shared some detail about the challenges of one aspect of its search operation, the Web indexing and processing that must take place before the results are delivered to users. The short version: Google has no choice but to think big.

First comes surfing. "We start at a set of well-connected initial pages and follow each of their links to new pages. Then we follow the links on those new pages to even more pages and so on, until we have a huge list of links," said software engineers Jesse Alpert and Nissan Hajaj. "Even after removing...exact duplicates, we saw a trillion unique URLs, and the number of individual web pages out there is growing by several billion pages per day."

Next comes analyzing the "link graph"--the mathematical representation of what links to what. That's a key foundation of Google's PageRank algorithm, which brought the company's search engine to prominence by assigning importance to those pages that other important pages point toward.

In the early days of Google, computing PageRank for the company's collection of a mere 26 million pages took a workstation "a couple hours," and the results would be used for some unspecified period of time. Today, Google surfs the Web continuously and recalculates the link graph "several times per day."

"This graph of one trillion URLs is similar to a map made up of one trillion intersections. So multiple times every day, we do the computational equivalent of fully exploring every intersection of every road in the United States. Except it'd be a map about 50,000 times as big as the U.S., with 50,000 times as many roads and intersections," the engineers said.

Google likes to talk about how users have choice and competition just one click away, and that's a fair point. But the blog post also makes it even clearer just how high barriers to entry are in the search market. That's one of the reasons Yahoo's BOSS (build your own search service) program is intriguing: it lets search start-ups take advantage of Yahoo's crawling, indexing, and search technology in exchange for advertising or revenue-sharing partnerships.