Yahoo beckons coders to gussy up search results

Yahoo on Thursday will take one of its first big steps to make its site a more vital part of the Internet.

The company will offer developer tools to let programmers start using SearchMonkey, technology to make search results more elaborate and, the company hopes, more useful. SearchMonkey lets programmers write applications that can turn dry textual listings in search results into a much more elaborate display, and Yahoo hopes its search business will benefit.

"We want this to be the most productive search experience anywhere," said Amit Kumar, whose title is chief searchmonkey.

In addition to taking the programming tools out of closed testing, Yahoo is opening a "sandbox" where programmers can test their SearchMonkey applications and announcing a contest with $20,000 in prizes to try to jump-start development. Eventually, users will be able to choose the SearchMonkey applications they want from a gallery.

SearchMonkey is interesting, and attracting developers is a time-tested way for technology companies to build stronger, richer technology in years to come. But Yahoo has serious competitive challenges. In search specifically, it continues to lose ground to its top rival, Google, where more than three times as many searches now take place in the United States. According to search market share statistics from HitWise released Wednesday, Google had 67.25 percent share of U.S. searches in April to Yahoo's 20.28 percent, Microsoft's 6.26 percent, and Ask.com's 4.17 percent.

This compares standard and SearchMonkey-enhanced search results.

(Credit: Yahoo)

SearchMonkey isn't the sole part of the Sunnyvale, Calif.-based company's effort to reclaim lost relevance and to placate investors unhappy with the Yahoo's financial performance. SearchMonkey also is the first example of an effort called Yahoo Open Strategy (YOS) to wire the company's many Web properties closer together, expose their features for consumption by outside Web sites, and make them a foundation for programmers who want to build their own applications.

What does SearchMonkey do?
SearchMonkey is an example the latter category, an application foundation. With it, Yahoo lets programmers package some specific search results on Yahoo's site with more sophistication.

For example, a result that lists a local restaurant can feature its phone number, address, and a photo. Or a result with a movie can feature an average reviewer score and nearby theaters.

That's the basic version. Things get a notch fancier with what Kumar calls Infobars, or frames around a result that a user can click for further information. For example, a movie Infobar could expand to show cast and crew information from the Internet Movie Database and a plot summary and movie-ordering button from Netflix.

I see it as similar to building small Web pages into the search results.

Infobars can provide more elaborate information to people who click.

(Credit: Yahoo)

The idea is that publishers will write their own applications that can be added to the search results. They can also promote them to try to get people to install them.

So will your search results become cluttered with results that can look a lot more like flashy ads?

Not if you don't want them to, Kumar said. "Users will quickly be able to remove the apps if they feel they're encroaching into what's supposed to be an ad-free area," Kumar said.

Yahoo might turn some applications on by default, but only if there is strong interest and utility. One candidate is business information from sites such as Yelp or Yahoo Local, he said.

At this stage, Yahoo isn't considering deals by which publishers pay Yahoo to switch on the SearchMonkey applications by default, Kumar said.

"Our first recommendation would be to spread the application virally," he said. "If an application isn't good enough for hundreds of thousands of people to add it, it's not good enough to turn it on."

Monkeying with the Semantic Web
SearchMonkey also is interesting because it fits into the broader sweep of Internet history. Tim Berners-Lee, who initially developed the protocols behind the World Wide Web, has for more than a decade been advocating a move toward a more advanced sequel called the Semantic Web. SearchMonkey specifically takes advantage of Web site features designed to fulfill some of the promise of the Semantic Web.

The Semantic Web is a somewhat counterintuitive concept. Although the Web is geared for human consumption, the Semantic Web seeks to imbue Web pages with meaning that computers can understand. Better computer comprehension, though, will mean better data processing and information presentation for the ultimate benefit of us humans.

Or so the thinking goes. It's been a long, slow shift toward the Semantic Web. But there are real moves under way now, and the promise of more prominent or useful search results at a major site such as Google could further advance the Semantic Web.

The inner workings of the Semantic Web are not simple.

(Credit: Tim Berners-Lee, W3C)

One manifestation of the Semantic Web that's catching on is something called microformats. It's not the full-blown Semantic Web vision (see Berners-Lee's graphic to the right for the whole shebang), but it's a close relative.

Microformats are essentially small labels tucked into Web pages that describe content so that computers can understand the nature of specific bits of data: phone numbers and other contact information, addresses, events, calendar items, for example. In March, Yahoo announced it's begun looking for microformat data as it indexes the Web.

"If there is a structured mark-up you can put on the Web pages, then the machine is going to understand it better," Kumar said. "That's what the Semantic Web is all about."

SearchMonkey is geared to surface micrformat-labeled data, Kumar said. The company hopes developers who build SearchMonkey applications will "help us get the structured data to the users who are going to use it," he said.

One interesting possibility of microformats is that it can take some of the guesswork out of indexing the Web and therefore make accurate search results easier to achieve. Could that help Yahoo catch up to Google?

Possibly. But I suspect there's still plenty of informal information on the Web. And even when microformats provide some structure or guidance, I suspect there's a lot more computer science work to be done in inferring the meaning of Web pages.

[wango2007]

$20,000 for multi-millions worth of R & D creativity? Good one, Yahoo. No wonder you're going down. Only foolish developers would jump on this one.

[igl00lgi]

I know. I don't see it either.

[open-mind]

I prefer search results that are simple, good, and fast ... ala Google. The example search results they show look like an advertisement to me.

[tacit]

I can hardly wait to see how the spammers abuse this.

The problem with machine-readable "semantic Web" ideas is that all these ideas assume the machine-readable tags are honest and trustworthy. In the past, we have seen that spammers and others will abuse any mechanism they can in order to try to boost their search results. Remember meta tags? No major search engine uses them any more because spammers quickly abused them--filling their keyword tags with hundreds, or thousands, of irrelevant but popular keywords in an effort to snag more search engine traffic.

Semantic web space only works as long as everyone is honest and the tags are truthful. When that isn't the case, it all falls down.

[igl00lgi]

Why? WHy would someone smart enough to develop their own applications do so for a controlled platform such as Yahoo, or even Google for that matter? Just like the cloud computing buzz going around, why would I spend my time programming something that in the end I have very little control over? I want to make the next product/company to replace Yahoo not help them survive.

[magicmaster]

Hasn't Yahoo learned the lesson?

Google prospers because it offers minimal graphic while users doing their searches. I would rather sit with bunches of useful text results.

[flickrz]

this is way ahead of time. People are still stupid enough that they like to keep clicking on links after link.