Webware

Yahoo BOSS engineer Vik Singh

(Credit: Stephen Shankland/CNET News)

Here's Web 2.0 at its finest: A Yahoo programmer has combined his own project, Yahoo BOSS (Build Your Own Search Service), with Twitter and Google App Engine to create a new way to determine what news is both new and important.

The service, called TweetNews, presents Yahoo news search results in a different way, using results from the same search on Twitter to determine what should get high placement, according to a blog posting about it by BOSS engineer Vik Singh.

BOSS supplies Yahoo search results in a form that can be repackaged, processed, and published for free, though Yahoo asks for revenue sharing for popular services.

TweetNews combines human interest, as judged by Twitter users, with a measure of authority, as judged by publications that make the cut for Yahoo News search. The application also includes an expandable "related tweets" button that supplies links to people's Twitter references to the various news stories.

"Twitter as a ranking signal for search freshness may prove to be very useful if constructed properly," Singh said in the blog.

Here's a screenshot of the search in action, using the terms "hudson plane" to illustrate the news items Twitter users find most pertinent.

Vik Singh's TweetNews application shows how Twitter can be used to find the most pertinent breaking news.

(Credit: Vik Singh/TweetNews/CNET News)

The application is publicly available as a service running on Google App Engine--not the first time Singh has demonstrated BOSS ideas on his main competitor's application hosting system. Google hosts applications on App Engine for free, but only within various limits, and Singh's However, the application exceeded its quota within a few hours of his posting.

The application isn't just a novel demo, though. It's an attempt to solve a challenging problem in determining what breaking news is most pertinent to people. Here's how Singh describes the challenge:

Freshness (especially in the context of search) is a challenging problem. Traditional PageRank style algorithms don't really work here as it takes time for a fresh URL to garner enough links to beat an older high ranking URL. One approach is to use cluster sizes as a feature for measuring the popularity of a story (i.e. Google News). Although quite effective IMO this may not be fast enough all the time. For the cluster size to grow requires other sources to write about the same story. Traditional media can be slow however, especially on local topics.

I remember when I saw breaking Twitter messages describing the California Wildfires. When I searched Google/Yahoo/Microsoft right at that moment I barely got anything (< 5 results spanning 3 search results pages). I had a similar episode when I searched on the Mumbai attacks. Specifically, the Twitter messages were providing incredible focus on the important subtopics that had yet to become popular in the traditional media and news search worlds.

What I found most interesting in both of these cases was that news articles did exist on these topics, but just weren't valued highly enough yet or not focusing on the right stories (as the majority of tweets were). So why not just do that? Order these fresh news articles (which mostly provide authority and in-depth coverage) based on the number of related fresh tweets as well as show the tweets under each. That's this service.

Yahoo Buzz now shows what your contacts have buzzed.

(Credit: Yahoo)

Yahoo's Buzz service, which lets people spotlight and find interesting Web sites, is now getting a piece of the Yahoo Open Strategy action.

A new Updates section on Yahoo Buzz site now displays sites that a person's social contacts have buzzed, Yahoo announced Thursday. In addition, people can share a personalized Buzz page that shows activity such as what sites they've buzzed, the sites they've voted up or down, and the comments they've left about buzzed sites.

"Yahoo is committed to creating the most relevant experience by enabling social interactions inside and outside of Yahoo's network," the company said in a statement.

The new version of Yahoo Messenger, still in beta testing, also can show what a person's social connections are buzzing and other activity that's shared through the Yahoo Open Strategy.

Note: this story was updated at 12:34 p.m. PDT with analysis and information about Google.

It's like watching a race between two glaciers.

Yahoo and Google are undeniably the two biggest powers on the Internet, each with a vast number of features and users. Each is trying to remake itself during a time when faster-moving start-ups show how hard it is for a massive site to transform itself.

But change is indeed in the works, and given the companies' scale, it's profound for both companies and for millions of people using the Internet.

On Tuesday, Yahoo in effect released version 1.0 of its Yahoo Open Strategy (YOS), providing public access to programming interfaces that will let developers build applications on top of Yahoo sites and build Yahoo information in to their own. The beleaguered Internet pioneer is betting on strategy to increase the activity of existing Yahoo users and draw new users to the site.

The same day, Google added a new option to its Gmail service that can show a user's calendar, list the user's online Google Docs files, and add small Web applications called gadgets. It's a modest move, but it's a new indicator that Google is serious about expanding beyond its core search operation into an all-purpose Internet tool.

Google's move also shows that bonhomie notwithstanding, the two companies both are trying to become a foundation on which people rely for a much richer interaction with the Internet.

YOS 1.0

A key part of the Yahoo Open Strategy is making various Yahoo APIs (application programming interfaces) available that will let Yahoo and third-party programmers take advantage of all the existing social links between Yahoo users. That means a new level of interaction for Yahoo sites, third-party applications running on Yahoo, and third party-sites drawing on Yahoo data--once Yahoo members give their permission for each application.

This diagram shows various components developers can use to work with the Yahoo Open Strategy.

(Credit: Yahoo)

An earlier step of the YOS 1.0 transition was the release of Yahoo's new Profiles site. Another is testing of a more personalized and customizable Yahoo.com front page. And the upcoming version of Yahoo Messenger, currently in beta testing, can tune into the stream of updates from people's contacts.

YOS abilities will arrive in other Yahoo properties in coming months. One big one: Yahoo Mail, one of Yahoo's core applications, one of its most social, and one where people are often deeply engaged rather than just skimming the surface of the Net.

With YOS, Yahoo Mail will be able to house applications that dovetail with people's in-boxes--one contest winner being a tool that assembles all the photos a person sent or received into a photo album. Another, which Yahoo has shown off, can speed interactions with Netflix and expand them to include one's peers.

Details of the Yahoo Open Strategy interfaces are available at the Yahoo Developer Network site. That's also where programmers will need to sign up to participate, Yahoo said. The company plans a series of free workshops to introduce programmers to the technology. Yahoo announced details of the YOS features at its Yahoo Developer Network blog on Tuesday.

Google getting personal
Google doesn't have a grand name for its strategy, but it's nevertheless emerging. The company puts search at the core of its mission, and that's where the vast majority of its revenue comes from. But it's spending that money on efforts to become a much more versatile--and personal--online resource.

New Gmail Labs features bring Google Calendar and Google Docs views.

(Credit: Google)

Gmail is gaining steadily in popularity for Web-based e-mail, though it doesn't match membership totals of the top services, Yahoo Mail and Microsoft's Hotmail.

Along with that are Google Calendars, the iGoogle customizable home page service, newly revamped, and Google Docs for online word processing, spreadsheets, and presentations.

iGoogle is the premier way to house applications at Google, but their arrival at Gmail is telling for two reasons. First, e-mail is a more naturally social activity, so it can make more contextual sense to build on people's contacts. Second, it shows the interface flexibility that Google built into its most recent Gmail overhaul.

That flexibility could be an important asset for the company--and on Tuesday, Google added another experimental option in the form of Labs for Google Apps, which brings some optional collaborative applications to organizations using Google's suite of online tools. These opt-in features can ease the arrival of new features.

The larger a company is, the harder it is to change a site. Even if a new design doesn't break, just retraining users can be a monumental task. It's one reason start-ups can be more adaptable; another is that early adopters who try the latest thing are more tolerant of change than the mainstream users who arrive in force on large-scale sites.

Yahoo and Google both have faced outrage about updates to some high-traffic properties. Google's approach to its e-mail service, though, is to add new features through Gmail Labs, which means people must specifically enable the choices themselves. That gives Google a more graceful way to test and phase in new features.

Some of those Google Labs features have been pretty trivial if not downright silly, but adding Calendars and Docs support helps Gmail grow beyond a communication hub into a general-purpose Google hub. At the same time, iGoogle is headed the same direction, with the ability to house applications for Gmail, calendars, and instant messaging.

Outside help
What's so different about the companies' current efforts to remake themselves is the involvement of others. The companies are opening up their Web sites to third-party programmers because they're trying to capitalize on others' work.

Conveniently for programmers, Yahoo and Google both support the OpenSocial standard for Web applications, which at least in theory should make it easier for third-party programmers to work with either company as well as with other allies such as MySpace. But the area is still immature

It's all cooperative, of course. An outside programmer can get exposure to huge numbers of users by riding Yahoo's or Google's coattails. Those outside programmers, if they do their jobs right, will help bring new uses, new users, and new activity to their hosts.

And, Yahoo and Google no doubt hope, speed up those glaciers.

SAN FRANCISCO--Phase one came last week, when Yahoo launched its new profiles site. Phase two begins next week, when Web developers can start sinking their teeth into Yahoo's attempt to replace its present static design with one that's customizable, application-rich, socially connected, and woven into other parts of the Internet.

Developers are essential to what the company calls the Yahoo Open Strategy. Yahoo is building the foundation, but it will be the arrival of others' applications that will show whether Yahoo's transformation attempt is fulfilling those hopes.

Ash Patel, head of Yahoo's Audience Products Division

(Credit: Stephen Shankland/CNET News)

"That starts changing Yahoo from a walled garden to the best of the Web," said Ash Patel, executive vice president of Yahoo's Audience Product Division, speaking to reporters at Yahoo's Brickhouse site here Friday. Patel has a heavy burden: in his new role, he's responsible for a major part of Yahoo's attempt to reverse its fortunes amid a rough economy.

If the strategy works, more people will use Yahoo, and they'll use it more deeply. "We should see a lot more time spent and bigger engagement with the front page and mail and My Yahoo," Patel said. "The average Yahoo user who may use two or three things (today) will now start using four or five or six things."

Applications using the Yahoo foundation can run at Yahoo or outside it, and Yahoo will release a software developer kit to help programmers get started.

For example, when a commenter is posting on a publisher's Web site, the publisher could offer the commenter an option to have that activity broadcast on his stream of activity on Yahoo. That would let the commenter share what he's up to with his contacts while exposing the publisher's site to more potential readers.

Another example--indeed, the winner of the Yahoo Open Hack 2008 programming contest augmented Yahoo Mail to present all photos a person has sent or received into photo albums. More photos are shared daily on Yahoo than are uploaded to the company's Flickr photo-sharing site, Patel said, so moves like this could open new windows of activity on Yahoo properties.

New developer tools
Yahoo has opened some developer-oriented projects already, notably BOSS (Build Your Own Search Service) for repackaging Yahoo search results, and SearchMonkey for adding new depth and pizzazz to Yahoo's search results, but those were narrower in scope. At some point next week--Yahoo won't promise which day exactly--the more powerful tools will go live at the Yahoo Developer Network.

Neal Sample, Yahoo's chief architect for platforms

(Credit: Stephen Shankland/CNET News)

There are three broad categories of technology that developers will get access to next week. At the base is a social platform that applications can use to draw upon Yahoo users' social connections--as long as users have given permission. While sites such as Facebook, LinkedIn, and MySpace capitalized on the social-networking phenomenon, Yahoo argues that it already has the social data built into its properties. It's now a matter of bringing it to the fore so applications and users can draw on that information.

"The idea is to create a single social experience that can be shared," said Jay Rossiter, head of the Yahoo Open Strategy.

One oft-cited example is a revamped Yahoo Mail that spotlights mail from people's close contacts. If you spend a lot of time e-mailing your boyfriend, mom, or college roommate, chances are you'll want to know when they e-mail back.

One level above the social plumbing is the foundation for running applications, called the Yahoo Application Platform. Initially, Yahoo will house standalone applications, but as third parties' products mature, they'll also be able to run on Yahoo users' profile pages, My Yahoo pages, and other locations. Some will even run on the Yahoo.com home page, as long as they can meet tough requirements for high performance.

This diagram shows various components developers can use to work with the Yahoo Open Strategy.

(Credit: Yahoo)

And the third level is the services level. Here, Yahoo provides the Yahoo Query Language, a close relative to the Structured Query Language many use to extract data from databases. YQL is designed to make it easier for programmers to extract and process data from Yahoo and many other Web sites, and Yahoo says it'll do the heavy lifting to make the data workable through YQL.

Overshare?
Of course, users might get the willies thinking about just how much their own activity is becoming part of the information flow of the Internet. Do you really want an application sharing what you do with your friends or indeed the entire world?

Yahoo doesn't want any privacy surprises, though. Each new application must declare to the user exactly what Yahoo services it wants to use and must obtain the users' permission to do so through a "scary" warning screen: the more services, the more exclamation mark alerts are shown--an interface designed to encourage developers to use the bare minimum and to ensure that users know what they're getting into, said Neal Sample, Yahoo's chief architect for platforms.

"Yahoo's going to put up essentially another skull and crossbones" for each service the application uses, Sample said.

And users will have fine control over what's shared or not. People will be able to broadcast what music they're listening to publicly while confining their movie habits only to close friends, for example.

Some socially connected services will require signed-in participation from both a Yahoo user and outsiders. For example, a person could selectively share photos without making them public, and those viewing the photos would have to sign in. Today, such a move requires that all people be Yahoo members, but the company will add a fast, lightweight registration process that can use any e-mail address.

Yahoo, of course, hopes receiving invitations from Yahoo members effectively will upsell those outsiders to Yahoo services. "It's valuable for Yahoo to have a way to draw more users into Yahoo," Rossiter said.

Yahoo has finished a redesign of its Flickr home page that emphasizes the photo-sharing site's social aspects.

The new home page shows off more of a user's own photos and more from the user's contacts, and it surfaces social activity such as comments on the user's photos, replies to comments the user made on others' photos, and new photos posted to the user's Flickr groups. (See a screenshot below.)

The move is part of Yahoo Open Strategy, which aims to expose Yahoo users' social activity across different Yahoo properties, let others build applications on Yahoo properties, and let outside sites use Yahoo data. Next up for Flickr is a redesign of the photo pages that house each image, the company said earlier.

Yahoo offers a screencast describing the new look on its Flickr blog.

The redesigned Flickr shows more photos and, through a 'recent activity' tab, more social interactions. (Click to enlarge.)

(Credit: Stephen Shankland/CNET News)

Yahoo has broadened use of its SearchMonkey technology to spruce up search results with specific information from Citysearch and Zagat, the company said Thursday.

SearchMonkey draws upon "semantic Web" information used to help computers better understand the data on Web sites. SearchMonkey can build that data, such as movie ratings or restaurant phone numbers, into the search results Yahoo presents. (See an example below.)

But SearchMonkey-enhancd results won't show unless a user turns on a specific application--or unless Yahoo turns on that application by default. Now it has with the applications from Zagat and Citysearch, so some search results can be framed with those companies' extra information about various businesses.

The move is part of the Yahoo Open Strategy. Yahoo announced the SearchMonkey move on its blog on Thursday.

A Yahoo search for 'Bacar San Francisco' shows Citysearch and Yelp results spruced up via Yahoo's SearchMonkey technology.

(Credit: CNET News)

SUNNYVALE, Calif.--On Friday, 300 programmers will descend upon Sunnyvale, Calif., to plant the seeds of what Yahoo hopes will be an answer to Google's Internet might.

Yahoo co-founder and Chief Yahoo David Filo

(Credit: Stephen Shankland/CNET News)

The event is called Open Hack Day 2008, and at it the coders will be the first from outside the company to get their hands on a number of programming interfaces Yahoo is releasing in an attempt to enliven its stodgy but still powerful Internet properties.

There's no guarantee that the release, a key step in what the company calls its Yahoo Open Strategy, will improve Yahoo's financial misfortunes. But it holds promise a strategy that could help Yahoo without having to try to out-Google Google.

That's because YOS marries the best of what Yahoo is with the best of what's happening on the Internet today. More than 500 million people come to Yahoo sites each month, 300 million of them registered users who log on, and they're coming for Yahoo's content and services. Yahoo may not be able to match Google's search engine and accompanying search ad money machine, but YOS ultimately could help improve Yahoo's assets, attract new partners, and bolster the company's advertising revenue.

"We believe openness is going to happen with or without us. We'd rather be at the center of it," said co-founder David Filo in an interview.

Here's an example of YOS in action that Ash Patel, head of Yahoo's audience products division, showed Thursday. The Yahoo home page, which is being revamped to show content customized for each user, houses an application from Netflix showing the movies a user ordered and new recommendations. Yahoo search is augmented to let people order more movies straight from the search results. And an application within Yahoo Mail could let users rate their movies and chat with Netflix members on their buddy list who've already seen it.

Ash Patel, head of Yahoo's audience products division

(Credit: Stephen Shankland/CNET News)

The pressure's on
Here's the rub, though: YOS will take time to build, and time is not on Yahoo's side.

Yahoo, faced with near-term pressures from Microsoft and activist investor and now board member Carl Icahn, would have preferred a quick fix to its business, and perhaps a cash infusion from Yahoo's search-ad partnership with rival Google will help in that regard.

YOS is a longer-term strategy, though. It's taken months to rewire the company's infrastructure to accommodate the vision. It'll take more months to coax programmers and business partners into using it. And still longer to attract Yahoo users to adopt the new features.

As Yahoo languished in recent years, new online services squarely in Yahoo's back yard, such as Facebook and Twitter, had time to put down serious roots. And of course Google has encroached, too: its search-ad revenue has funded any number of affronts to Yahoo, including Gmail, Google Docs, Google News, Google Finance, Orkut, and Blogger.

Of course, Yahoo believes that its clout on the Internet will give it the necessary leg up. So the next start-up, for example, could get traction quickly by drafting off Yahoo's page views and user base.

Yahoo CTO Ari Balogh

(Credit: Stephen Shankland/CNET News)

"Yahoo allows developers to create applications for the world's biggest audience," Patel said. "And they're able to do monetization for advertising. Those two are huge value propositions."

The company is betting the money will come its way, too. Yahoo expects to gain better insight into what users are doing, and consequently better predict what sorts of content or advertising the users want. "The better signals you have, the better you can serve the right content," and being able to target ads better means Yahoo can charge higher rates, said Chief Technology Officer Ari Balogh. "We expect lots of material benefit."

What's coming Friday
So what will be new on Friday? For those programmers who made the Hack Day cut, a pizza- and soda-fueled opportunity to toy with two broad categories of new Yahoo APIs (application programming interfaces), said Neal Sample, chief architect for Yahoo's platforms.

First is a collection of social APIs that let programmers use data such as a Yahoo user's address book contacts, status messages, profile information, and news feed items. Second is the Yahoo Application Platform (YAP), which will be used to write the applications that actually will run on Yahoo Web pages. YAP has some similarities to the OpenSocial project initially begun by Google but now supported by several others, including Yahoo.

The first Yahoo property to get the application support will be a redesigned profile page, a "control panel" site where people can record personal information, update their status, and see their social connections, Sample said. "We're going to get to the point where all our profiles can start coalescing so you have the concept of a single identity on Yahoo."

Neal Sample, Yahoo's chief architect for platforms

(Credit: Stephen Shankland/CNET News)

Over this year and into 2009, the support will extend to the My Yahoo personalized home page, Yahoo Mail with its 270 million users, and the Yahoo front page that's being redesigned with a customization feature called the content optimization engine.

If all goes according to plan, the collection of new interfaces and applications will "light up a user's social graph," building Yahoo more deeply into a person's online interactions, for example by spotlighting a person's most important contacts in Yahoo Mail.

Fresh air
Yahoo will call Open Hack Day a success if it produces developers happy with the company and feedback about the interfaces, said Chris Yeh, head of the Yahoo Developer Network. But there's something in it for Yahoo, too: a breath of fresh air. "Big companies do become insular at times...We do everything we can to try to avoid that."

Programmers not at Open Hack Day will only be able to see the API documentation at first, but the final APIs will be public soon. "In a few weeks, we're making them generally available," after Yahoo gets feedback from the early testers, Sample said.

These new programming interfaces will join other parts of YOS already released recent months: SearchMonkey lets people write applications that spruce up search results with elements such as LinkedIn profiles or restaurant reviews. BOSS (Build Your Own Search Service) lets others build their own search engines on Yahoo's, reordering or modifying results however they want and sharing search ads or revenue if they get popular.

Chris Yeh, head of Yahoo Developer Network

(Credit: Stephen Shankland/CNET News)

Newest is Fire Eagle can keep track of a person's location information, including a mechanism to let users control what services may employ that information.

A few other APIs are planned for later release, Sample added, including some for geographic services.

Proceed cautiously
Embracing openness--standards, open-source software, open interfaces--is a tried and true way technology companies try to leapfrog incumbent competitors. But retrofitting openness to a company that's been closed is difficult, and Yahoo clearly is concerned about breaking what it's built by moving at start-up speeds.

"Getting it right with hundreds of millions of users is harder than if you're starting from scratch," Filo said.

And it's not just about revealing APIs and doing some marketing. "We have to get the platform right so we can ensure the applications don't degrade the user experience," Patel said, for example by caching applications on Yahoo servers so pages load fast. "It is stuff that does keep us up at night."

Consequently, Patel said, the company will vet applications before letting them onto Yahoo sites--especially for Yahoo Mail site, where so much personal information resides.

Venkat Panchapakesan

(Credit: Stephen Shankland/CNET News)

Yahoo also wants to ensure users are in control when it comes to the permissions they grant to applications, said Venkat Panchapakesan, leader of Yahoo's audience technology group.

So it's tricky work for a lumbering giant. If successful, though, Yahoo will be able to reclaim some of the Internet initiative it once had in spades, potentially rearranging today's competitive landscape.

"Yes, we have lots of competitors," Filo said. "In some ways, we're opening up new level of competition by letting people build on top of us. Ultimately, this is good for the consumer and the Internet."

See also:
• Yahoo gives a taste of its 'open' overhaul
• David Filo: No browser for Yahoo
• Yahoo makes the case for Google search ads
• Yahoo 12-month price target cut
• Yahoo announces social networking app for iPhone
• Top Yahoo sales execs: One in, one out

Yahoo has released an API (application programming interface) that lets outside Web sites automatically extract information such as top 10 lists from the Yahoo Music site.

(Credit: Yahoo)

For example, a programmer can use the service to search for the numeric ID that Yahoo gives a particular musician, then use that ID to retrieve all the albums by that artist or to retrieve a list of artists Yahoo deems similar, according to self-described Yahoo music nerd Jim Bumgardner on the Yahoo Developer Network blog. The item ID for a particular video can be used to retrieve a thumbnail image for the video or the video itself.

There's a limit of 5,000 queries per day with the API.

The API is an example of Yahoo Open Strategy, the company's effort to open up its infrastructure to use by outside Web sites and to permit more third-party developers to build software that runs directly on Yahoo's sites.

For full details, see the Yahoo Music API site or its developer guide

Update 12:41 p.m. PDT: I corrected a reference that should have been to Yahoo.

Yahoo has begun sharing some future plans it has for Groups, its service where people with shared interests can get together online through mailing lists, calendars, polls, and other features.

In the "coming year," Yahoo plans to add many attributes that expand the scope of groups, according to the Yahoo Groups blog on Tuesday. Those features include tools for product reviews, service directories, wanted boards, address books, and event planners.

And upgrades to existing features include: a better system for hosting photos that permits more storage and larger pictures; better message boards; the ability to store e-mail attachments; and the ability to set the site for non-English languages.

There's no official word about whether there will be room for some of the sharing and collaboration that's coming with Yahoo Open Strategy on Groups, but it seems possible to me. For example, hosting OpenSocial widgets that are available to members of the group seems a natural fit.

Yahoo already has launched a Groups Lab, a team that creates new features such as the recently released People Map beta test that shows group members' locations on a map.