Business Tech

Betting that the benefits of the move will outweigh the risks, Yahoo has released the source code underlying in-house software called Traffic Server that can speed up Web site operations.

The software works by moving some data and operations closer on the Internet to the people trying using those services. Yahoo released it as an "incubator" project under the auspices of the Apache Software Foundation, a seasoned organization for managing open-source projects and also the site that houses the Hadoop open-source project Yahoo favors for large-scale data-processing challenges.

Shelton Shugar, Yahoo's senior vice president of cloud computing, plans to announce the move at the Cloud Computing Expo in Santa Clara, Calif., on Tuesday in a keynote speech, but the software actually arrived at Apache last week.

Shelton Shugar, Yahoo's senior vice president of cloud computing

(Credit: Screenshot by Stephen Shankland/CNET)

"We've donated Traffic Server to Apache because we think it's a great piece of code, and we want to build a community around that in the same manner we built a community out of Hadoop," Shugar said in an interview.

Traffic Server is a battle-hardened package with more than 200,000 lines of C++ code. Yahoo originally got the software through its acquisition of Inktomi earlier this decade, and it's been using it ever since. Today, the software delivers 30 billion Web objects and 400 terabytes of data each day.

And Yahoo can rightly be proud of Traffic Server's performance: that comes from a surprisingly small number of Yahoo servers--between 100 and 150, said Chuck Neerdaels, vice president of data services at Yahoo. The software is set up particularly to run multiple tasks at the same time, a design well-suited to today's servers with multicore, multithreaded processors.

Source code is what humans write in a higher-level programming language; only after it's been translated into binary machine code can a computer actually run that program. When associated with an open-source project, this software is available for anyone to see, modify, and distribute, in contrast to the locked-down world of proprietary software such as Microsoft Windows. So in effect, Yahoo is allowing others not only to use Traffic Server for their own ends, but also to modify it--for example, by taking advantage of its ability at to accept plug-ins that can adapt it for different tasks.

Giving away the farm?
So isn't there a risk that Yahoo is giving away some pretty important technology that's central to its business? Plenty of start-ups today are trying to grow to Yahoo's scale, and many of them are competitors.

Some Yahoo rival might very well gain as a result, but on balance, the company thinks that it'll come out ahead. For one thing, Traffic Server in isolation is not as powerful as Traffic Server woven into Yahoo's computing fabric, the company argues.

"What we're giving up is a generic building block. What makes it really interesting at Yahoo is how we've connected it with other things to make a bigger service," Neerdaels said. As for Yahoo's major rivals: "We suspect our larger competitors already have some solution they're happy with."

Yahoo expects a number of benefits from broader development and use of Traffic Server.

"We think a lot of folks can benefit from this, and by raising the tide, we think we can benefit as well," Shugar said.

For one thing, making Traffic Server open-source software will mean that people will grow familiar in its use, making it easier for Yahoo to hire engineers who already are up to speed.

"By virtue of basing services on open-source software, we attract people who want to work on open source. They like it, and they like the idea of it. It's a skill they can take with them from one place to another," Shugar added.

For another, Yahoo can benefit from others adapting the software to a broader range of uses, he said.

Gaining influence among developers
There are intangible benefits, as well, when it comes to recognition among programmers, whose influence in some ways makes them the digital elite. Microsoft long ago learned that much of its power comes from developer allies, and Google is trying to put that lesson to good use as well by releasing many open-source projects--Google Chrome being one recent example.

Yahoo isn't in the business of selling technology to others in the manner of Amazon Web Services, Microsoft Azure, or Google App Engine. But having solid technology is essential to Yahoo. While it's willing to sell its search business and engineering skills to Microsoft, it still needs in-house expertise to power its many Web properties and to reduce its operating costs.

Here, Traffic Server is important. For example, one area where Yahoo uses Traffic Server was at Yahoo Sports for handling scores. A regular Web server sends out the Web page to a person's browser, but Traffic Server handles the JavaScript technology that periodically refreshes the contents of a scoreboard element on that page.

It's only a "trickle" of data, but at Yahoo's scale, that can be some pretty heavy work. "When they moved to using the Traffic Server front end, they shaved something like 200 machines off their back end because session management was more efficient," Neerdaels said.

Another part of Yahoo operations retrofitted with the software is Yahoo Mail, he said. Traffic Server can be used to process the cookie text files on a person's browser to figure out whether that person can be logged in automatically or the person needs to authenticate anew. It also can route traffic appropriately when, for example, a person who is "homed" to Yahoo's servers in India visits the site while in the United States.

Traffic Server also manages a lot of more nuts-and-bolts tasks. For example, it can cache Web data closer to browsers so the original Web servers that house the data aren't as overtaxed. And it can store a Web address stored in the Domain Name System to speed up network speeds.

What's it good for?
Some of these chores can be handled by existing software, such as Squid, which is already open source. But Yahoo is on a roll with its open-source work, as the company seeks to advance its internal cloud-computing infrastructure. Expect more to come.

"As various pieces of our cloud get to a point of maturity, we will open-source specific pieces," Shugar said. Future candidates include Yahoo's foundation for hosting its Web applications on a virtualized, more flexible foundation, and its Sherpa and Mobstor services for storing data.

Winning open-source allies can be difficult, and Neerdaels said it takes an engineer a good six months to fully comprehend all Traffic Server's code, so immediate gains beyond fostering goodwill are unlikely.

But in the long run, Yahoo's program could pay significant dividends. Building a series of significant open-source packages could lead to a Yahoo infrastructure that's high-power but more standard than custom-made.

It's not every day that large, significant software packages arrive on the Net in open-source form--much less a series of them that are increasingly relevant to a competitive market of large-scale Web sites.

In this case, Yahoo's gift may indeed become Yahoo's gain.

There was a day when information technology personnel toiled behind the scenes to make their corporate computing infrastructure work.

But in the Internet era, those experts increasingly are getting starring roles in corporate computing leadership rather than being supporting cast members. Such is the case for Shelton Shugar, Yahoo's senior vice president of cloud computing.

"It becomes more a topic at cocktail parties," he said of his present job, which he took shortly after Yahoo formed the group in June 2008. "I was at a wine tasting, and an acquaintance said, 'I did a search on your name and found cloud computing. What's that all about?'"

Shugar isn't running any publicly available cloud computing services, be they nuts and bolts like Amazon Web Services or full-on applications such as Google Docs. Instead, he's in charge of building a computing infrastructure crucial to Yahoo's ability to operate at large scale and improve its services rapidly. Those are essential for the company's attempt to fend off Google, which arguably is nimble for its size, and start-ups such as Twitter or Facebook that can change course a bit more easily.

Shelton Shugar, Yahoo's SVP of cloud computing

(Credit: Screenshot by Stephen Shankland/CNET)

"Most Yahoo properties you interact with use the cloud to some extent," said Shugar, who came to Yahoo from eBay and who's a headliner at the Cloud Computing Expo in November. "Over time, that percentage will continue increasing. Almost anything you touch uses some of it."

Most telling about his role: although rebuilding Yahoo on its own cloud-computing foundation is expected to save some money, the primary motivation is to liberate the company's programmers from the difficulties and drudgery of coding for gargantuan audience on the Internet.

"If we have a thousand developers who no longer have to build a lot of infrastructure, and who (instead) work on products and features, that puts us way farther ahead than squeezing a few nickels out of the infrastructure," Shugar said.

It's a mammoth chore. For example, when it comes to background data processing that underlies search results, behavioral ad targeting, site trend spotting, spam filtering, and Yahoo.com content selection, Yahoo uses open-source software called Hadoop.

"I've got 25,000 machines running Hadoop," Shugar said. They're divided into several "grids," the largest with about 4,000 servers. "It's a fascinating activity...watching the different dynamics of usage push things in different ways. Some tasks are very heavy on computation, some are heavy disk input-output. It's pretty complex."

Yahoo's 'private cloud'
Cloud computing is a popular buzzword these days, and as a result it means many things to many people. It generally refers to moving services to the Internet, which in innumerable technical diagrams over the years has been represented by, in fact, a cloud.

Cloud computing has grown less ephemeral in recent years, though. The Amazon Web Services suite offers a highly specific set of interfaces that Internet operations can use for everything from data storage to computing capacity. At a higher level, Google Docs lets people perform word processing and spreadsheet calculations through a Web browser. Microsoft is spanning that spectrum, working on its Azure foundation for generic Windows Server chores and on a Web-based version of Office.

One somewhat controversial concept is that of a "private cloud"--computing services that embrace some of the principles of publicly available clouds but that are used just by one company. If it's in-house only, what makes it any different from just the IT department's computers?

Yahoo makes a reasonable case that it's got a cloud of its own, though. It operates on a scale larger than many public cloud companies, and it embraces some of the principles cloud computing in that infrastructure.

For example, it's got a variety of interfaces that many Yahoo services can use--a concept often called multitenancy--so they don't have to build them on their own. For another, it's global, handling thorny issues such as operating at large scale and replicating data for reliability and responsiveness. And it's got a degree of elasticity built in, so the infrastructure can expand, contract, or otherwise adjust to changing work load demands.

This computing foundation is designed to ease the pain of developing Yahoo services. Some features and projects are easy to build at a small scale but hard to expand.

"It's like the cat going up the tree. It looks good at the beginning," Shugar said.

Yahoo's cloud services
Yahoo has four cloud services in varying degrees of availability:

• Operational storage, for housing data such as e-mail attachments or a user's social connections.

• Batch processing, for crunching oceans of data to sort Web search data, tailoring content and ads for individual users, and figuring out who's spamming Yahoo Mail.

• Edge content, for presenting Web pages, balancing the load from many users, caching data for fast use in many areas. This service is already in widespread use.

• Online serving, a flexible foundation for designing and housing complicated applications.

This last service is under development, due to arrive in 2010. "We're working with a few Yahoo properties as anchor tenants," Shugar said, declining to say which exactly.

The online serving interface will be based on Red Hat Enterprise Linux running atop the open-source Xen virtualization software offered by Citrix, Shugar said. "Virtual machines are going to get more and more and more important."

Though open-source software is freely available for the do-it-yourself crowd, Yahoo is in negotiations with some outside parties that will involve commercial relationships, he added.

Linux and virtualization
With virtualization, one server can house multiple operating systems at the same time, and operating system instances can be shuffled from one physical machine to another to adjust to changing demands. Yahoo has worked with virtualization leader VMware, but the company wants the nitty-gritty control enabled by open-source software.

"We need to be able to tweak it quite a bit for performance, to match it with our hardware," Shugar said.

Open-source software helps with Hadoop, too. Yahoo is the primary contributor to the project at the Apache Software Foundation, but others' participation helps ensure Hadoop stays generally useful.

Most people aren't going to wire up their own Hadoop computing cluster, of course. But "data center" no longer is a term understood just by server administrators and CIOs.

"There are these big data centers behind the scenes supporting all these searching, social networking, information-gathering activities," Shugar said. "Before, it was a business function inside a company. Now the awareness has increased as result of people using the Web. It becomes a topic of general interest."

Yahoo on Tuesday released version 3 of its Yahoo User Interface library, a software collection programmers can use to endow Web sites with fancy user interface elements written in JavaScript and Cascading Style Sheets.

"YUI 3's core infrastructure and its utility suite are all considered production-ready with today's release," Yahoo's Eric Miraglia said in a blog post Tuesday. "The code we're shipping today in 3.0.0 is the same code that drives the new Yahoo Home Page, and it goes out with confidence that it has been exercised vigorously and at scale."

The YUI libraries are open-source, freely available, and used widely around the Internet for Web site tasks including animation, drag-and-drop, fetching data from various types of sources, and responding to events--chores that are more complicated but that often are useful as the Web moves from static Web pages toward interactive applications.

Compared with YUI 2, the new version is smaller, faster, easier to program with, and more secure, Yahoo said. It's easier to break code into minimum-size pieces through a dependency configurator or YUI's ability to download required components on its own. Also, Yahoo is working to add widget abilities for creating small programs.

A convenient YUI feature is that Yahoo is willing to host most of it on its own servers, saving hassle and Web server bandwidth.

The new version is the first ground-up reworking of the software since 2005. No doubt YUI will be the subject of discussion at Open Hack Day and YUIConf, both in October.

This chart illustrates the smaller file sizes of one YUI 3 libary that helps with retrieving data from various sources.

(Credit: Yahoo)

The platform wars are over, and open source has won. It's not that open source has displaced Windows or the iPhone or anything else, but that every platform will necessarily include open source. It's simply too expensive and too difficult to go it alone anymore, whether you're an aspiring start-up or Microsoft.

IDC captures this thought in a recent Asia Pacific survey, which highlights open-source software as a foundation for flexible platforms, rather than as point solutions:

Vendors position [open source] as a solution, rather than a point product, by customizing to the needs of specific verticals....Other perceived benefits of adopting open source, apart from the traditional cost savings, include no vendor lock-in, access to the source code, and the flexibility to further customize the software to match individual needs. All these in turn increase the ease of integration with the existing infrastructure of an organization, as well as the compatibility with different platforms. This gives the organization an opportunity to use and test open source without changing their whole IT infrastructure.

It's this flexibility that is arguably open source's biggest benefit, and why companies like Yahoo are actively contributing to open-source projects. Yahoo's senior vice president of cloud computing, Shelton Shugar, argues,

We believe that the developer community is a key component in making Yahoo! a success. The challenges the industry is facing today in terms of large-scale, global cloud solutions are bigger than any one company (big or small) is able to solve on its own. As we contribute to the [open-source] community, we also learn from the community, and third party developers are a valuable resource helping to speed innovation.

Companies that care about developers must care about open source. Like Amazon with its Kindle. Or Microsoft, whose CEO famously sang the praises of developers. So long as Google and its crowd compete using open source, Microsoft will, of necessity, follow suit.

It's not about peace and love. It's about capitalism and competition. That's the new face of open source.

This isn't to suggest that the world will go 100 percent open source tomorrow. But we'll see a lot more open source as vendors strive to meet CIO's need to cut costs while boosting productivity, and as they seek to become flexible platforms to meet the demands of increasingly complex enterprise IT requirements.

Credit: Stephen Shankland/CNET

Brad Garlinghouse

AOL announced Tuesday that it has appointed former Yahoo executive Brad Garlinghouse, famed for his "Peanut Butter Manifesto" at that company, as the new president of its Internet and Mobile Communications segment.

Garlinghouse also will run AOL's Silicon Valley operations from its Mountain View, Calif., headquarters and serve as the West Coast lead for AOL Ventures, the company's venture capital arm. He will report directly to AOL Chairman and CEO Tim Armstrong, who was named to those posts in April.

Garlinghouse's most recent position was as an in-house senior adviser for Silver Lake Partners.

"In addition to leading our efforts to grow our communications products, Brad will be bringing his global leadership and business experience as a key member of our company's executive leadership team," said Armstrong. "He will also be a major force for AOL in Silicon Valley, working to expand our presence there and in the tech community in general."

A former Google executive, Armstrong faces the daunting task of reviving AOL, a company once nearly synonymous with the Internet for many people but which, in recent years, has strugged with fleeing subscribers and declining sales. AOL's blockbuster marriage with Time Warner never worked out, leading inexorably to the announcement in May that AOL would once again become a separate company.

At his recent 100-days-at-AOL strategy summit, Armstrong identified communications as one of AOL's five key focus areas.

Garlinghouse knows all too well what a lack of focus can to do a business. In his Peanut Butter Manifesto in late 2006, he complained of Yahoo--which has gone through its own series of troubles and reorganizations: "We want to do everything and be everything--to everyone...The result: a thin layer of investment spread across everything we do and thus we focus on nothing in particular."

Starting in 2003 as vice president of communication products for Yahoo, Garlinghouse climbed the ladder to become a senior vice president for two other communications segments. He has also overseen the company's Flickr photo-sharing service and Yahoo Groups.

Garlinghouse left Yahoo in June 2008, at a time when the company was shedding executives at a rapid rate.

Yahoo has signed an agreement to buy Maktoob.com, the leading online community in the Arab world with more than 16.5 million users.

Through the acquisition announced Tuesday, Yahoo will join its own services with Maktoob.com's local content, providing people with Arabic-specific information and Arabic versions of Yahoo Messenger and Yahoo Mail.

Popular in such countries as the United Arab Emirates, Jordan, Kuwait, Egypt, and Saudi Arabia, Maktoob.com says it reaches one-third of those online in the Arab world.

And there is room to grow. Citing figures from the World Bank, Yahoo noted that more than 320 million people worldwide speak Arabic, but less than 1 percent of all online content is in Arabic.

"Access to information and communications tools can positively impact people's lives in many ways," Yahoo CEO Carol Bartz said in a statement, "and with the acquisition of Maktoob.com and our investment in the region, the Arab world will soon get a Yahoo experience in Arabic with relevant local language content, programming and services."

The two companies are also touting the purchase as an opportunity for advertisers to access a new market. Yahoo cited statistics from Madar Research, which predicts that spending on Internet advertising will rise by 35 to 40 percent in the Arab region this year.

"Internet users in the Arab world will have access to Yahoo's vast content portfolio, as well as world-class communications products, which will be available in Arabic for the first time," Ahmed Nassef, general manager of Maktoob.com, said in a statement. "In addition, advertisers will be able to leverage the vast reach of the newly combined audiences to effectively market to consumers across the region."

The acquisition is expected to be completed in the fourth quarter, after which Maktoob.com will become a wholly owned subsidiary of Yahoo. Financial terms of the acquisition were not disclosed.

Maktoob Group, the parent company of Maktoob.com, was founded in 2000 and is based in Amman, Jordan. After the Maktoob.com acquisition is wrapped up, the remaining Maktoob Group companies--which include Souq.com, CashU.com, Araby.com, and Tahadi.com--will function under a newly created entity called the Jabbar Internet Group.

As finalization of a Microsoft-Yahoo search deal reportedly nears, activist investor Carl Icahn--who played a key role in trying to broker a broader partnership between the companies last year--is speaking out in favor of such an agreement.

"I've been a strong advocate of getting a search deal done with Microsoft," Icahn, who owns about 5 percent of Yahoo and sits on its board, told Reuters in a phone interview Friday. "It would enhance value if a deal got done, because of the synergies involved."

According to an All Things Digital report late Thursday, several top Microsoft players--including online executives Yusuf Mehdi, Satya Nadella, and Qi Lu--are in Silicon Valley to try to finalize a search deal with Yahoo.

The report says the two sides are "down to the short strokes" after years of closely watched on-again, off-again talks. A deal could come within a week, All Things Digital said.

Icahn, for his part, wouldn't comment on where the latest supposed negotiations between Yahoo and Microsoft stand, according to Reuters. Icahn was a central figure in Microsoft's highly scrutinized $47.5 billion takeover bid for Yahoo, which fell apart last November.

During the negotiations, he launched a proxy fight in a bid to take over Yahoo's board. Among his wishes was that then-CEO Jerry Yang step down. The company and Icahn eventually reached an agreement that got him a seat on the board, and the number of seats was expanded, with Yahoo appointing two new members from Icahn's slate of candidates.

Since the full-out acquisition fell through, both Microsoft CEO Steve Ballmer and current Yahoo CEO Carol Bartz have indicated they are open to some sort of a search deal.

As my CNET News colleague Ina Fried pointed out, with Microsoft's Bing getting some good reviews and Microsoft having billions in cash on hand, the pieces would seem to be in place if both sides have the will to make it happen.

Bing may be catching on as a new search engine, but it has yet to generate growth in ad dollars for Microsoft, according to a report released Tuesday.

Microsoft's share of search engine ad spending for the second quarter stayed flat at less than 6 percent, according to the report by research firm SearchIgnite. That's the same level it's been for the past several years.

"Microsoft appears to be focusing its efforts on driving consumer interest and capturing increased search query share," said Roger Barnette, president of SearchIgnite. "We have not yet seen this translate into more paid search advertising dollars for Microsoft, although typically consumer adoption precedes advertiser adoption."

Meanwhile, Microsoft's one-time acquisition target Yahoo lost ground in the second quarter, with only 17 percent of the market. But top dog Google continued to rise, grabbing a 77 percent market share for search engine ad spending.

(Credit: SearchIgnite)

However, don't count out Bing just yet. The report noted that research groups have tracked Bing's share of the search query market growing since its launch last month. Ad spending typically lags behind search queries. If consumer interest continues, Bing could enjoy a boost in ad dollars for the third quarter.

Overall, the market for search engine ad spending flourished in the second quarter. The report noted that retail firms spent 36 percent more on paid search engines than in 2008's second quarter. Spending just for the month of June shot up 55 percent from June 2008.

(Credit: SearchIgnite)

"We've seen very strong paid search spend from retailers for the last several quarters," said Barnette, "a trend that can be attributed to an increase in retailers' promotional activity as they turn to heavy discounting and sales to drive purchases."

For this latest report, SearchIgnite tracked 500 marketers using Google, Yahoo, and MSN/Bing for the quarter ended June 30.

Apple, Google, and Yahoo are among the tech giants being investigated by the Justice Department for possible antitrust violations related to negotiations over the recruiting and hiring of one another's employees, according to a Washington Post report.

The review is said to be "industrywide" and in preliminary stages, according to the report, which cited two unnamed sources. Companies that agree not to hire away talent could be stifling competition, the report noted.

Representatives for Apple, Google, Yahoo, and the Department of Justice did not immediately respond to requests for comment.

Tech companies, known for their exhaustive recruiting efforts, have waged fierce battles to maintain top talent. In one closely watched case, Google was sued by Microsoft in 2005 over Google's decision to hire Kai-Fu Lee away from Microsoft to run Google's research operation in China. The two parties eventually settled out of court.

IBM has proved particularly territorial about departing executives. Last month, the company filed a lawsuit in federal court to prevent its former head of mergers and acquisitions, David Johnson, from joining Dell, saying it would be a violation of his contract.

Last year, IBM sued Mark Papermaster to keep him from joining Apple. The lawsuit's claims were nearly identical, with IBM charging that Papermaster's joining Apple would cause him to divulge trade secrets and was a violation of the noncompete clause to which he agreed. IBM and Papermaster settled after three months, and Papermaster finally started working at Apple three months later.