Webware

For those who are having a little trouble understanding exactly what Google Wave is all about, seeing it in action might help you wrap your head around the concept.

Google has released video of Thursday's keynote speech at Google I/O in San Francisco, where the company publicly demonstrated Google Wave for the first time before about 4,000 developers. Google Wave is an ambitious, if incomplete, attempt to reinvent e-mail and Internet communication in general.

Developers are just starting to get their hands on Google Wave to try it out for themselves, but the public is not expected to get the same chance for several months. We hope to post a hands-on review ourselves in the coming days, but for now, check out the video if you'd like to see Google Wave in action. Be forewarned, it's long (90 minutes).

SAN FRANCISCO--Google wants its Native Client technology to be a little more native.

Google Native Client, still highly experimental, lets browsers run program modules natively on an x86 processor for higher performance than with Web programming technologies such as JavaScript or Flash that involve more software layers to process and execute the code. But to use it, there's a significant barrier: people must install a browser plug-in.

However, Google wants to make the technology more broadly accessible in browsers through new technology coming to HTML, the standard used to build Web pages, and at the Google I/O developer conference Thursday demonstrated its work to make that happen.

Specifically, David Sehr, a tech lead for Native Client, showed off Web Workers standard to let Web pages assign different tasks to independent processing "threads," effectively letting a browser walk and chew gum at the same time rather than waiting for one chore to be finished before the next begins. Web workers are one element of the ambitious but still not finalized HTML 5 standard.

Why care? Because today your browser runs software excruciatingly slowly compared to native applications that run on your computer, but Google wants to speed them up tremendously, a move that would add a lot of muscle to its ambition to make Web-based software more competitive.

"We want to be within single-digit percentages of what you can do with the best desktop native code," said Brad Chen, engineering manager of the Google Native Client (NaCl) project in a talk at Google I/O.

Examples of what can be done include decoding video, encrypting data, video game physics engines, and face recognition. More interesting, perhaps, is when Native Client can work in conjunction with another Google browser plug-in, O3D, that lets browsers take advantage of hardware to accelerate 3D graphics.

"With O3D, we think we'll be able to enable high-quality games, the kind you're accustomed to seeing on consoles, as well as CAD applications," Chen said.

Although Google is working hard to enable more powerful Web applications, it's not all altruistic. The company has a growing stable of applications including Google Docs, Google Maps, and Gmail that can become much more competitive with desktop technology such as Microsoft Office. For now, though, Google is trying to hammer out Native Client security issues before promoting it more widely among programmers, much less mainstream users.

Sehr said Google's browser, Chrome, will introduce Web Worker support, he hoped within the next couple weeks. Google has been touting HTML 5 features at Google I/O, and Chrome gives Google a way to advance the state of Web application art.

Though other browsers are building in Web worker support, too, for now the technology is rough and certainly not a foundation a Web programmer could expect widespread support for among browsers.

Google plans to support Native Client both through Web Workers and the plug-in, Chen said in an interview. Built-in support in the browser is helpful, but Chen said Web Workers have undesirable limitations for many chores. For example, the plug-in is necessary for applications that require a fast response to user input, he said.

One such example Google showed at the conference was a photo editor. With it, images could be rotated, zoomed, and have colors and tones adjusted with a variety of sliders. There are online photo editors available today, but they typically use Adobe Systems' Flash plug-in.

SAN FRANCISCO--When developers are comparing your new product to the unveiling of the iPhone, you know you've probably got something on your hands.

Such was the reaction at Google I/O in the hours following Google's first demonstration of Google Wave, a bid to redefine the way people communicate on the Internet by blending e-mail, instant messaging, file sharing, and collaboration software into one service. Following a session in which developers were given a peek under the hood at the technology and what it might let them do, several were quite impressed and already pondering what Google Wave would allow them to create.

"I haven't been this jazzed since the release of the iPhone," said Michael Rexroad, a software engineer with Cisco's telepresence systems business unit. He was referring to the way ideas immediately sprung to mind Thursday morning regarding how to use the technology demonstrated Thursday to create new types of applications, much the same way Apple's first public demonstration of the iPhone in January 2007 inspired a generation of software developers.

Developer support is crucial to the success of Google Wave. The company is releasing Wave as a developer preview to attendees on Thursday, and it is still filled with lots of rough edges, bugs, and incomplete details.

But the genius behind Google Wave is not in the individual parts, rather in the way Google has assembled a set of existing technologies into an attractive platform for developers, said Andreas Schobel, chief technology officer for mobile start-up 3Banana.

Schobel compared Wave to how Google Maps (perhaps not coincidentally developed by the same people behind Google Wave) awoke developers to the possibilities presented by Ajax technologies, which had been around for some time but had yet to gain traction as some of the core technologies used to build the modern Web.

Daniel Jefferies, president of Newmind Group, a Google Apps reseller, is not a developer, but was intrigued by the possibility of using Wave as an internal tool for improving the productivity of his company. Newmind provides consulting services for helping small and medium businesses implement Google Apps inside their groups, and thought he could better manage his team, their tasks, and their relationships with clients with this sort of tool.

Perhaps the most ringing endorsement came from a software engineer employed by one of Google's rivals, who declined to be identified for obvious reasons. "This will revolutionize e-mail," he said.

While that may be a stretch at this juncture, developers filling the halls of the Moscone Center are definitely buzzing about Google Wave.

The search giant rallies attendees at its San Francisco developer conference around the next generation of Web technologies, in which the browser will become the desktop and everyday people can do more with their Web sites.

Gmail in real-time: Google does the Wave

Google is set to unveil an ambitious project to create what it calls 'the e-mail of the future,' and the reactions of developers at Google I/O will be telling.
(Posted in Webware by Tom Krazit)
May 28, 2009 9:00 AM PDT
• Photos: Google shows off Wave's potential
• Google releases Wave demonstration video

Google Wave has developers buzzing

The search giant's ambitious Google Wave project has developers at Google I/O mulling the possibilities, and even comparing it to the iPhone.
(Posted in Webware by Tom Krazit)
May 28, 2009 2:27 PM PDT

Google shows Native Client built into HTML 5

Google showed a version of its technology for putting Web applications on steroids that's built into its Chrome browser. Also: expect Web Worker support in Chrome soon.
(Posted in Webware by Stephen Shankland)
May 28, 2009 3:01 PM PDT

Android 1.5 apps from Google I/O

We bring you a close look at six applications for Google's soon-to-be-released 1.5 Android operating system, some of which have have never been released on this platform.
(Posted in Download.com by Jessica Dolcourt )
May 28, 2009 3:28 PM PDT

Google--market disruptor or destroyer

The search and advertising giant aims to take the "gate" position in many markets, sitting between users and their content or services.
(Posted in Webware by Rafe Needleman)
May 28, 2009 9:00 AM PDT

Chrome gets HTML video support

Google's browser joins Firefox, Safari, and Opera with the ability to display video without a plug-in such as Adobe's Flash. But the HTML standard is rough at best.
(Posted in Webware by Stephen Shankland)
May 28, 2009 8:01 AM PDT

Google's 'Idol'-like Android challenge

Nine developers will go home with $100,000 in a contest that seems to borrow elements from "American Idol." A lucky 10th will pocket $250,000.
(Posted in Crave by Jessica Dolcourt)
May 27, 2009 7:47 PM PDT

Android developers get their Oprah moment

Google gives away 4,000 Android handsets Wednesday, delighting developers at Google I/O who thought they were merely getting an Android 2.0 preview.
(Posted in Webware by Tom Krazit)
May 27, 2009 12:26 PM PDT

Google: The browser is the computer

HTML 5 technologies will allow Web developers to build applications almost as cool as anything found on the desktop, according to Google executives and engineers.
• Photos: Inside Google I/O
• Video: Google CEO touts always-on computing
(Posted in Webware by Tom Krazit)
May 27, 2009 11:04 AM PDT

Google creates Web Elements for easy news feeds

Web publishers without development skills will be able to add news feeds, maps, and other types of Google products to their Web pages with a simple cut-and-paste operation.
(Posted in Webware by Tom Krazit)
May 27, 2009 9:00 AM PDT

Updated 12:28 p.m. PDT with additional comments from Google.

Google is ready to start talking about its answer to demand for real-time--yet organized--Internet communication.

Google on Thursday publicly demonstrated Google Wave for the first time at the Google I/O conference in San Francisco. Billed as "the e-mail of the future," Google Wave is the result of a multiyear project inside of Google to reinvent the inbox, blending e-mail, instant messaging, photo sharing, and perhaps, with input from developers, connections to the world of social networking.

Google Wave is an attempt to "combine conversation-type communication and collaboration-type communication," said Lars Rasmussen, who launched the project with his brother Jens after Google acquired their mapping start-up in 2004. The brothers Rasmussen said they were inspired by the fact that two of the most commonly used Internet communication technologies--e-mail and instant messaging--are based on relatively ancient offline communication techniques, namely the letter and the telephone.

The Rasmussens were given the authority to create "one of the most autonomous independent groups we've had at Google," said co-founder Sergey Brin in a press conference following the demonstration. Given the success the brothers had in developing the technologies behind Google Maps, Brin was inclined to "give them the benefit of the doubt" when Lars came to him pitching a bid to reinvent Internet communication.

They came up with Google Wave, which organizes Internet discussions in the trendy stream of consciousness fashion. It's a little bit Twitter, a little bit Friendfeed, and a little bit Facebook all in one service, allowing you to send direct messages to online contacts with real-time replies, share photos or documents, and add or delete members of the conversation as needed.

In that sense, it's not a completely public discussion, nor a completely private one. A user creates a "wave" by typing a message or uploading photos and adding contacts to the wave as they see fit. Other contacts can be added later, and those people can add other contacts to the wave unless the original wave starter forbids new entrants.

"Each person that we show it too, something different resonates as useful" to their way of communicating on the Internet, said Stephanie Hannon, project manager for Google Wave.

At the moment, the functionality is somewhat limited, but Google is introducing Google Wave at its developer conference for a reason: "a lot of this depends on developer uptake," Rasmussen said. The company will release APIs (application programming interfaces) at the conference so that developers can start testing how to build Wave into their own sites, or how to integrate their services with Google's.

Google envisions three types of developer projects using Wave. The first is the most obvious; using Wave as a gateway for conversations that you're already having elsewhere on Twitter, Friendfeed, Facebook, blogs, and other social media sites.

There are plenty of reasons for Google to try to tap into the "stickiness" of various social networks, where users spend obscene amounts of time. And the company thinks that services such as Twitter recognize the value of letting others build a front end into their services: there are dozens of Twitter apps for PCs and smartphones that grant such access without having to use Twitter's own front end, and those apps don't seem to have put much of a dent in Twitter's overall traffic. For starters, Google Wave will allow users to post new items to blogs created with Blogger from within a wave, and see comments and replies within a wave.

The second category involves creating applications that run within a wave, similar to how developers have used Facebook as a platform to create all sorts of applications. Collaborative games are expected to be among the first applications to appear within Google Wave.

Lastly, Google wants developers to think of Wave as a possible enhancement to an existing workflow within an enterprise. The example Rasmussen used was a bug tracker used by software developers to identify and assign bugs. Bugs could be organized in waves; participants post the new bug to a global wave, then the team leader can assign bugs to individual team members within the wave, and developers can comment on their fix for a particular bug as they are tackled and cleared, all within the same thread.

The software has a long way to go: Google is releasing it as a developer preview on Thursday, and is actively looking for feedback on how it can improve. Sometime later this year Google expects to release it to the general public, but Rasmussen would not commit to a more specific timeframe.

Google also plans to open-source the format at the heart of Google Wave as a protocol in order to let developers build their own waves. The company has not determined the license that will be used to open-source the code, Rasmussen said.

Developer feedback will be crucial toward gauging the impact of Google Wave in a marketplace crowded with similar ideas. For months, Google has been pressed with inquiries about whether or not it plans to buy companies like Twitter or others that specialize in real-time Internet communication, and thus far, the company has demurred.

Now we know why.

Updated 12:15 p.m. PDT with additional comments from Google and Mozilla.

Corrected at 12:57 p.m. PDT: This post initially mischaracterized Firefox, Safari, Chrome, and Opera. They are all open standards browsers.

SAN FRANCISCO--Google spent Wednesday morning trying to get developers excited about the next generation of Web technologies by showing off how future Web applications will mimic desktop apps.

"It's time for us to take advantage of the amazing opportunity that is before us," said Google CEO Eric Schmidt, kicking off Google I/O 2009 in San Francisco. Schmidt was referring to the growing sense that the Internet and browsers--rather than a computer's operating system--will be the future foundation for application development.

The industry isn't quite ready for that yet. Many of applications demonstrated before the crowd of around 4,000 developers will require the widespread adoption of HTML 5 technologies, which are still under development by a consortium of companies and organizations.

Still, Google's Vic Gundotra, vice president of engineering, noted that the four modern open standards browsers (Firefox, Safari, Chrome,and Opera) are all adopting some HTML 5 technologies as they become more stable, taking every opportunity possible to ding Microsoft's Internet Explorer for lagging behind the other four browsers.

However, offstage after his keynote, Gundotra downplayed any conflict with Microsoft, noting that its task in moving toward HTML 5 is more complex because of the large number of enterprises running Internet Explorer, and the possibility that internal applications developed for that browser could break. "As they follow through, they are going to have a huge impact on moving the Web forward," he said.

Gundotra showed off how Web applications will be able to take advantage of five main HTML 5 concepts: canvas tags, video tags, geolocation, application caching and database, and Web Workers.

For example, canvas tags help developers bring all kinds of sophisticated graphics to their Web applications without having to use a plug-in--which is also the appeal of the video tag. Google showed off an "experiment" with YouTube videos coded using the video tags, which gives developers quite a few more options when it comes to how those videos can be embedded into a Web page.

Geolocation is another huge topic of late with mobile applications. Google showed off how its Google Latitude application takes advantage of a new iPhone geolocation API that Apple will release as part of the iPhone 3.0 software to run in the mobile Safari browser. Mozilla's Jay Sullivan also showed off how Firefox 3.5 will come with a button that allows the browser to pinpoint your location in Google Maps using Wi-Fi and cell tower positioning data.

In a briefing session following the keynote, Google's Matthew Papakipos, engineering director, said Google has various aspects of the HTML 5 capabilities demonstrated Wednesday in different parts of the development process. For example, the canvas tags that allow for richer graphics are present in Chrome 2.0, which is currently available, but others, such as the video tags, are farther away from reaching the world.

Mozilla plans to support all the HTML 5 technologies demonstrated Wednesday with the release of Firefox 3.5, Sullivan said. That is expected to arrive fairly soon.

One interesting question, perhaps saved for another day, concerned how, or if, Google plans to index the coming wave of Web applications as part of its search efforts. Attempting to index these rich applications at the moment presents quite a "challenge" for search bots trying to figure out how to categorize the content within an application.

"Content won't be a problem, but how do you index Gmail? Should you index Gmail?" Gundotra pondered.

Google has made it easier for novice Web publishers to spruce up their sites with feeds of Google's products.

Google Web Elements, set to be unveiled Wednesday at the Google I/O developer conference in San Francisco, is an easy cut-and-paste way to add a Google News feed, for example, to a Web page. The company plans to demonstrate the service later on Wednesday at the conference.

Web publishers have been able to add such feeds to their sites in the past using Google's APIs, said DeWitt Clinton, technical manager of Google's developer relations team. But using Web Elements is a much easier process; if a Web site publisher wants a customized Google News feed, say, they just select the type of feed, type in custom categories, and cut and paste the resulting code into their Web page code.

"We're trying to nail the simplicity story," Clinton said. It's not clear yet whether or not this is something that professional publishers would want to add to their site, or just a tool for individual bloggers or small businesses; Google's main focus in the early going was to make the process as easy to understand as possible, he said.

Google Web Elements was expected to go live here at 9 a.m. PDT Wednesday. Eight feeds are available at first, with more possibly to come over the next several months.

SAN FRANCISCO--The inner workings of Google just became a little less secret.

The search colossus has shed only occasional light on its data center operations, but on Wednesday, Google fellow Jeff Dean turned a spotlight on some parts of the operation. Speaking to an overflowing crowd at the Google I/O conference here on Wednesday, Dean managed simultaneously to demystify Google a little while also showing just how exotic the company's infrastructure really is.

On the one hand, Google uses more-or-less ordinary servers. Processors, hard drives, memory--you know the drill.

On the other hand, Dean seemingly thinks clusters of 1,800 servers are pretty routine, if not exactly ho-hum. And the software company runs on top of that hardware, enabling a sub-half-second response to an ordinary Google search query that involves 700 to 1,000 servers, is another matter altogether.

Google doesn't reveal exactly how many servers it has, but I'd estimate it's easily in the hundreds of thousands. It puts 40 servers in each rack, Dean said, and by one reckoning, Google has 36 data centers across the globe. With 150 racks per data center, that would mean Google has more than 200,000 servers, and I'd guess it's far beyond that and growing every day.

Regardless of the true numbers, it's fascinating what Google has accomplished, in part by largely ignoring much of the conventional computing industry. Where even massive data centers such as the New York Stock Exchange or airline reservation systems use a lot of mainstream servers and software, Google largely builds its own technology.

I'm sure a number of server companies are sour about it, but Google clearly believes its technological destiny is best left in its own hands. Co-founder Larry Page encourages a "healthy disrespect for the impossible" at Google, according to Marissa Mayer, vice president of search products and user experience, in a speech Thursday.

To operate on Google's scale requires the company to treat each machine as expendable. Server makers pride themselves on their high-end machines' ability to withstand failures, but Google prefers to invest its money in fault-tolerant software.

"Our view is it's better to have twice as much hardware that's not as reliable than half as much that's more reliable," Dean said. "You have to provide reliability on a software level. If you're running 10,000 machines, something is going to die every day."

Breaking in is hard to do
Bringing a new cluster online shows just how fallible hardware is, Dean said.

In each cluster's first year, it's typical that 1,000 individual machine failures will occur; thousands of hard drive failures will occur; one power distribution unit will fail, bringing down 500 to 1,000 machines for about 6 hours; 20 racks will fail, each time causing 40 to 80 machines to vanish from the network; 5 racks will "go wonky," with half their network packets missing in action; and the cluster will have to be rewired once, affecting 5 percent of the machines at any given moment over a 2-day span, Dean said. And there's about a 50 percent chance that the cluster will overheat, taking down most of the servers in less than 5 minutes and taking 1 to 2 days to recover.

While Google uses ordinary hardware components for its servers, it doesn't use conventional packaging. . And, Dean said, the company currently puts a case around each 40-server rack, an in-house design, rather than using the conventional case around each server.

The company has a small number of server configurations, some with a lot of hard drives and some with few, Dean said. And there are some differences at the larger scale, too: "We have heterogeneity across different data centers but not within data centers," he said.

As to the servers themselves, Google likes multicore chips, those with many processing engines on each slice of silicon. Many software companies, accustomed to better performance from ever-faster chip clock speeds, are struggling to adapt to the multicore approach, but it suits Google just fine. The company already had to adapt its technology to an architecture that spanned thousands of computers, so they already have made the jump to parallelism.

"We really, really like multicore machines," Dean said. "To us, multicore machines look like lots of little machines with really good interconnects. They're relatively easy for us to use."

Although Google requires a fast response for search and other services, its parallelism can produce that even if a single sequence of instructions, called a thread, is relatively slow. That's music to the ears of processor designers focusing on multicore and multithreaded models.

"Single-thread performance doesn't matter to us really at all," Dean said. "We have lots of parallelizable problems."

The secret sauce
So how does Google get around all these earthly hardware concerns? With software--and this is where you might think about dusting off your computer science degree.

Dean described three core elements of Google's software: GFS, the Google File System, BigTable, and the MapReduce algorithm. And although Google helps with a lot of open-source software projects that helped the company get its start, these packages remain proprietary except in general terms.

GFS, at the lowest level of the three, stores data across many servers and runs on almost all machines, Dean said. Some incarnations of GFS are file systems "many petabytes in size"--a petabyte being a million gigabytes. There are more than 200 clusters running GFS, and many of these clusters consist of thousands of machines.

GFS stores each chunk of data, typically 64MB in size, on at least three machines called chunkservers; master servers are responsible for backing up data to a new area if a chunkserver failure occurs. "Machine failures are handled entirely by the GFS system, at least at the storage level," Dean said.

To provide some structure to all that data, Google uses BigTable. Commercial databases from companies such as Oracle and IBM don't cut the mustard here. For one thing, they don't operate the scale Google demands, and if they did, they'd be too expensive, Dean said.

BigTable, which Google began designing in 2004, is used in more than 70 Google projects, including Google Maps, Google Earth, Blogger, Google Print, Orkut, and the core search index. The largest BigTable instance manages about 6 petabytes of data spread across thousands of machines, Dean said.

MapReduce, the first version of which Google wrote in 2003, gives the company a way to actually make something useful of its data. For example, MapReduce can find how many times a particular word appears in Google's search index; a list of the Web pages on which a word appears; and the list of all Web sites that link to a particular Web site.

With MapReduce, Google can build an index that shows which Web pages all have the terms "new," "york," and "restaurants"--relatively quickly. "You need to be able to run across thousands of machines in order for it to complete in a reasonable amount of time," Dean said.

The MapReduce software is increasing use within Google. It ran 29,000 jobs in August 2004 and 2.2 million in September 2007. Over that period, the average time to complete a job has dropped from 634 seconds to 395 seconds, while the output of MapReduce tasks has risen from 193 terabytes to 14,018 terabytes, Dean said.

On any given day, Google runs about 100,000 MapReduce jobs; each occupies about 400 servers and takes about 5 to 10 minutes to finish, Dean said.

That's a basis for some interesting math. Assuming the servers do nothing but MapReduce, that each server works on only one job at a time, and that they work around the clock, that means MapReduce occupies about 139,000 servers if the jobs take 5 minutes each. For 7.5-minute jobs, the number increases to 208,000 servers; if the jobs take 10 minutes, it's 278,000 servers.

My calculations could be off base, but even qualitatively, that's enough computing horsepower to make the mind boggle.

Fault-tolerant software
MapReduce, like GFS, is explicitly designed to sidestep server problems.

"When a machine fails, the master knows what task that machine was assigned and will direct the other machines to take up the map task," Dean said. "You can end up losing 100 map tasks, but can have 100 machines pick up those tasks."

The MapReduce reliability was severely tested once during a maintenance operation on one cluster with 1,800 servers. Workers unplugged groups of 80 machines at a time, during which the other 1,720 machines would pick up the slack. "It ran a little slowly, but it all completed," Dean said.

And in a 2004 presentation, Dean said, one system withstood a failure of 1,600 servers in a 1,800-unit cluster.

Next-generation data center to-do list
So all is going swimmingly at Google, right? Perhaps, but the company isn't satisfied and has a long to-do list.

Most companies are trying to figure out how to move jobs gracefully from one server to another, but Google is a few orders of magnitude above that challenge. It wants to be able to move jobs from one data center to another--automatically, at that.

"We want our next-generation infrastructure to be a system that runs across a large fraction of our machines rather than separate instances," Dean said.

Right now some massive file systems have different names--GFS/Oregon and GFS/Atlanta, for example--but they're meant to be copies of each other. "We want a single namespace," he said.

These are tough challenges indeed considering Google's scale. No doubt many smaller companies look enviously upon them.

SAN FRANCISCO--When it comes to search quality, Google has a split personality.

Google uses a method called split A/B testing to measure exactly what changes it should make to its main search Web site--both to its famously Spartan search box and to the results it produces. With the approach, Google shows different versions of the pages to users and measures how they respond, said Marissa Mayer, vice president of search products and user experience, in a speech at the Google I/O conference here Thursday.

For example, Mayer said, the company wanted to find out how many search results to show users--the customary 10, or 20, 25, or 30? When asked directly, users said they'd like more results on a page, but testing showed otherwise.

Specifically, Google found that when the results increased to 30 per page, people searched 20 percent less overall, Mayer said. After much analysis of server logs, the company found it was because it took about twice as long to display the longer results list for the user, and speed matters.

"As Google gets faster, people search more, and as it gets slower, people search less," she said.

The same effect happened with Google Maps. When the company trimmed the 120KB page size down by about 30 percent, the company started getting about 30 percent more map requests. "It was almost proportional. If you make a product faster, you get that back in terms of increased usage," she said.

Split A/B testing also led Google to refine exactly how much white space to pad around its logo and other elements on the search results page. And it changed from the industry practice of a pale blue background behind ads to a pale yellow background. People not only clicked on ads more, they also searched more in general, she said.

The subject clearly is close to Mayer's heart. She's an engineer who also has an interest in the more aesthetic realm of design.

"On the Web in general, (creating sites) is much more a design than an art," she said. "You can find small differences and mathematically learn which is right."

A history of Google's search page
Google's search page, with its abundance of empty white space and its almost boastful "I'm feeling lucky" button, looks downright ordinary today. But it wasn't always the case.

... Read more