News Blog

Here's a roundup of recent juicy Google tidbits:

• Amid general praise for Steve McQueen's famed car chase in the 1968 movie Bullitt, there are jeers about the recurring green VW Beetle and the geographic hash it makes of San Francisco. You might be amused to see this side-by-side view of the Bullitt chase and a Google map that shows just how much they jump from one patch of the city to another. (Via Google Maps Mania.)

• Ever wonder what the limits on Gmail activity are? Well, here's the answer, according to a Google Apps posting: "500 messages per day (i.e., you can hit 'Send' a maximum of 500 times); 500 unique recipients; 2,000 total e-mails (for example, you could send one message to a group of 500 people four times)."

• Google is advertising in China, just like they did in Russia. Google rarely takes out ads, but apparently in countries like China and Russia, where Baidu and Yandex, respectively, are more widely used than Google, the company is willing to think differently.

• Google is trying to build up discussions at a newly launched Google Mobile Community. "We envision this community as being a place where you can discuss the world of mobile in general...We also want the community to be a place where you can tell us what you think about our very own products," said Bret Luboyeski, a Google mobile product specialist, on the Google Mobile blog.

• Google said it extended its YouTube Partner Program to Germany and France. That means popular members in those countries can make ad money from their videos through the revenue-sharing program.

• Miss the Google I/O conference? All the Google I/O videos are online now on YouTube.

Google on Monday unveiled a new Web-based tool, Map Maker, that lets people add roads, lakes, businesses, and other features to unmapped regions of Google Maps.

With the tool, people can using tracing tools to build maps in Cyprus, Iceland, Pakistan, and Vietnam, according to the Google LatLong blog. Also open for cartographic contributions are several Caribbean nations: Antigua and Barbuda, Bahamas, Barbados, Bermuda, British Virgin Islands, Cayman Islands, Grenada, Jamaica, Netherlands Antilles, St. Kitts and Nevis, St. Lucia, St. Vincent and the Grenadines, and Trinidad and Tobago.

I'm of two minds about this. On the one hand, it's great that this kind of activity can be crowd-sourced (please excuse the jargon) so the community (please excuse the jargon again) can contribute to a project that reduces the amount of digitally uncharted terrain. Google has given us a way to help make a difference that, while small, could collectively become quite large.

But on the other hand, I can think of worthy causes in greater need of charity or free labor than Google. If we're all going to be augmenting Google Maps with user-generated content, wouldn't it be nice if we could do it through a more neutral mechanism that lets others benefit from the work, too? Geotagged entries in Wikipedia show on Google Maps, but not Google Maps alone, at least theoretically.

Overall, I think my first reaction will carry the day for me.

That's because, fundamentally, Google Maps is a service not just consumed by many but also repackaged by many through the availability of the Google Maps API (application programming interface). So until the day Google flips its Don't Be Evil switch to the "off" position, Google Maps is in effect a public utility, and many can benefit from contributions to the service.

Google Map Maker looks slick, but it would be slicker with better satellite imagery. Parts of Iceland, one of my favorite places on Earth, are too coarse for any tracing.

Yahoo has licensed detailed data that describes 40,000 neighborhood boundaries in more than 2,000 U.S. cities, from a company called Urban Mapping.

"Allowing users to search by neighborhood yields more appropriate results, adding value and relevancy to the overall experience," Bob Upham, director of business development for Yahoo's Geo Technologies group, said in a statement.

Urban Mapping's Urbanware database works with a thorny geographic issue, trying to describe informal and often ill-defined neighborhood boundaries.

Meanwhile, Yahoo hopes people will help define neighborhoods through its Flickr photo-sharing service, too.

Who would have believed Google's geographic Web services could actually get your adrenaline going?

Granted, these aren't real video games, but two Web sites are pushing what can be done with interactive interfaces to Google Maps and Google Earth.

The first, taking advantage of Google Maps' new ability to work with Flash applications, lets you drive a car, bus, or truck around Google Maps. It won't bat an eye if you drive through a building or into the ocean, but Katsuomi Kobayashi, the programmer from Osaka, Japan, who wrote it, was happy to note that the software can display images at 40 frames per second vs. 20 at best for JavaScript. And it uses less CPU power, too.

Another novelty is a flight simulator for the browser plug-in version of Google Earth announced at Google I/O a week and a half ago. (This is different from the flight sim that works with the Google Earth standalone software.) It works with recent versions of Firefox, Internet Explorer, and Flock, but on Windows only.

Again, the software is crude by gaming standards, but it does illustrate what can be done these days inside a browser. I'm among those who are interested to watch Google Earth abilities gradually pop up in Google Maps and in the browser. It's easily conceivable to me that we'll soon be seeing all manner of games that run on the 3D models of the real world that Google and Microsoft are building. Lower network latencies, faster server responses, and higher network data capacity all point in that direction.

(Via Google Geo Developers Blog and Google Maps Mania.)

A small town in Minnesota has told Google that its Street View feature can hit the road.

North Oaks, a private community of 4,500 residents north of St. Paul, isn't too keen on outsiders traipsing through its privately owned streets--even if is only on the Internet. According to the city's Web site, the roads are privately owned, and a no-trespassing sign greets potential visitors to the city.

So city officials were really unhappy when images of their streets and homes appeared on the Google Maps Street View feature, which presents a view of dozens of United States cities from a driver's perspective.

The North Oaks City Council sent the Internet search giant a letter in January demanding that images be removed or risk being cited for trespassing, the Minneapolis Star Tribune reported.

"It's not the hoity-toity folks trying to figure out how to keep the world away," Mayor Thomas Watson told the newspaper. "They really didn't have any authorization to go on private property."

The company removed the images shortly thereafter, a Google representative told the newspaper.

"This is very rare, where an entire town would request to be taken off," Google spokeswoman Elaine Filadelfo told the paper, adding that the company removes images when individuals make the request.

Google is no stranger to complaints about its Street View service. Not long after the feature launched in May 2007, privacy advocates criticized Google for displaying photographs that included people's faces and car license plates. In May, the company announced that it had begun testing face-blurring technology for the service.

In April, a Pittsburgh couple sued Google over photographs of their home that appeared on the company's site, saying Google should honor a private road sign on their street. It claims that Google's "reckless conduct" has "exposed plaintiff's private information to the public."

For those who weren't exactly comfortable with ordinary photos of their property appearing on the Net, get ready to reveal a little more. A couple of weeks ago, Google confirmed that it is gathering 3D data, along with the photographs it takes for its online Street View service.

SAN FRANCISCO--The inner workings of Google just became a little less secret.

The search colossus has shed only occasional light on its data center operations, but on Wednesday, Google fellow Jeff Dean turned a spotlight on some parts of the operation. Speaking to an overflowing crowd at the Google I/O conference here on Wednesday, Dean managed simultaneously to demystify Google a little while also showing just how exotic the company's infrastructure really is.

On the one hand, Google uses more-or-less ordinary servers. Processors, hard drives, memory--you know the drill.

On the other hand, Dean seemingly thinks clusters of 1,800 servers are pretty routine, if not exactly ho-hum. And the software company runs on top of that hardware, enabling a sub-half-second response to an ordinary Google search query that involves 700 to 1,000 servers, is another matter altogether.

Google doesn't reveal exactly how many servers it has, but I'd estimate it's easily in the hundreds of thousands. It puts 40 servers in each rack, Dean said, and by one reckoning, Google has 36 data centers across the globe. With 150 racks per data center, that would mean Google has more than 200,000 servers, and I'd guess it's far beyond that and growing every day.

Regardless of the true numbers, it's fascinating what Google has accomplished, in part by largely ignoring much of the conventional computing industry. Where even massive data centers such as the New York Stock Exchange or airline reservation systems use a lot of mainstream servers and software, Google largely builds its own technology.

I'm sure a number of server companies are sour about it, but Google clearly believes its technological destiny is best left in its own hands. Co-founder Larry Page encourages a "healthy disrespect for the impossible" at Google, according to Marissa Mayer, vice president of search products and user experience, in a speech Thursday.

To operate on Google's scale requires the company to treat each machine as expendable. Server makers pride themselves on their high-end machines' ability to withstand failures, but Google prefers to invest its money in fault-tolerant software.

"Our view is it's better to have twice as much hardware that's not as reliable than half as much that's more reliable," Dean said. "You have to provide reliability on a software level. If you're running 10,000 machines, something is going to die every day."

Breaking in is hard to do
Bringing a new cluster online shows just how fallible hardware is, Dean said.

In each cluster's first year, it's typical that 1,000 individual machine failures will occur; thousands of hard drive failures will occur; one power distribution unit will fail, bringing down 500 to 1,000 machines for about 6 hours; 20 racks will fail, each time causing 40 to 80 machines to vanish from the network; 5 racks will "go wonky," with half their network packets missing in action; and the cluster will have to be rewired once, affecting 5 percent of the machines at any given moment over a 2-day span, Dean said. And there's about a 50 percent chance that the cluster will overheat, taking down most of the servers in less than 5 minutes and taking 1 to 2 days to recover.

While Google uses ordinary hardware components for its servers, it doesn't use conventional packaging. . And, Dean said, the company currently puts a case around each 40-server rack, an in-house design, rather than using the conventional case around each server.

The company has a small number of server configurations, some with a lot of hard drives and some with few, Dean said. And there are some differences at the larger scale, too: "We have heterogeneity across different data centers but not within data centers," he said.

As to the servers themselves, Google likes multicore chips, those with many processing engines on each slice of silicon. Many software companies, accustomed to better performance from ever-faster chip clock speeds, are struggling to adapt to the multicore approach, but it suits Google just fine. The company already had to adapt its technology to an architecture that spanned thousands of computers, so they already have made the jump to parallelism.

"We really, really like multicore machines," Dean said. "To us, multicore machines look like lots of little machines with really good interconnects. They're relatively easy for us to use."

Although Google requires a fast response for search and other services, its parallelism can produce that even if a single sequence of instructions, called a thread, is relatively slow. That's music to the ears of processor designers focusing on multicore and multithreaded models.

"Single-thread performance doesn't matter to us really at all," Dean said. "We have lots of parallelizable problems."

The secret sauce
So how does Google get around all these earthly hardware concerns? With software--and this is where you might think about dusting off your computer science degree.

Dean described three core elements of Google's software: GFS, the Google File System, BigTable, and the MapReduce algorithm. And although Google helps with a lot of open-source software projects that helped the company get its start, these packages remain proprietary except in general terms.

GFS, at the lowest level of the three, stores data across many servers and runs on almost all machines, Dean said. Some incarnations of GFS are file systems "many petabytes in size"--a petabyte being a million gigabytes. There are more than 200 clusters running GFS, and many of these clusters consist of thousands of machines.

GFS stores each chunk of data, typically 64MB in size, on at least three machines called chunkservers; master servers are responsible for backing up data to a new area if a chunkserver failure occurs. "Machine failures are handled entirely by the GFS system, at least at the storage level," Dean said.

To provide some structure to all that data, Google uses BigTable. Commercial databases from companies such as Oracle and IBM don't cut the mustard here. For one thing, they don't operate the scale Google demands, and if they did, they'd be too expensive, Dean said.

BigTable, which Google began designing in 2004, is used in more than 70 Google projects, including Google Maps, Google Earth, Blogger, Google Print, Orkut, and the core search index. The largest BigTable instance manages about 6 petabytes of data spread across thousands of machines, Dean said.

MapReduce, the first version of which Google wrote in 2003, gives the company a way to actually make something useful of its data. For example, MapReduce can find how many times a particular word appears in Google's search index; a list of the Web pages on which a word appears; and the list of all Web sites that link to a particular Web site.

With MapReduce, Google can build an index that shows which Web pages all have the terms "new," "york," and "restaurants"--relatively quickly. "You need to be able to run across thousands of machines in order for it to complete in a reasonable amount of time," Dean said.

The MapReduce software is increasing use within Google. It ran 29,000 jobs in August 2004 and 2.2 million in September 2007. Over that period, the average time to complete a job has dropped from 634 seconds to 395 seconds, while the output of MapReduce tasks has risen from 193 terabytes to 14,018 terabytes, Dean said.

On any given day, Google runs about 100,000 MapReduce jobs; each occupies about 400 servers and takes about 5 to 10 minutes to finish, Dean said.

That's a basis for some interesting math. Assuming the servers do nothing but MapReduce, that each server works on only one job at a time, and that they work around the clock, that means MapReduce occupies about 139,000 servers if the jobs take 5 minutes each. For 7.5-minute jobs, the number increases to 208,000 servers; if the jobs take 10 minutes, it's 278,000 servers.

My calculations could be off base, but even qualitatively, that's enough computing horsepower to make the mind boggle.

Fault-tolerant software
MapReduce, like GFS, is explicitly designed to sidestep server problems.

"When a machine fails, the master knows what task that machine was assigned and will direct the other machines to take up the map task," Dean said. "You can end up losing 100 map tasks, but can have 100 machines pick up those tasks."

The MapReduce reliability was severely tested once during a maintenance operation on one cluster with 1,800 servers. Workers unplugged groups of 80 machines at a time, during which the other 1,720 machines would pick up the slack. "It ran a little slowly, but it all completed," Dean said.

And in a 2004 presentation, Dean said, one system withstood a failure of 1,600 servers in a 1,800-unit cluster.

Next-generation data center to-do list
So all is going swimmingly at Google, right? Perhaps, but the company isn't satisfied and has a long to-do list.

Most companies are trying to figure out how to move jobs gracefully from one server to another, but Google is a few orders of magnitude above that challenge. It wants to be able to move jobs from one data center to another--automatically, at that.

"We want our next-generation infrastructure to be a system that runs across a large fraction of our machines rather than separate instances," Dean said.

Right now some massive file systems have different names--GFS/Oregon and GFS/Atlanta, for example--but they're meant to be copies of each other. "We want a single namespace," he said.

These are tough challenges indeed considering Google's scale. No doubt many smaller companies look enviously upon them.

TomTom , Europe's largest maker of car-navigation devices, Wednesday received approval from the European Commission to buy digital-mapping company Tele Atlas.

The deal was accepted by the EU without conditions after a six-month antitrust probe. The deal is worth $4.5 billion and is expected to be finalized in June, according to a statement from TomTom. Both companies are based in the Netherlands.

For TomTom and Tele Atlas, this was "the best possible outcome allowing the new combination to go ahead with the full execution of its strategy," the companies said in a joint announcement.

The decision came a week ahead of a May 21 deadline, suggesting that even bigger deal under scrutiny by the EU will go through. Nokia, the Finnish mobile handset manufacturer, wants to buy the world's largest company in digital mapping technology, Chicago-based Navteq for $8.1 billion. The deadline for that review is August 8.

Both deals were accepted by the U.S. antitrust regulators last year without further investigation and the fact that the EU took a closer look surprised many industry experts. But as the potential for location-based services is growing, the commission wanted to make sure that these two deals--creating a virtual duopoly in digital mapping--would not hinder competition.

"I am now satisfied that the innovation and competition we have seen in satellite-navigation devices until now will continue after this merger," Neelie Kroes, EU Commissioner for Competition in Brussels, said in a statement Wednesday.

Navteq and Tele Atlas produce digital maps and software for navigation systems in cars and portable navigation devices such as Garmin or mobile phones. They also provide the data for Internet maps on sites like Google Maps.

Nokia argues concerns of the regulators that its acquisition will restrict the access for others to digital maps as unfounded.

"Why would Nokia pay the amount of money we are paying for having a very good base of customers and then try to aggrieve those customers?", Michael Halbherr, vice president of Context Based Services at Nokia said in an interview earlier this week with CNET News.com.

Nokia executives believe the Navteq-deal is related to the decision on Tele Atlas. Both companies have refused to accept demands from the regulators that they should guarantee that digital maps will be available for everyone. "I think that the Tele Atlas deal is not being reviewed in isolation. The European Commission has had enough time with both deals to basically form a complete thinking about the industry," said Michael Halbherr.

For Nokia, location based services is very much a promise of the future. The company expects to ship 35 million GPS-enabled phones this year.

From a European industry point of view, these deals consolidate world dominance of location based services in the home base. Navteq is the world's largest maker of maps used for car navigation, while Tele Atlas is No. 2.

In the U.S. the development is seen as positive. At ESRI, a company that sells software for geographic information systems, founder and Chief Executive Jack Dangermond welcomes more concerted efforts.

"The world needs utility companies that create and manage geospatial data," he said. "And these are companies that have fought their way to the top and they're very good: they serve their data and sell their data to our users in the private domain."

One company that has teamed up with ESRI is Google Maps, which now use digital maps from both Navteq and Tele Atlas. John Hanke, head of Google Maps and Google Earth, said that there is a vast amount of geodata locked up in different government agencies that should be made publicly available.

"I don't think it will happen anywhere in the near future. It's hard to share that data, to export it, but they should open up their servers as its public information," Hanke said during a break at the Where 2.0 conference in Burlingame, Calif.

Dangermond agrees. There will be no shortage of data despite concentration in a few hands.

"There are others, like Openstreetmap, emerging startups that I think also will participate in a smaller scale and then also individual agencies like cities. They all create and maintain their own data. It will remain an open and competitive environment I think."

He said he doesn't see a duopoly problem.

"There will be open competition and both of them are sincerely interested in the geo market. By having two competitors they're fighting against each other and keeping the prices low. If there was only one, then I might have a different opinion, but I think with two it's a healthy market. And it doesn't prohibit other people from getting in."

"They are doing the world a great service by taking on the responsibility of building these infrastructure layers."

He also said he feels the companies should be compensated.

"Well, free, they have to pay for all their investments. I suppose they are spending hundreds of millions of dollars a year just to create and maintain these infrastructures. So this kind of investment needs to be capitalized and get returns. But they both are good citizens in the GIS world at this point."

Google Maps launched a new feature on Wednesday that shows you photos, videos, and maps that people have created about different locations around the world.

After typing in a city or address there is a new "Explore this area" link near the top left. Clicking on that brings up thumbnails of photos and videos with links to more of each, as well as custom maps for that location.

For instance, a search on San Francisco brings up photos of the Golden Gate Bridge, the city enveloped in fog (a not uncommon sight), and sunsets. There are videos of penguin chicks at the San Francisco Zoo and video from a YouTube user event.

Underneath that is a list of popular searches (such as parking garage, De Young Museum) for the location and then user-created interactive maps, including one showing the scenic 49-mile drive and car chases from the movie "Bullitt."

Dragging and zooming in on the map brings up different options for those results, and clicking on the icons on the map brings up a pop-up box with more information.

In separate but map-related developments this week, Google also has added real estate listings to maps and has been testing face-blurring technology in Street View.

The news is coming out of the Where 2.0 conference in Burlingame, Calif., where Poly9 unveiled on Tuesday a Flash-based Google Earth competitor, Free Earth, which doesn't require a client-side download.

Zillow, watch out.

Google Maps now can show real estate listings, presenting pushpins that show houses for sale.

To show real estate results, click "Show search options," then select "Real Estate" from the drop-down list. The Web site then shows a list of properties for sale on the left tied to pushpins on the map on the right.

Search results can be refined by specifying price range or number of bedrooms and bathrooms. In addition, there's a text mode that will be more familiar to the classified ad crowd. (Huh? Text mode for a mapping site? There's still a small map visible.)

I can't help but be reminded of one of the first really interesting uses I encountered of Google Maps, a mashup by that showed Craigslist real-estate listings overlaid on Google Maps. At the time, back in 2005, Google hadn't released its maps programming interface, but times have changed. Not only is the API public, but the programmer behind the service, Paul Rademacher left DreamWorks Animation and joined Google.

(Via Google Maps Mania)