Underexposed

Chris DiBona's job--manager of Google's open-source programs--is a balancing act.

Google consumes a lot of open-source software for its own highly profitable business. But as he oversees the search powerhouse's open-source work, DiBona has to ensure that the company reciprocates. It can't be all take and no give.

Free and open-source software advocates can be powerful allies--but also vocal critics. For example, some have critized Google for its lack of support for the Affero GPL license, which can require those using software for a publicly available network service to share modifications they've made to an AGPL software project.

DiBona thinks Google strikes the right balance, though, by offering its own modifications back to many open-source projects, advocating the philosophy in general, and trying to nurture the next generation of open-source programmers.

DiBona has been steeped in open-source software for more than a decade. Before his job at Google, he worked for Slashdot, still an influential virtual water cooler for open-source discussion. Slashdot was part of Linux server maker VA Linux Systems, which had a spectacular initial public offering in 1999 followed not long after by a drastic cutback.

DiBona will be preaching the open-source gospel at the Google I/O conference Wednesday--"open source is too good to be true and thus must be magic," according to the agenda--but I sat down with him beforehand to hear his view of open-source software at Google.

What's the view of open source within Google?
I asked myself, "Who am I trying to address?" The world of open-source business? No. The world of the open-source enthusiast? No. I'm really looking to work with open-source developers. We came up with these goals for our group: to support open-source development in general, which means to support open-source infrastructure; support the release of open-source code, from Google and in general; and to create more open-source developers, because especially when I started, there was a perception that Google took a lot of people from the open-source world and then went away. It was partly true, because people would come here and say, "Wow, I've been working on my open-source project forever, and I want a new problem," and we have a very good class of new problem. So they kind of went away.

That was too bad. The last thing we wanted as a company was to hurt the release of open-source software, because we consider it pretty important. We use a ton of it. Every engineer we bring on--how much open-source do they want to use? We have new packages and new libraries being brought into the company all the time. It's our group's job to track that. As we brought people in, we wanted to be sure more open-source developers were being created. So that's where we came up with the Google Summer of Code, and now we have a high-school flavor of that as well. I think we've made a very real impact in creating new people in the open-source world.

I'm curious about maintaining a balance between contributing back to upstream projects vs. maintaining your own internal forks. How do you go through that evaluation?
Google considers some projects more important than others. Obviously the Linux kernel is incredibly important. Every time you use Google, you're using a machine running the Linux kernel. We have a fairly large kernel team, and we employ people whose job is just to work on the external kernel. Andrew Morton is a good example of that. We try to make sure those guys patch out (submit their modifications to the main open-source project) whenever they can. It's usually more dictated by the engineer's time than it is any lack of desire on our part. I always wish we were able to release more, but it takes time for an engineer to do that. For the larger efforts, it's a little easier because there are more personnel on it.

The same thing goes for our compilers (software that translates programmers' code into instructions a computer understands). The great thing about our compiler team is they patch as a matter of their jobs. They're always patching out things from the compiler work we do internally to the outside world. We recently released the new linker, Gold--Ian Lance Taylor works for us on our compiler team. He's been on the GCC team forever. He used to be at Cygnus (a company that developed GCC). We have a lot of ex-Cygnus people.

Then there are Googlers who just want to patch into an existing projects. They found a bug, they want to add a feature. That takes no time at all. Our team looks at the first couple patches an engineer wants to send out, makes sure the engineer knows what they're doing with the outside world, then they're basically given free rein to do that. They keep us posted on what they're patching. We want to make sure our code gets out to the projects as fast as possible because projects keep on iterating. If you don't get your patches in, they won't get accepted, because they'll be too old or won't matter. If you've got a patch, getting it out there fast is better for us, because then as that project iterates and comes back into the company, we don't have to reapply a patch.

What are the most important open-source projects you ingest?
The kernel, compilers--GCC, the Python interpreter. Python is very important to us. Google App Engine--it's a Python hosting system, basically. Java is very important to us, and that's become open-source now. We have some very good Java people working for us--Josh Block, Neil Gafter--they've got a great handle on that technology.

Once you get past those three projects--the compilers, the languages, the kernel--then you go to the libraries. For us that's OpenSSL, zlib, PCRE. MySQL is hugely important to us. Past that, it starts tapering off pretty quick.

Has the open-sourcing of Java changed anything for you?
Not really. I think it had more impact on the outside world than for us. Java is a fairly mature language now. We've been using it for a long time. Before, it was the JCP (the Java Community Process to govern Java's future)--it had the rubric of openness around it. It was never really not so open. There are questions around what open source means now around Java, specifically J2ME (Java's mobile edition for gadgets such as cell phones) and the TCK (the technology compatibility kit).

Are you using a super-uber-customized Linux kernel, or are you guys pretty much vanilla?
I don't think there's such thing as a customized Linux kernel anymore. The kernel is incredibly flexible. It's got all these different architectures. I think the Linux kernel itself is this ubercustomized thing.

But do you have a lot of in-house customizations?
Not a lot. Google is exposed to some interesting hardware before the rest of the world. So internally we'll be sampling code for that hardware. So that's pretty custom stuff. But eventually that goes to the outside world. We funded some work with a group in Berkeley called Xorp to bring high-speed Broadcom networking chip functionality to Linux. It's not in our interest to keep control of it ourselves. So is it customized? Absolutely. But is it heavily customized? I don't think it is as heavily customized as you might think.

Is it true you still use 2.4 kernels?
In some places, sure.

How about for the core search product?
I don't know how it's partitioned out. When you think of Google, you think of search being on top of a kernel that's static. It's not always like that. It differs on data centers. I think 2.6 predominates, though.

There's been discussion about reciprocity. When General Public License (GPL) version 3 came out, the Free Software Foundation dumped the Affero clause out of GPLv3 and split it out into a separate license. Eben Moglen (co-founder of the Software Freedom Law Center and then counsel to the Free Software Foundation) said, to paraphrase, "If Google starts getting too parasitic, then we'll re-evaluate it." How worried are you of getting a negative perception of using more than you contribute?
I do worry about this. I think it is a largely incorrect perception. You can always give out more, and there are always people who will never be satisfied. Could we be giving back more? Sure. One of the ways I ameliorate that problem is (through) projects like the Summer of Code. Google is releasing every year, not counting Android or the really large open-source projects like GWT, a new project every two or three weeks. Or patching hundreds of projects a month. I conservatively estimate we're releasing about a million lines of code a year from the company.

If you talk to open-source developers--people who are working on projects--I think they understand that. It came back to who do we want to interact with. I always felt the enthusiast community would understand that eventually, and I think that's true. There are some people who are upset with us because we didn't embrace the Affero-style GPL, but it's not practical for us to do so. When they had an Affero-style clause in GPLv3, the thing I told Eben was, "Listen, you can adopt whatever you want. We'll still keep on backing up the FSF and the SFLC as much as we can, but it means we won't be able to use that license inside, because it won't be practical for us to do so." I think that's a very realistic response. The Affero GPL is out there. That's great for the people who use it. It's just not for us.

That's the thing about free software. You're not obligated to use it. We have enough fine-grained control within the company that we don't use things we don't want to use.

What are your preferred licenses?
We generally release under the Apache License--Apache 2. We think it has the fairest language of the licenses. And the GPL requires a lot of management--more than we have time for to run a project well under that license--patch flow and all that. Apache 2 encourages people to take the thing and run with it. That's what we're going for when we release code, whether it's to have people adopt technologies we really like, or for API examples. That said, we've released things under the GPL, LGPL, GPL version 3, BSD. We default to the Apache License.

To what extent to you subsidize gurus to sit around and work on important projects?
We've got people like Jeremy Allison and Andrew Morton and some of Guido (van Rossom)'s time. He's been working pretty heavily on Google App Engine and Mondrian. It's more common that we...try to make open source a part of their job, so they're patching out to the libraries they use. We think that's more healthy than having people whose job is just working on an open-source project.

You use open source a lot internally. Do you have some kind of intellectual property vetting or review before you use it?
We do. There are two ways we do this. When somebody wants to bring a piece of code in from the outside world--open-source or commercial--you need to put it inside a special directory we call "third party." They're required to put in a file called readme.google (that describes) where they got that software, how it's licensed, what category that license falls under. We look for things that are obvious. There are some projects that have dubious intellectual property provenance, and we know those, and we know the people who run them, and we tend not to use those ever.

Since Google doesn't distribute a lot of software, we have it easier than companies that ship hardware and software. We have a couple situations where that does happen--the Google Search Appliance, some of the downloadable applications. Those get a little extra attention. Similarly, when we have larger projects like Google Android, we have a higher ceremony--every two weeks we get together and see if the license picture has changed.

The tracking model works really well for us. We have tools written where a program manager or a release manager can turn on a certain level of warning within the build tool and it will tell them what open-source software they have and how they have to comply with it. At that point we set up a mirror for them as they get closer to release.

So that's the first way we track things. The second way is whenever a Googler puts in a changelist now--this is something we're just starting to do--we compare it against all known open-source code on the Internet using our Code Search product. We compare the changelist that comes from your average Google engineer against that database of code and we look for intersections. When we find an intersection, we take a look and see if it's truly a copy. And if it is, we make sure it's in the right directory and that it's properly labeled. And we call up the engineer if it isn't and make sure it gets tagged properly so we can do the right thing by these licenses.

That tool is kind of in its infancy. We're trying to figure out ways to automate what it does. But it's great because it scales programmatically. Our group's goal is not to break builds or stop development. It's to enable developers to use as much open-source as possible. We think it's healthy, because then they're not writing that code, they're writing other code.

Do you vet code for patent or copyright?
No. We have legal people on our lists. We have two main lists that track these things. Open-source licensing for incoming code and open-source releasing for outgoing code. Legal has a presence there. Patents are incredibly tricky.

Is it easier to get hired at Google if you have experience maintaining your own open-source product or patch?
If you have made a name for yourself in open source, clearly it helps. If you have a healthy project in open-source, I believe it helps. One thing I see on hiring committees is when somebody has an open-source history, it's really great. You can just look at that history. Interviews are great, but they're not very deep. They're only 45 minutes long. So how can you really get a feel for if a person is good at programming, at computer science?

Or at social relations, for that matter.
Open source really reveals that incredibly quickly. You can look at their code, at their activity on mailing lists, how they deal with bugs from real people, and real user problems. That's an incredible resource.

The Summer of Code isn't really a recruiting program. If it is, it's a really expensive one. Last year we created about 2 million lines of open-source code across the 900 students who took part. Of those probably a third are going to stick around with their projects, because the rest have to go back to college.

We have a couple students who have been in the program two or three years. The whole point is to support kids over the summer so they can go and program and not get some other job that has nothing to do with computer science. It's our fourth year doing it. This year we've go 1,109 students doing it across 95 countries.

Releasing 8.6 million lines of source code and expecting open-source programmers to join Google in its development is a technological challenge.

But when Google does make its Android mobile phone software an open-source project later this year, it looks likely it will take a page from the Linux playbook and use a tool called Git to manage that part of the work.

Linux leader Linus Torvalds originally developed the Git source-code management software in 2005. He didn't like available open-source tools for the chore, but encountered resistance in using a proprietary tool, BitMover's BitKeeper.

Torvalds liked the distributed approach enabled by BitKeeper and Git, in which individuals could maintain their own "trees," variations of a project that branch off a main trunk. Git also can be used to track and manage software patches sent "upstream" by contributors working on code branches to the programmers responsible for maintaining various open-source projects.

Google currently uses a source-code management tool called Perforce to manage Android, but the company is moving to another code repository technology in preparation for moving Android into an open-source project, said Android leader Andy Rubin.

"We need an open-source repository. Currently we're on Perforce. That has to be moved to Git," and there's an effort now to make the transition, Rubin told me in an interview about Android.

That sounded to me like Android had settled on Git, but Rubin wasn't willing to go that far. "We have no announcements at this time," he said.

Maybe we'll hear more at the Google I/O conference next week for programmers interested in Google's work. One theme of the conference is Android.

Benjamin Lynn of Google's developer programs group offered a basic guide to Git on a Google open-source blog posting this week. And Google uses Git elsewhere, for example, to help Linux kernel programmers with support for Qualcomm mobile phone processors.

Junio C. Hamano currently maintains Git.

One choice Google won't pick for source code management is the centralized Subversion software.

"Subversion we don't think is enough of a repository to handle 11 million lines of code. If this is adopted, and there are 10,000 people checking out, it'll die," Rubin said. (Android today consists of about 8 million lines of Linux code plus 11 million lines of higher-level code; of the latter, about 8.6 million will become open-source software.)

• Novell Financial Results for Fourth Fiscal Quarter and Full Fiscal Year 2007. $22 million was from Linux Platform Products, up 69 percent year-over-year
• Photoshop Insider » Nikon D300 Review by Scott Kelby. The Photoshop guru likes the D300. Autofocus, lower noise, sensor cleaning, HDMI output. Wants: noise as low as the D3. "The only thing that I can think of that my D2Xs has, that my new D300 doesn't have, is High Speed Crop Mode."
• Auto Adjustments; Always, Never, Occasionally?--O'Reilly Digital Media Blog. Are photo auto adjustments finally getting worthwhile?
• Scary Santa pictures. 178 photos of various children crying on Santa's lap. Holiday cheer my left hind foot. Thank heavens my parents never made me do this.
• Google Zeitgeist 2007. Google's Zeitgeist reflects the fleeting popularity of search terms. Alas, Sudoku has faded from 2006 :(
• Sun releases Netbeans 6.0. New features include the ability to code not just Java software, but also C, C++, Ruby, Ruby on Rails, JavaScript. Open-source software.
• Google Developer Podcast: Android with Dianne Hackborn and Jason Parks. BeOS and Palm coders talk Android.
• Google Code Blog: Embed charts in Web pages with Google APIs. Cool embeddable chart feature. I couldn't figure out how to define data sets on first glance, though.
• Books: None of the Above: Books: The New Yorker. Malcolm Gladwell reviews James Flynn's "What Is Intelligence?": "An I.Q., in other words, measures not so much how smart we are as how modern we are."
• 451 CAOS Theory » Open source fashion?! Nah.. The fashion industry picks up on the fashionable term "open source."
• JBoss Founder Marc Fleury Joins Appcelerator Board of Advisers. Outspoken JBoss ex-CEO can't stay out of the fray after all.

Google's Android software gives Sun Microsystems' Java technology a starring role--but not the version of Java the rest of the mobile phone industry has been developing since the 1990s.

Instead, Google struck off on its own in an attempt to improve performance and openness for the software used in the Open Handset Alliance phones. That means programmers will have a new variety of Java to reckon with--offset somewhat by Google's $10 million code contest to draw developers in.

One difference is Google's development of its own core Java virtual machine (JVM) technology called Dalvik, the software that actually executes Java programs on an Android phone, which Google says means Java programs run fast even on the constrained hardware of mobile phones. But a more significant departure than just using an in-house JVM is the fact that Android isn't part of the Java Community Process that Sun established in 1999 to oversee the development of new Java features.

The JCP governs Java by codifying new features as application programming interfaces (APIs), so programmers can have a standard way of calling upon new technology such as Bluetooth support or 3D graphics. But that existing Java realm wouldn't accommodate the developer freedoms Google thought were important in Android.

"We wanted the platform to be open in a lot of different ways," said Mike Cleron, a Google senior staff engineer working on Android. "The idea is that anybody can come along and replace the pieces of the Android experience on a very fine-grained level. The existing APIs didn't really allow the level of openness we were hoping to achieve in Android."

It should be noted that Google isn't working in a Java vacuum. For example, one of the OHA partners, Motorola, has helped lead development of Java for mobile devices, and Google wants to keep the Java programming experience familiar to developers. And Google is an executive committee member of the JCP, though only for the Standard and Enterprise editions that run on PCs and servers, not the mobile edition for phones and other devices.

"We have people on the team who are active in the Java community. They've been helpful in informing us and guiding us, making sure what we were doing is familiar to folks in the Java community," said Steve Horowitz, Android's engineering director.

Further fragmented?
But the bigger issue is whether Google's effort will worsen the already fractured world of Java. Not all phones support all the same Java standards, so programmers can't be sure that their software will run on a multiplicity of devices, as the "write once, run anywhere" Java tagline promises.

"They are using Java, but they aren't implementing any well-known Java framework, and really that just creates another standard to support. The risk they take here is that they might fragment the market further," Benoit Schillings, Trolltech chief technology officer, told my comrade Maggie Reardon. Trolltech, which sells tools and components for programmers whose software runs either on PCs or on mobile phones.

Mauro Lollo, CEO of mobile phone video-streaming company Movidity, saw Google's work similarly. "In essence, they've created another standard. Standards are great, but the challenge is that there are so many of them," he said.

Google also faces a common risk of open-source software, that the openness will mean programmers can "fork" projects in different, incompatible directions. (Indeed, this was one of the earlier reasons Sun resisted its eventual decision to make Java open-source software.) "In the end, you could have 20 different versions of the Android technology that are incompatible, because anyone can take the license, modify it, and create another variation," Schillings said.

For its part, Sun supports Java and open-source software on mobile devices, but expessed some caution about joining Google's alliance. "We were interested in being part of the Google ecosystem, but we were interested in getting more clarity on what this program entails," said Rich Green, executive vice president of Sun's software effort.

Asked if there's any possibility of unifying the Android work with the Java Community Process, Horowitz said, "It's an open alliance. We can welcome anybody who wants to join."

Android uptake
Techno-politics aside, Google clearly has grand aspirations for Android. And it wants outsiders to be part of the development.

In stark contrast to Apple, which plans to release a software developer for its iPhone in February, half a year after the product began shipping, Google is releasing its SDK about a year before any Android phones ship.

"We're making it available pretty early--early enough that we can get feedback at a point where we can still impact the direction of the software," Horowitz said. "People tend not to ship SDKs until the products are done. In this case we thought the platform was such an important part that we wanted to get that out early."

Of course, there's another advantage to releasing an SDK early: the open-source community can help build interesting applications that give Android phones more than just the basic set of programs.

So far, so good, said Horowitz, pointing to "unprecedented" interest in Android compared to other projects hosted at Google's open-source projects site, code.google.com. "It is above and beyond anything Google has seen to date," Horowitz said.

Among details in the SDK:

• It makes mention of support for GSM mobile phone networks, the leading technology for mobile phone networks, but is silent on support for the top rival, Qualcomm's CDMA. That will come, though, Horowitz said, pointing to CDMA allies such as Qualcomm that are members of OHA. "It's clearly something on the roadmap, but we're not talking about specific support for it at this time," he said.

• OHA supports touch-screen technology, but Horowitz declined to comment on support for multitouch, a notable iPhone ability that opens up user-interface possibilities, beyond saying multitouch support isn't in the first version of the Android SDK.

• Google will release a new version of the Android SDK once feedback from programmers starts coming in. "We're committed to a regular release cycle," Horowitz said.

• Software should run quickly on mid-range phone hardware such as those with a 200MHz ARM 9 processor. "One of the key goals of the project was to ensure we can run on a broad range of phones that don't require a high-end processor at all," Horowitz said. "When we bring it to higher-performance devices, it's just going to scream."

• The SDK so far permits development only of software that runs on the Java foundation, not natively on the hardware itself. "We are aware of the interest in native application development, but we having nothing to comment on right now," Horowitz said. But performance shouldn't be an issue: "Our system is designed to take full advantage of native code in performance-critical areas and expose this functionality through our framework APIs."

Marguerite Reardon and Dawn Kawamoto contributed to this report.

Correction: An earlier version of this blog misstated Google's connection to the JCP. Google is a member of the Java Community Process, though not for the Java Mobile Edition version to which the Android software is most closely related.

Google on Monday released programming tools for its Android mobile-phone alliance for download, giving developers the ability to start writing software for phones due to start shipping in 2008 and $10 million in prizes to lure them.

The software development kit (SDK), an open-source package available for download for Windows, Linux, and Mac OS X machines, shows that Java is indeed the programming language for software running on the Linux-based phones.

Accompanying the SDK is a raft of details that wasn't available when Google and its partners announced the Open Handset Alliance a week ago. The Android software includes the Google-created Dalvik virtual machine for running Java programs, a browser based on the WebKit engine, and support for many media and image file formats. (Note: I clarified that the browser is only based on the WebKit engine.)

And hardware abilities permitting, it also supports wireless communications using GSM mobile-phone technology, 3G, Edge, 802.11 Wi-Fi networks. Conspicuously missing from the list is the widely used CDMA mobile-phone technology developed by Qualcomm.

To jump-start the Android programming effort, Google is offering $10 million total in prizes, each ranging from $25,000 to $275,000, to programmers picked by a panel of judges.

Android programmers can use the open-source Eclipse programming tool, founded by IBM and now supported by many companies, along with an Android plug-in for Eclipse.

The SDK includes an emulator so programmers can write software even without phone hardware. However, as programmer Jason Chen cautions on his blog, "The look and feel of the user interface in the emulator is a placeholder for a final version that is under development."

The SDK also describes application programming interfaces (APIs) that enable programmers to take advantage of underlying support for location-based services, video and audio streaming and playback, and 3D graphics. However, support for Bluetooth and 802.11 wireless networking APIs isn't yet available, though they'll be added to the SDK, the site said.

Google mentions support for two APIs for using Google services, too: Google Maps for displaying maps and XMPP for device-to-device communication tasks such as playing checkers.

Forgive me if I appear a little skeptical here about Google's Open Handset Alliance. By my count, it's the fifth consortium so far to attempt to craft something useful for mobile phones out of Linux and open-source software.

OHA has by far the highest profile, it's got the most persuasive list of members, and its timing is the best. What's not yet clear is whether the "Android" work of Google and its allies will unify or further fragment work in the area.

Rallying programmers behind a unified effort could help determine whether this effort will accomplish more than the Linux Phone Standard (Lips) Forum, the Open Source Developer Labs' Mobile Linux Initiative, the Consumer Electronics Linux Forum (CELF), and most recently, the LiMo Foundation begun in 2006. Related efforts one step removed include Intel's Moblin and, Nokia's Maemo, and any number of other open-source projects.

Just as with PCs, somebody has to write a "stack" of software spanning from basic operating system functions all the way through communication utilities, user interfaces and Web browsers. Unlike PCs so far, though, the mobile phone market has suffered from a profusion of incompatible software foundations, despite some efforts to use Linux and Java to bring some common ground.

... Read more