The "Is VMware violating the GPL" question is circulating again (Matt Asay follows up his own post here) so I thought it would be useful to dust off an Illuminata Perspectives that I wrote when this same thing cropped up about a year ago. I've excerpted the most salient points from the original post and added a little updated commentary.
The basic issue is as follows. As most folks involved with servers know by now, VMware ESX Server is a server virtualization product that allows multiple "guest" operating systems to co-exist on a single physical server independently of each other. ESX provides what is known as "native" virtualization--that is, the VMware software sits directly on the physical hardware; it isn't "hosted" like an application in the manner of, for example, Microsoft's Virtual Server. We usually use the term "hypervisor" to refer colloquially to this layer of software that lets virtual machines be created on top of it.
This is somewhat of an oversimplification, however. ESX Server, similarly to the Open Source Xen Project, actually has two major pieces. One is a virtual machine manager (VMM)--the layer that actually controls the virtual machines. In VMware's case, it's called the "VMkernel" and is proprietary code. The other is a service console that lets the user control and monitor the functions of the VMM. This "Console OS" is based on the Linux 2.4 kernel; it's Open Source under the terms of the GPL.
How Things Work:
The VMkernel and the Console OS are two separate pieces of code. VMware's Zachary Amsden describes how they work (in a comment to the VentureCake blog post that kicked this discussion off last summer):
First, the vmkernel is not a Linux kernel module. The vmkernel is a completely isolated and separate piece of software which is loaded by a module called vmnix. The vmkernel has no knowledge or understanding of Linux data structures or symbols, and as a necessary result, does not depend on the Linux kernel for any services whatsoever.
Second, the vmkernel does not run inside or as part of the Linux kernel. It simply takes over control of the CPU and switches into a completely alien operating mode - one where Linux itself no longer exists. The former kernel used to boot the systems is still alive, but to switch back to it is a complex and involved process, similar to the well-defined copyright boundary of switching between two user processes, which are completely separate programs, running in their own address spaces. The vmkernel and the console OS Linux kernel are two completely separate entities, and the process of going from one to another is even a stronger separation than that given to user processes - more like rebooting the processor and re-creating the entire world.
What's described here is the standard ESX Server product. My understanding is that VMware's "embedded hypervisor," ESXi, uses a Remote Command Line Interface for management that does not rely on a Linux-based foundation.
At issue here is that Linux (vmnix is part of the Linux-based Console OS) is involved with bootstrapping the VMkernel. Does this constitute a form of linkage that would make the entire resulting "work" (i.e. the whole of ESX Server) subject to the GPL--thereby requiring that the source code be made public?
As Matt notes, this looks a lot like what happens when Linux loads "binary blobs" such as closed source device drivers, a long-standing practice that Linus Torvalds has blessed. There would also seem to be similarities to the bootstrap loaders used to bring up Linux (such as GRUB or LILO), which can also be freely used with Windows and other proprietary operating systems.
It is also true that some Linux kernel developers, such as Alan Cox, are on record as questioning the legality of loadable kernel modules with proprietary licenses. So, does this mean as Matt suggests, that all these uses are illegal?
This is a corner case that involves a type of linkage and relationship that the GPL doesn't explicitly cover. It's also a type of linkage and relationship between software components that Linux Torvalds, the Software Freedom Law Center, the Free Software Foundation, etc. are well aware of and have explicitly or implicitly decided that it's something they'll tolerate even if they don't enthusiastically endorse it.
But is it legal? Well, yes, until some court decided it isn't. Which is a battle that none of the most relevant players have any interest in fighting.