Study lauds open-source code quality

A consulting group that scrutinizes the source code underlying several operating systems has found that a key networking component of Linux is of higher quality in several ways than that of competing closed-source software.

Reasoning, which sells automated software inspection services, scrutinized part of the code of the Linux and five operating systems, comparing the number and rate of programming defects. Specifically, Reasoning examined the TCP/IP, a key networking technology, and found fewer errors in Linux.

"The open-source implementation of TCP/IP in the Linux kernel clearly exhibits a higher code quality than commercial implementations in general-purpose operating systems," the company said in a report released last week. Reasoning also compared the code with that used in two special-purpose networking products and found it superior to one of them.

The Linux defect rate was 0.1 defects per 1,000 lines of code, Reasoning found. The rate for the general-purpose operating systems--two of them versions of Unix--was between 0.6 and 0.7 per 1,000 lines of code. The rates for the two embedded operating systems were 0.1 and 0.3 per 1,000 lines of code.

Source code is the collection of instructions written by people and later translated into "binaries" that computers can understand. Companies such as Oracle and Microsoft typically sell binaries incomprehensible to humans rather than the comparatively understandable source code.

Reasoning's findings help to validate the views of open-source advocates, such as Eric Raymond, who argue that the wider scrutiny possible with open-source software means that problems are found more quickly. "Given enough eyeballs, all bugs are shallow," the reasoning goes.

It's an argument that Reasoning Chief Executive Scott Trappe agrees with.

"Open-source applications...allow anyone to look at the source code. For major open-source applications, such as the Linux kernel, the Apache Web server, etc., dozens or hundreds of people will read the source code either to learn how it works, make modifications or look for mistakes," Trappe said. "Because the development process is also open, these independent reviewers can report the defects they find and even suggest appropriate fixes."

"Unfortunately, this process takes too long for most commercial product development cycles," Trappe said.

Reasoning declined to disclose which operating systems it compared with Linux, but said two of the three general-purpose operating systems were versions of Unix. The comparison was done with version 2.4.19 of the Linux kernel. For the comparison products, the company had access to the source code that for proprietary software is usually a closely guarded secret.

Prevailing versions of Unix on the market today include Sun Microsystems' Solaris, IBM's AIX and Hewlett-Packard's HP-UX. They compete with Linux from companies such as Red Hat and SuSE, as well as Microsoft's Windows.

Microsoft, a strong advocate of proprietary software, has backed off its earlier legal argument against the General Public License (GPL) that governs Linux and many other open-source projects. The company had argued that the "viral" open-source software license could force other software projects to become open-source as well if used together.

Now seeing more benefits to sharing its source code, though, Microsoft has begun letting some countries look at the code behind Windows and even build versions of the product themselves.

Reasoning looked for programming problems such as memory that was marked as free when it was in fact still in use, memory that was being used without being properly initialized and attempts to store data that exceeded the space reserved for it. This last problem is often associated with buffer overruns, a major weakness that under some circumstances can let an attacker take over a computer.

Trappe said his company didn't measure the comparative performance of the different versions TCP/IP, something that would have been difficult because of hardware differences such as network acceleration hardware on the network-specific products.