Does open source need consolidation?

I was reading this OStatic interview with Ken Drachnik, marketing manager for open source software infrastructure products at Sun and a co-founder of GlassFish (Sun's open-source application server), and it made me wonder if it's time for some consolidation in the open-source stack. Yes, I'm the one who argues against consolidation in enterprise software, but another part of me wonders why we spend time reinventing wheels....

Yes, we have de facto winners in most software categories: SugarCRM in CRM, JBoss in application servers in enterprise adoption (with Tomcat winning out for unpaid deployments), MySQL in the database market, etc. Think of how much better these projects would be if we concentrated development on these, rather than creating a range of lightly developed and even more lightly used open-source alternatives.

I guess my underlying question is, "Do we need a myriad of open-source alternatives to the proprietary software stacks, or would we be better served with one or two rock-solid open-source alternatives?" I'm inclined toward the latter, as I think Linux, for example, is much better off for having three robust competitors (Red Hat, SUSE, and Ubuntu), rather than dozens of also rans with no strong options.

To make this musing more concrete, would we be better off having Sun contribute its Solaris expertise to Linux? Should Red Hat give up on its ESB efforts and contribute to the Mule project? Should IBM dump its Geronimo efforts and give code to JBoss instead? And so on.

Thoughts?

[rizarsurf]

Divide and conquer. Amazingly the only thing that has held back Linux and Open Source from becoming mainstream players is the many distributions and way too many choices. It is a haphazard array of great coding. Hardware manufacturers do not want to make several drivers for their goods. And they shouldn't. They should only have to make one Linux driver that works on Linux and that's it. It works. Right now the open source landscape has a lot of excellent programs out there, but it's very difficult to pick the one. Choice is good, but having way too many things to choose from is very bad.

[odubtaig]

After having to explain to someone who just about knows of the existence of Red Hat (and is vaguely aware that they're not the only vendor) the other day that SUSE is a) a linux distro and b) second largest commercial seller I feel even safer in saying that the actual number of distributions any vendor or driver writer actually needs to care about is equal to or less than five.

Yes, there are 300+ distro, 295+ of which are niche, experimental, special purpose or written for otherwise obsolete hardware like PPC Macs and in the occasional case rebadged exact clones of other distros for legal reasons (see RHEL and CentOS).

Really, if anyone's trying to work commercially and theyre writing for anything more than Ubuntu, SuSE, Fedora, Mandriva and maybe PCLinuxOS they're wasting their time. I wouldn't even expect most vendors to bother past the first three.

[The_Decider]

Go to hardware vendor websites who have Linux drivers. A good one is www.nvidia.com

Look at the various Windows drivers, not look at the Linux drivers: There is one!

It works on opesuse, fedora, ubuntu, etc.

Why? Because it compiles against the linux kernel that is running. There aren't drivers for each distro.

Nice attempt at some FUD though.

[anon8mizer]

No. Open Source does not need consolidation. Open Source needs product managers.

[pjhenry1216]

A myriad of choices is the double edged sword for open source. With that much choice, you're almost guaranteed to find something that suits your needs. However, given so much choice you can see the problem (exaggerated so my point) clearly if you think of it as having a finite amount of development spread amongst an infinite amount of projects.

Choice makes open source awesome, but it also hinders it greatly. Its kinda paradoxical.

[pablonhess]

My opinion is that the best scenario would be having one or two rock-solid Open Source alternatives **for enterprises**, with lots and lots of wheel reinvention going on in the community alternatives.

This way, it would be much easier for companies to choose an Open Source software to replace their proprietary ones, while keeping the chaotic inventiveness of OSS programmers at the community projects.

If/when one of those community projects reached the maturity, quality, performance and realiability levels of the rock-solid enterprise alternatives, it might even replace them and become (one of) the de facto winners.

As I see it, this would be the "perfect" ("most desirable", if you will) dynamics for the enterprise Open Source software market.

Oh, and yes, I think Sun SHOULD contribute its expertise to Linux, as well as IBM with JBoss and Red Hat with Mule. Open Source is only really interesting for the companies developing it when someone else (i.e. other company) contributes with the project.

[Matt Asay]

@pablonhess: I think this is dead on. I just don't know how to select out those two "for enterprise" projects in each category. I may have my favorites, but yours would likely differ. Still, I 100% agree that we want to keep all that "wheel reinvention" at the edges.

[splendidcrm]

Isn't the whole point of open-source to encourage derivatives? SplendidCRM, for example, aims to produce a near identical SugarCRM but on the Microsoft .NET platform. While you might think that SplendidCRM and SugarCRM could merge, I suspect that doing so would lead to a culture clash. The SplendidCRM folks obvious praise Microsoft and the SugarCRM developers probably bashes Microsoft.

[jamesurquhart]

I think its important to look at this from the prospective of a complex adaptive system. Yeah, there are a lot of "agents" in the system today, but look at your example of Linux. In that specific system, the market chose the strongest players and killed off or weakened the rest. New distributions (and OpenSolaris) have difficulty knocking those established players out of their place in the ecosystem, but the constant challenges keep the latter on their toes.

The rest of open source is just a few steps behind. Yeah, there are too many CM alternatives out there, for instance, but you can already see the market consolidating around the strongest "agents". In five years, there will be just a few open source alternatives in that space that risk-adverse enterprises will even consider.

Again, even after the strong agents have established a beachhead in the system, the introduction of new competitors in the space brings in new "DNA", which challenges the established players to adapt to survive. Are they distracting? I guess, but do you really think open source would thrive better off of a few dedicated communities, all of which are competing to get their features into the limited distribution options dictated by a few strong projects?

Think evolution. Think "only the strong survive". Think "watch out for that new project, it's disruptive!"

[ewriter21]

I agree and disagree. The reinventing of the wheel is very real but in the reinventing is the finding of opportunity and action of innovation, invention, a better way, an easier way and sometimes a leaping off point for something altogether different.

To quash all that opportunity by consolidating an open community would be akin to not using past algorithms or languages to teach computer science.

Should Linus have helped make UNIX or Solaris or Windows a better OS instead of reinventing an operating system?

I think the key is to separate the learning, the fun, the interest and the variety from the commercial successes and the ecosystem building. As an ISV, I can't afford mentally or financially to target every single Linux distro, app server, database, message bus, etc. and I won't. That doesn't mean all those projects shouldn't exist and it doesn't mean one of them might not become more compelling in the future than the first one I choose.

[Renegade Knight]

We need both 800lb gorilla's and the side applications that may break now ground or provide the next gorilla. I'm not sure that thousands of non stand out applications helps with the main stream.

[odubtaig]

I think what most Linux projects need is better marketing. I've just wasted a goodly amount of time trying "open" project management software, one of which had an 'export to PDF' menu option that redirected me to their SaaS product and the other of which had a similar agenda (and was only partially open source). I finally ended up with GanttProject for my immediate needs which is a little buggy, slightly difficult to use and lacking in some minor features but, most importantly, exports to png, jpeg, html, pdf and some other things besides.

I'll have to find something more suitable for more detailed scheduling but for a prelim report it'll do.

So, what's the real problem here? I have no problem with lots of projects, plenty will be experimental, there's always the possibility of cross-pollination between projects if one comes up with something the others haven't thought of and if something of a higher standard presents itself it'll hopefully get the most attention. Not only that, but there always is a certain amount of consolidation if two projects have weaknesses where the other has strengths.

But, the problem here is that there's not necessarily a lot of the kind of promotion these projects need. I had to go through two complete duds (both of which were covert sales pitches) before finding anything useful, how many developers and users are not finding the more useful software because it's buried below badly thought-out reviews and superior self-promotion?

It's something that's already worked very well for the top Linux distros, OO.org and others.

[rizarsurf]

There should only be one "main" Linux distribution and from there anyone can fork and do whatever they want. That would allow drivers to be written for that "main" distribution and they would be compatible with almost all derivatives or forks.

[odubtaig]

That's a nice theory, but it's also exactly how Linux started out and look what happened to that.

[jrepenning]

One hallmark of pure-play open source has always been "cheap failure": there might be a hundred "competing" implementations of the same thing, but 95 of them are DOA ... an no one's any the worse for that, because no one invested payroll and infrastructure and commitments in them. Commercial open-source loses that: as soon as you pick one, you need it to be the one to succeed.

You've always been pretty commercial-open-source focused, Matt, so maybe you're the one to answer the question: if it's attractive for commercial-os to give up this great pure-os value, how can commercial-os compensate for the loss?

[The_Decider]

No

I download the same software and drivers and it works on all the various linux distros I run, that is if they aren't in the repository, which solves all the problems.

You don't know the first thing about Linux, so why do you insist on commenting on it?

[Beenthinking]

Is this a self serving question? Or does it pertain to the amazing creativity and freedom that open source allows for? By that I mean, your incentive as a stake holder in Alfresco can be interpreted to say, "like the traditional proprietary software market, wouldn't it be nice if consolidation through profitable acquisitions existed"? Pardon me but open source provides its customers with the ability to chart their own course, be in control over their own destiny. Why would Google be handcuffed by Microsoft's systems agenda? Why would Yahoo? Why would any company? So, through consolidation, would this bring about the same constraints? OR, would consolidation provide open source company's with much needed liquidity for their investors? I'm not suggesting this to be a bad thing, I'm wondering about the motivation to write the article. What does it really add to the open source discussion? Who cares how many linux distributions there are? People will select what works for them. Isn't that the freedom and the promise? As soon as you pick one, you don't need it to succeed for the market, you need it to meet your needs. You have the source, you fix it, you add to it, you create what you need. Thats the beauty of it.

[kennonk]

I believe that falling back onto traditional concepts like consolidation is counter productive in a free software world. We need the zillions of bits of code so that companies like Red Hat, Novell and Canonical can compile the "best" into a distribution. The Bazaar is where the best work is done. Open Source isn't just about seeing the source code, it is a eco-system in which the chaos is required for progress. Once you build any walls around that, once you try to direct that energy you begin snuffing the spark that made Linux go from hobby OS to king of the data center in 10 years. ESR might be a little far afield with some of his early ideas but the foundation of his and RMS's philosophies are sound IMHO.

[pelegri]

I think the current dynamics are fine. There are simultaneous competing trends inside the Open Source community towards consolidation (everybody wants to be in a winning team, the larger the more users/customers/partners) and towards innovation (I think I have a better idea). I think the single strongest benefit of Open Source is its Darwinian dynamics that, over time, almost always improves the products.

Besides, I'll bet you will encounter people that will argue with your "de facto winners" for pretty much any "software area" you might chose :-). I'm very biased, but I'll argue that GlassFish momentum is quite stronger than JBoss right now. JBoss probably still has more deployments, but GF has many more downloads; give us another year, and we will see..

[chinhuawang]

Open source is no different from commercial software in terms of evolution and adoption. Selection will encourage the evolution. Man-made consolidation is called monopoly. Customers or users will eventually converge and settle on a limited number of choice as a result of market development.

I have been in application server space for long time. When we just started, there were dozens of application server vendors. Now commercial application server is less than a handful. Open source application server starts with Tomcat, then JBoss, Apache Geronimo and Glassfish. Time will tell some will disappear some will grow stronger. If you think there are too many choices, only because the market maturity is not there yet or there are a number of niches.

[dogStar1000]

Consolidation will occur through the operating of a free market in open source.
A free market drives innovation. 'Forced' consolidation, if that is what Matt is talking about, would stifle this.
Perhaps I'm being unfair, but I can't help thinking that what Matt Asay *really* wants (and what's driven this article) is the consolidation of the Open Source enterprise CMS market around a specific, favoured vendor.