CIA invests in open-source enterprise search

If any organization needs to make sense of unstructured data it's the government--especially agencies like the CIA and other intelligence groups that comb through a myriad of disparate information on an hourly basis.

Last week, In-Q-Tel, the technology arm of the CIA, invested in Lucid Imagination, which provides support, maintenance, and add-on software for Apache Lucene and Solr. According to Lucid, the Lucene/Solr technology is downloaded more than 9,000 times per day, and more than 4,000 organizations are using the software for enterprise search.

I've wondered aloud quite a few times as to whether or not open-source projects (and specifically Apache projects) can turn into businesses or if they are simply the cogs and wheels that make other products function better (aka the Oracle syndrome).

I probably would have argued that enterprise search would fall into one of those no-man's lands where the technology is important but not quite a standalone business. There has been a huge amount of venture capital investment in search but few big winners in the category.

But the investment from In-Q-Tel adds some credence to the value of the function as well as the technology in the respect that the government is actually using the software and not just making an investment as we see in the venture capital world. Lucene and Solr are "sufficiently complex" open-source products that require a commercial entity to support ongoing efforts once they are adopted. This gives Lucid a legitimate shot at building a business.

Stephen Arnold highlights three interesting points about the investment. From his story on the topic:

• First, the U.S. government appears to perceive significant potential for Lucene/Solr adoption within its government partners. Several of these entities have already been using the technology and require more reliable, predictable support and services for better risk management. Lucid Imagination provides the commercial backing IQT's partners need in order to deploy Lucene/Solr in mission-critical applications.
• Second, in today's business climate, open-source solutions provide one way to tap into a broad community and its programming expertise. Important innovations often bubble up from open source, thus reducing the time between a good idea and a concrete implementation of a function or feature. Some proprietary systems impose a "time friction" on licensees.
• Third, open source allows some applications to reduce or eliminate what I call the "one way street" that commercial software often requires of licensees. Flexibility can deliver both financial and technical advantages in my opinion. In my experience, it can be time consuming and expensive to "get information out" of some commercial systems or expensive to figure out how to tap or repurpose a proprietary content processing system's outputs.

I would use the same logic for Cloudera and Hadoop (also an Apache project). A complicated piece of software needs to have an organization that stands behind it and makes users successful. I wouldn't be surprised to see Cloudera receive an investment from a government agency using Hadoop.

Generally speaking, this bodes well for open source and exemplifies the ideal that users support projects by giving back code, dollars, or development time in order to make a better product.

Follow me on Twitter @daveofdoom

[wesley_davis2]

The CIA and Lucid should check out the latest Open Source Search Engine Benchmarks here:

http://zooie.wordpress.com/2009/07/06/a-comparison-of-open-source-search-engines-and-indexing-twitter/

Cheers