Software, Interrupted

One of the headlines that caught my eye today is this blog post from the Amazon Web Services team about a new Memcached as a service offering from Gear6.

For years, Memcached has been used quietly to provide in-memory caching for many popular Web sites, from LiveJournal (for which it was first developed six years ago) to Twitter, Flickr, and Facebook. In the last couple of months, Facebook has opened up about how it scales, and one the key technologies enabling this is Memcached, which services 120 million queries every second. To achieve this, Facebook admits it has had to do some engineering work to improve Memcached's performance and memory efficiency.

Similarly, Gear6 has added features such as replication, clustering, optimized memory utilization and management to create what it calls a Memcached distribution, much in the same manner as Linux distributions are packaged. Joaquin Ruiz, executive vice president of products at Gear6, provided me with additional insight into why Memcached is popular with Web 2.0 sites and why it matters for cloud computing.

The problem, according to Ruiz, is dynamic data services. In a recent blog post, he pointed to the tight connection between dynamic content and Web 2.0; that is, one defines the other. In this Web 2.0 world, the LAMP (and to some extent Java and Ruby) stack "provided a low-cost, efficient development foundation for Web 2.0 but did not free us from the monolithic, vertically oriented, "scale-up" platforms. Memcached provided the heavy lifting in terms of horizontally scaling ("scale-out") on non-monolithic SMP server architectures from Intel and AMD."

In the Facebook example, the Memcached tier stores members' personalized dynamic content, such as status updates, wall posts, etc., so that they can be quickly accessed when queried. It's a similar set up for Twitter tweets or comments on photos on Flickr. While latency in a social application is mildly annoying, latency in a transactional application could mean lost revenue.

Dynamic data services will likely remain an important part of cloud services, which brings us back to the idea of a Memcached service on a cloud platform. Amazon's Jeff Barr noted, "powerful, high-level services like this allow application developers to spend more time focusing on the novel and value-added aspects of their application and less time on the underlying infrastructure."

Anything that developers and companies can take advantage of to serve data faster and more efficiently means they have time to do other things, including increasing their bottom line.

information technology is expected to play an important part in the global economic recovery, according to a new survey released Wednesday.

Some 72 percent of business and information technology executives say their "organizations place greater value on the IT function today than they did before the economic crisis" and that they "view IT as an important part of their economic recovery efforts," according to Accenture's Global Survey on IT Investments.

This is not an unfamiliar sentiment and is one we've heard from United States CIO Vivek Kundra as he's attempted to use IT to kick start a variety of programs on the federal level that will set the pace for innovative new uses of technology across the globe.

The results of the Accenture survey are similar to last week's Goldman Sachs cautiously optimistic survey results that suggested IT spending would trend upward in 2010 and normalize to pre-recession levels with the majority of countries represented planning to increase investment selectively next year.

... Read more

Most businesses seek competitive advantage through some kind of change. Whether they want to beat the competition to market with a new service or introduce new product categories, disruption is the norm.

The challenge in today's IT-centric world is that every one of those disruptions requires a software change, introducing the potential for downtime and lost revenue.

Change control and the associated risk mitigation is a big problem that every large organization faces. Last year, the London Stock Exchange crashed during a software change and was down for more than seven hours, costing traders millions, if not billions of dollars in lost business. This year we've had high profile outages at Salesforce.com, Twitter, and Amazon's EC2, among others, affecting tens of millions of people.

No company is immune to this type of risk and companies that want to stay on the leading edge need to embrace these changes in order to stay competitive.

Coverity, a software integrity firm perhaps best known for its SCAN project of open-source software sponsored by the Department of Homeland Security thinks it has the preventive medicine to help organizations avoid the inevitable errors, defects, and failures that software change can introduce.

The company's latest release, Coverity 5, promises to mitigate the business risk of software changes across an organization's entire software portfolio. It claims this is the first product that lets developers automatically map and identify how a single defect impacts multiple code bases, projects, and products. Through a unified defect management interface, it also can help organizations review, prioritize and triage their C/C++, Java and C# defects in a single work flow.

This approach lets an organization quickly answer five key questions of software change management:

1. How do I find defects introduced by changes?
2. How do I know the severity of new defects?
3. How do I know the impact to my code, my projects, my products?
4. How do I fix them fast?
5. How do I know I fixed them?

Today, market opportunities are changing faster than businesses can deliver. When your organization changes software, how quickly can answer the five questions above?

The Tennessee Valley Authority is the nation's largest public power provider serving approximately 9 million consumers in seven southeastern states. The organization also happens to be a big supporter of open-source projects, including Hadoop, a tool designed for deep analysis and transformation of very large data sets.

Earlier this year, the Tennessee Valley Authority (TVA) announced that it open sourced its data system used to collect data from smart grid devices called Phasor measurement units (PMUs). The data collection system is known in the industry as a Super Phasor Data Concentrator (SuperPDC), which can be used to determine the health of a power grid.

The open-source version of the SuperPDC is now called the "OpenPDC." I spoke to both Ritchie Carroll (RC), the project's creator, and Josh Patterson (JP), the person responsible for introducing Hadoop to the project, to discuss what the OpenPDC is and why TVA turned to Hadoop in building the system.

What sort of data volumes are you working with?
RC: Currently there is around 20 TB of archived data, we expect this to grow quickly as a result of the SmartGrid stimulus funding which includes the addition of 850 phasor measurement devices. This may well grow the archive to half a Petabyte within the next few years.

How is this data currently captured and managed? Is any data discarded?
JP: Data is collected directly from field devices at 30 times per second. This data is then time-aligned and processed in real-time--all data gets captured into a binary data file as time-series data for mass processing by Hadoop.

RC: No data is currently discarded, if we get to the point of needing to discard data because of cost--this will be a decision based on weighed importance of collected data. It is likely the data around major events will never be deleted because it will always be valuable for future student researchers. There is also value in being able to go back in time and look for newly discovered event signatures to see how long they might have been occurring.

... Read more

As much as Twitter is a powerful communication and social application, it's a relatively simple Web app. As part of a new contest sponsored by Engine Yard, Ruby on Rails developers are going to turn Twitter into their own application server.

The contest asks developers to program the "Worst App Server Technology Ever" (Waste) using Twitter as the message bus. While much of the contest is being done tongue-in-cheek, it's actually an interesting use case to see if a service like Twitter can take the place of a more traditional message bus like IBM MQ series or AMQP (Advanced Message Queuing Protocol).

Contest participants register up to five Twitter handles and code the function that each would perform in a program. When the contest challenge is issued on November 12, participants will have to use at least 10 of the pre-designated Twitter handles (other than their own) as endpoints to perform functions on data sets located at unique URLs. All messages will work through a series of automated public Twitter replies.

This is somewhere between an application server, a social game, the "telephone game" and service-oriented architecture (SOA) where Twitter plays the role of the enterprise service bus and the Twitter API is the broker between data sources. SOA relies on services exposing their functionality other applications and services can read to understand how to utilize those services. In this case, Twitter can be used as an application server in the cloud. (Take that buzzword bingo players.)

The funny thing is that as absurd and comical as this sounded when the Engine Yard guys told me about it, I've started to think about this as a way to possibly achieve a real technological breakthrough. And while I don't think that Twitter will be the "cloud bus," I do think that there is a lot to be learned from applying this type of constraint to a data flow process.

Engine Yard VP of marketing Michael Mullany told me that the contest shows how developers can leverage a relatively straightforward platform in innovative ways. But it's also another example of an interesting marketing effort to use Twitter as the vehicle for one's own benefit. Also, in true open source fashion, developers wind up building new applications based on code written by their peers.

Let's hope Twitter can handle the attention and developers are not greeted by the ever-lurking fail whale. You can check out the contest and learn more details at Engineyard.com

Influence in open-source development communities is earned through years of writing and sharing great code. Perhaps not surprisingly, then, influence in the business side of open source is also gained through sharing expertise, and not necessarily from making mountains of cash.

At least, that's the lesson I take away from MindTouch's inaugural survey of 50 open-source business executives. MindTouch, an open-source collaboration company, has spent the last few months surveying executives within the commercial open-source community, asking them to name the most influential people within the commercial open-source ecosystem.

The result is effectively an all-star list of open-source business executives. The top five are as follows:

1. Larry Augustin, CEO, SugarCRM
2. Matt Asay, vice president of business development, Alfresco (and fellow CNET blogger)
3. Mårten Mickos, entrepreneur-in-residence, Benchmark Capital, and former CEO, MySQL
4. Jim Whitehurst, CEO, Red Hat
5. Dries Buytaert, co-founder and CTO, Acquia

The full list is available here.

The common theme running through these top-five vote getters is how open they've been with their peers. Larry Augustin sits on several boards of open-source companies, but he also frequently speaks at industry events and has been involved in open source from its inception.

Matt Asay, my friend and fellow CNET blogger, sits on more than 10 open-source advisory boards, chairs the Open Source Business Conference, hosts an informal get-together every year (called Open Source Goat Rodeo--don't ask why), blogs at an unhealthy rate for CNET on open source, and has actively helped a range of aspiring open-source entrepreneurs understand the mechanics of running an open-source business.

Mårten Mickos made the world safe for the $1 billion open-source acquisition, but he has also traveled the globe speaking at open-source events and is very generous with his time, sharing know-how and best practices with other open-source executives.

Jim Whitehurst, breaking the typical Red Hat mold, has been active in industry events, has hosted a range of dinners and other small-scale, intimate events with open-source executives. He is amazingly accessible, given that he has a fast-growing open-source company to run. It's unfortunate that Whitehurst is the only Red Hat executive to make the list; Red Hat should follow his lead and be more permeable to its peers. Its influence would grow accordingly, just as Whitehurst's has.

Finally, there's Dries Buytaert, who blogs frequently on his project, Drupal, but also regularly attends and speaks at industry events. He has also been active behind the scenes, working with other open-source companies to share information on how to optimize community development.

Open-source code becomes valuable when you give it away. The same holds true for open-source business expertise. There are individuals who have made more money than these with open-source software, but in terms of influence, the more you share, the more influential you become.

What do you think? Who else should be on the list? Who influences you?

JRuby is a relatively new high-performance Java implementation of the Ruby language that is showing increasing popularity among Java developers looking for additional productive frameworks.

JRuby allows for the incremental adoption of the Ruby language by allowing easy integration with existing Java libraries. It also lets Ruby and Rails applications to run easily on existing Java application servers that have been selected as standards within an organization.

I've been somewhat dismissive of Ruby as a language but there are more and more examples of large websites running extremely well. In fact RubyConf (already sold out) and JRubyConf are both seeing significant interest from developers for the upcoming events in San Francisco.

In the Q&A below I discuss how Jruby powers shopping site Gilt.com with CTO and co-founder Michael Bryzek.

If you are not familiar, Gilt Groupe has an interesting business model, somewhere between eBay and Woot, offering invitation-only sales of high-end fashion and luxury brands for men, women, and children. The site deals with unique spikes in traffic when new items are released as well as when an item becomes extremely popular. Accordingly, the IT infrastructure needs to be able to scale and burst in order to meet customer demands.

... Read more

IBM and Canonical, the commercial entity behind Ubuntu Linux, on Tuesday are launching a combined cloud and Linux desktop package designed for Netbooks and low-end PCs.

For those of us still waiting for Linux to hit the desktop, this type of packaging may be exactly how the move from Windows starts to pick up steam.

The IBM Client for Smart Work was first launched in South Africa in September and was initially geared toward emerging markets. IBM found that there was strong interest in the U.S. and other markets that had aging PC infrastructure and little desire for continued Windows upgrades.

The U.S. version of the package contains a number of IBM products including word processing and spreadsheets via Lotus Symphony, e-mail via Lotus Notes or LotusLive iNotes, and collaboration tools from LotusLive.com. As with the previously launched initiative, the package runs on Ubuntu Linux.

Bob Sutor, IBM's vice president of Linux and open source, told me that the target is not a drop-in replacement scenario, but rather something for IT shops that don't want to be stuck in an endless cycle of upgrading desktop operating systems and applications.

This is an interesting development for multiple reasons:

• IBM and Canonical are teaming up to bring Linux to the desktop, offering what could be considered a next-generation thin-client that relies on cloud services but remains based on an actual operating system rather than just running in a Web browser.
• IBM is targeting Windows installations in the co-opetition model the company excels in--effectively insulating itself regardless of who wins the desktop.
• Canonical is building a channel to deliver solutions rather than depend on individuals and organizations to roll their own.

IBM has been making some interesting moves of late, launching a cloud-based e-mail and collaboration suite to rival Google Apps and now a direct attack on Microsoft's operating system footprint.

If you need further proof that open-source applications are ready for prime time, take today's news from open-source business intelligence company Jaspersoft, which announced that British Telecom is using its business intelligence suite to support more than 8 million voice mail subscribers.

BT and Unisys, a longtime Jaspersoft partner, say they chose Jaspersoft for its modular design, which reduces maintenance and cost and gives them customization abilities that improve capacity planning.

The deal with BT also represents how important a solid channel strategy is for open-source software companies.

Jaspersoft CEO Brian Gentile has in the past mentioned that the BI market is heavily influenced by a few technical aspects, including SOA/Web services (and overall componentized design), in-memory analytics, integrated search, and the use of rich media services to provide more compelling (Web-based) user experiences.

The other obvious factor in the shift to open-source BI (and open source in general) is the economics behind the applications and ongoing operations. And perhaps more important is the control--both on-premise and online. As consultant Carlo Daffara noted recently, "the critical aspect is being able to assess this control and weight if the lack of control is compensated by the features you get (which is reasonable) or what kind of risk you are accepting in exchange."

In conversation earlier today, Gentile further asserted, "open-source software is both augmenting and displacing aged, proprietary solutions across industries and at the largest companies. British Telecom is just one example of a company that has realized traditional, proprietary software is just too expensive and too complex. The most aggressive companies figured this out long ago. But now, with heightened economic pressures and the feature maturity of open source, the secret is out and the choice is clear."

There was a time when people would debate whether or not open-source software was reliable enough to support a small office. Those days are long gone. The down economy and maturity of open source are the perfect storm for major disruption.