IBM tackles high-volume 'stream computing'

IBM launched this week System S, a software platform built following five years of research into the real-time analysis of large amounts of unstructured business or scientific data.

IBM calls the resulting technology "stream computing," because the software deals with streams of data.

Also this week, IBM opened the IBM European Stream Computing Center, headquartered in Dublin. The center will serve as a hub of research, customer support, and advanced testing for stream-computing applications.

System S is IBM's answer to the growing problem of data overload, the company said. In particular, it is a response to the growing amount of unstructured data--such as Web pages, e-mails, blogs, video and data captured from electronic sensors--that organizations are faced with processing.

The new IBM software is designed specifically to handle such information, as well as the structured data found in databases. It processes this data in real time, giving users the ability to make decisions based on that analysis right away, according to IBM.

"Traditional computing models retrospectively analyze stored data and cannot continuously process massive amounts of incoming data streams that affect critical decision-making," IBM said in a statement. "System S is designed to help clients become more 'real-world aware', seeing and responding to changes across complex systems."

The software is written using a programming language specifically developed for stream computing, called SPADE (stream processing application declarative engine). It is designed to run on a variety of hardware platforms, including clusters, multicore architectures and chips such as the Cell processor, IBM said.

The system can be used to analyze data such as stock prices, retail sales, and weather reports. IBM is aiming it at financial institutions, government and law enforcement agencies, and retailers, among other organizations.

System S is being used in a number of pilot projects that demonstrate the diverse types of applications IBM is targeting for the software.

Uppsala University and the Swedish Institute of Space Physics, for instance, are using a pilot system to analyze the way radio emissions from space affect energy transmission over power lines, communications via radio and TV signals, and airline and space travel, IBM said.

The Marine Institute of Ireland is using the system to monitor large volumes of underwater acoustic information, while TD Securities is using the software to develop a pilot of an automated options-trading system. A pilot at the University of Ontario Institute of Technology is using System S to monitor streams of biomedical data from critically ill premature babies.

The software is currently available in English directly from IBM, with prices ranging from $100,000 for a two-server installation up to several million dollars for a large cluster with hundreds of nodes, IBM said.

System S is scheduled to be released in multiple languages and through IBM business partners sometime in 2010.

Matthew Broersma of ZDNet UK reported from London.

[Commander_Spock]

This "stream(ing) computing technology" that IBM is talking about appears quite interesting; but, on the other hand - is is it that IBM is playing catch up - again.... having been there, done that..... in the past where "streaming media/computing" is concerned!!!

Also, it would be most interesting to know what the "competition" is doing in this area of computing?

[shusseina2]

Michael Stonebraker, the man behind Ingres and Postgres among other things, set up a project at MIT to address the issues around streaming data, called Aurora (www.cs.brown.edu/research/aurora/). In 2003 in he founded StreamBase Systems (www.streambase.com) which offers a streaming data platform. Seems IBM is late to the game.

[shusseina2]

Michael Stonebraker, the man behind Ingres and Postgres among other things, set up a project at MIT to address the issues around streaming data, called Aurora (www.cs.brown.edu/research/aurora/). In 2003 in he founded StreamBase Systems (www.streambase.com) which offers a streaming data platform. Seems IBM is late to the game.

[wmonk2]

There's a company in Oregon (Douglas Telecom, http://www.douglastelecom.com) that developed and has been using a similar system since 2004.