The invisible log data explosion

Just about every technology under the sun--from your cell phone to the biggest baddest core router or multi-processor server--tracks its activities by maintaining log data files. Most people rarely pay attention to this stuff, but it is a big deal when it comes to enterprise IT.

Want proof? According to ESG Research, 44 percent of large organizations (i.e., 1,000 employees or more) collect at least 1 terabyte of log file per month. Heck, 11 percent say that they capture more than 10 terabytes a month. That's a lot of logs from a lot of devices.

Just what do they do with this data? They analyze six ways from Sunday in order to monitor security events, regulatory compliance controls, and technology utilization. And when something goes wrong (insert Murphy's Law here)--a security attack, a network performance glitch, an application hiccup, whatever--you can be sure that a bunch of IT brainiacs are pouring through log data looking for clues.

Now here's the thing about log data, it's growing like a weed. ESG Research reveals that large organizations plan to capture lots more log data from more devices for more analysis over the next few years. Today 10 terabytes of monthly log data collection is an exception, but within two to three years it will be the rule.

Pretty soon, log file data capacity will be too much to handle for today's willy-nilly log management processes and technologies. My contention is that soon we will be talking about log management architecture and log management services the same way we discuss SOA and business intelligence today. In other words, IT will have a bunch of architectural services for collecting and presenting log data to a host of enterprise analysis applications through Web services interfaces. Think of a multi-layered architecture that separates the data from the applications and you'll get the picture.

I expect this architecture to evolve over the next few years, but the roots are already here. Companies like ArcSight, IBM, Log Logic, Network Intelligence and Symantec are already headed down this path. In the meantime, this log growth is certainly good news for "pipes and platters" guys like EMC, Cisco Systems and Seagate.