What's the fastest-growing data source at large organizations? Video? Maybe at YouTube, but not at Citibank. The answer is log files. Yup, those mundane text messages produced by every conceivable technology device are growing like a proverbial weed.
Why the log file explosion? First off, every IT shop has more and more devices to manage each day, and more devices mean more logs. In the past, most IT managers paid limited attention to logs, but this too has changed. Driven by security monitoring, regulatory compliance reporting, and IT troubleshooting, large organizations now aggregate, analyze, store, and archive terabytes of log data every month. Indeed, the log management snowball is rolling down a very steep and very snowy hill.
It's certainly time that the IT industry recognizes that log management is no longer a tactical domain of IT administrators. Rather, think of log management as the foundation of a Network Information System (NIS). Analysis of log data (as well as other data sources like NetFlow and SNMP) is quickly becoming the difference between effective IT security/operations management and flying blind.
As log management gains strategic importance, look for the big guys to swoop in. EMC/RSA and IBM are already there, and HP is working with SenSage on its play. There are also several strong pure plays including ArcSight, LogLogic, and LogRhythm. All this said, where is everyone else? Log management seems like a natural for Cisco Systems, especially in concert with its whole Data Center 3.0 initiative. Symantec is also missing from the party thus far, as is CA, McAfee, Microsoft, and loads of others.
Logs seem trivial, and log management appears like a tactical task at the bottom of the IT stack. Maybe in the past this was true, but in today's world, information is power and logs are device-specific information. Aggregate these logs, add some intelligence, and provide tools for analysis, and you get a great temperature reading of what's going on across the IT system. Seems pretty important to me.
Just about every technology under the sun--from your cell phone to the biggest baddest core router or multi-processor server--tracks its activities by maintaining log data files. Most people rarely pay attention to this stuff, but it is a big deal when it comes to enterprise IT.
Want proof? According to ESG Research, 44 percent of large organizations (i.e., 1,000 employees or more) collect at least 1 terabyte of log file per month. Heck, 11 percent say that they capture more than 10 terabytes a month. That's a lot of logs from a lot of devices.
Just what do they do with this data? They analyze six ways from Sunday in order to monitor security events, regulatory compliance controls, and technology utilization. And when something goes wrong (insert Murphy's Law here)--a security attack, a network performance glitch, an application hiccup, whatever--you can be sure that a bunch of IT brainiacs are pouring through log data looking for clues.
Now here's the thing about log data, it's growing like a weed. ESG Research reveals that large organizations plan to capture lots more log data from more devices for more analysis over the next few years. Today 10 terabytes of monthly log data collection is an exception, but within two to three years it will be the rule.
Pretty soon, log file data capacity will be too much to handle for today's willy-nilly log management processes and technologies. My contention is that soon we will be talking about log management architecture and log management services the same way we discuss SOA and business intelligence today. In other words, IT will have a bunch of architectural services for collecting and presenting log data to a host of enterprise analysis applications through Web services interfaces. Think of a multi-layered architecture that separates the data from the applications and you'll get the picture.
I expect this architecture to evolve over the next few years, but the roots are already here. Companies like ArcSight, IBM, Log Logic, Network Intelligence and Symantec are already headed down this path. In the meantime, this log growth is certainly good news for "pipes and platters" guys like EMC, Cisco Systems and Seagate.
- prev
- 1
- next
