In today's security analytics, every bit of data matters

There is a change brewing in information security and information management. In the early days, this discipline really came down to event detection. Security information management systems scanned a bunch of data looking for needle-in-the-haystack events that indicated trouble. All other data was considered "noise" and thrown away.

With the onset of regulatory compliance a few years ago, this model went through an initial change. The "noisy" data was now necessary information to demonstrate security controls for compliance audits. Still, event data and compliance data remained separate entities.

Now things are changing yet again. In today's dangerous security landscape, no data is considered "noise" anymore. Rather, security analysts now want access to terabytes of historical data for analysis. Furthermore, this underlying data has become more complex. Beyond just log files, security analytics now encompasses other data types like network flows, directories, physical access, and video surveillance. If there is reason to believe that Joe the IT administrator has been covertly accessing quarterly financial data, a subsequent security investigation will encompass everything and anything including when Joe was in the building, when he logged onto the network, which systems he accessed, and what he did.

This type of investigation requirement changes the security technology model. It means collecting, normalizing, and storing a ton of data. It means sophisticated algorithms and processor-intensive query engines. It means the integration of physical and information security, including video surveillance. Sound like law enforcement or the NSA? Perhaps, but large organizations are already headed down this path.

From an industry perspective, security information management systems will need to re-architected for this type of scale and power. Vendors like ArcSight, eIQ, Nitro Security, RSA, and SenSage have already anticipated this change--as have log management vendors like LogLogic and LogRhythm. This may also introduce the heavyweight security vendors like Comverse, Narus, and NICE into the enterprise space. In either case, I anticipate lots of activity in 2009 regardless of the current economic woes.

[concerned_buyer]

We've recently completed an extensive evaluation of security information management vendors and the one we found to be the most complete is Intellitactics which is not mentioned in this article. I mention it because their products were the only product s we looked at consistently demonstrated the ability to manage hundreds of millions of logs and make them instantly accessible. Yes all the data may eventually be important - but when needing to act quickly all the data is too much. We wanted all the data and the analytics to get what was important in the event of an incident, breach or attack. Their analytics were unmatched. Some of the products mentioned by Mr. Oltsik do not have the data management capabilities to perform the tasks he so aptly describes as essential. We offer this information to balance Mr. Oltsik's incomplete survey of available products.

[chopskie]

Jon, great post! I read that Verizon Business study of 500 breaches over 4 years and their assertion was that in over 80% of the cases, evidence of the breach was in the log data.

Happy new year!

Ed Chopskie
VP Marketing
SenSage