A few months ago Sam Lawrence of Jive Software spent some time grading Forrester and Gartner as analysts. His verdict? Neither does a fantastic job, but Forrester is much better at servicing a small but growing vendor like Jive.
My own experience with analysts is mixed. Analysts tend to be great at predicting the past, but far less adept at predicting the future, which is actually what customers expect from them. If you look at such things as Gartner's Magic Quadrant, it is great at showing where the industry was, rather than where it's going.
The problem is that analysts like Gartner get their information from the vendors that subsidize their research, as well as from CIOs. Neither is a good indicator of where the market is going.
As Billy Marshall classically wrote, the CIO tends to be the "last to know" about new IT initiatives. As for the vendors, the only ones with enough cash to subsidize research are the same ones that have a vested interest in protecting existing cash cows. In other words, the past.
Analysts, then, are a lagging indicator of success. They tell an enterprise buyer from whom she should have purchased software and hardware a few years ago, not where she should invest IT dollars tomorrow. As an example, despite the massive influx of open-source vendors in the enterprise, Gartner persists in believing that open source is years away from making a dent in the enterprise, and you'll rarely find an open-source vendor in a Gartner Magic Quadrant. Here's a recent Magic Quadrant for Business Intelligence. No open source need apply.
Putting open source aside, some analyst research is so egregiously off that it's almost comical.… Read more