Engineering world-class support for open source

During my trip to Raleigh, I was fortunate to catch up with Iain Gray, vice president of Global Support Services at Red Hat. With my Alfresco hat on, I wanted to find out how Red Hat manages support, and with my CNET hat on, I wanted to share that insight.

Specifically, I wanted to get Iain's perspective on how open-source support differs from support in the proprietary software world. (You can tune in to a Red Hat video of Iain talking about this topic, too.)

Iain brings to Red Hat over a decade of support and services experience honed at Sun and SCO Group (back when it was a Unix company, not a law firm). As such, the obvious question was...

What's the difference?

In a proprietary world, things like a knowledge base are exclusive to internal support engineers. You try to keep information from customers--you become the gatekeeper to critical information. In open source, however, it's completely different. We want to get as much knowledge in the hands of the customer so that they're enabled to be successful. We want as much information in the customers' hands so that their time to resolution is shortened.

But do you ever worry that a customer that doesn't need Red Hat to solve its support issues may not need to pay Red Hat for support?

I initially worried that we could make the customer so smart that we'd be out of business, but I've come to realize that this is the wrong way to think about it. If you can't supply smart customers, you don't deserve to be in business. We want to facilitate customers getting answers to their questions, whether that's online or through a phone call.

What is the structure of Red Hat's support organization?

You ask at an interesting question. We're actually in the middle of engineering "Red Hat Support 2.0." Red Hat's support had been fairly traditional in its structure, meaning that we offer:

• Production support--Level 1, Level 2, Level 3. Everyone that purchases a Red Hat subscription gets this.

• Technical account management--If you want an assigned contact who knows your environment, etc., customers can pay for this higher level of support.

• Developer support. This is a very small but growing component of our support services. Traditionally this was offered to provide API-literate engineers to work with a customer at a deep level, though it generally didn't involve code modification. Today, however, JBoss has changed this. It's now much harder to divide the line between production support and developer support.

So, not only are we working out how to provide a greater degree of developer support, but we're also figuring out how to support multiple products. Up until now, we've essentially been a single product company: Red Hat Enterprise Linux. But now with JBoss, MetaMatrix, etc., we're in a new world with multiple products to support. How do we better support multiple technologies and products? Do we hire engineers who can cover a broad spectrum or assign topic specialists?

One of the things we want to keep core is a high level of technical expertise. In the open-source world, people have generally done their homework before they pick up the phone. Hence, when they call they want to be talking with highly trained technical folks. Our struggle is how to provide this deep expertise across our products.

To do this we're exploding our Level 1, 2, and 3 layers of support and instead thinking in terms of "flocks." Units of technical expertise, if you will. For example, a "flock" around Hibernate will involve relatively junior or senior people. Some from engineering. Some from an SE team. This flock-based approach is more about pooling resources/expertise, though we continue to have a single point of responsibility for the customer's happiness. We think this approach will benefit customers by helping us to bring a breadth of expertise to bear on their pressing issues.

How do you hire well into an organization like this? Engineers want to code, but you need them to provide support, too. How do you overcome Support's lack of sexiness?

We accept that in many cases the best engineers want to be in Engineering, not Support. We therefore have a career progression from Level 1 support to Level 3 support. The more you move up, the more engineering you get to do. The Level 3 engineers write the supportability tools to help improve our support operations and improve the customer experience. Once they've progressed to this level, they are able to join the Engineering team.

That said, it's important to note that I report to Paul Cormier, who heads up Red Hat Engineering. Support, in other words, is considered part of the Engineering team, which makes it easy to facilitate this career progression.

Finally, how do you measure the value and performance of your support?

Through largely traditional means. Short customer satisfaction surveys after each incident's resolution. We also do quarterly relationship-based customer satisfaction surveys. Such measures are trailing indicators of customer satisfaction, of course. Operationally we're looking at end-to-end cycle times, among other things.

I really like how Red Hat offers a career progression for support engineers. Not only is it good for these engineers, but it's also good for the code they end up writing, as months or years resolving real customer issues should make them better developers.