Scaling Twitter redux--the ESB should be your best friend

As we Twitt-iots sit around bemoaning the fact that we can't send each other useless junk on a flaky service, I thought I would take this chance to address the notion that this message-scaling problem is new.

It's not. It's very common, and it can be solved.

Scaling a messaging platform is why IBM sells a boatload of MQ series, why the AMQP protocol was developed, and why JMS is nearly ubiquitous. Pretty much every large enterprise has similar scale issues related to messaging, especially in financial services. But they don't have downtime, and if they did as frequently as Twitter, the people behind them would all get fired.

This is a topic I actually know something about. (Disclosure: my company develops an open-source enterprise service bus, or ESB, called Mule.) All of our use cases involve some kind of complicated messaging architecture, whether it be Web-service based, publish and subscribe, one to many, direct connection, etc. And most deal with data transformations and a vast array of protocols.

In my view, what Twitter needs is to adopt a bus-type of architecture that separates the transport from the application and uses a middleman to process the transactions. This is a very common enterprise scenario that needs to be applied. This is what an ESB does.

One example in the Web 2.0 world is OpSource, which is delivering billions of transactions a day as an SaaS provider using the ESB at the crux of the transactions. This is much larger than Twitter and has actual revenue impact. Maybe OpSource can host Twitter?

[Perfeng]

Sorry, but you really need to not make things up if you want to come across as knowing what you're talking about. OpSource is NOT handling billions of transactions a day using Mule. Yes, their bus uses Mule, and yes, they serve lots of transactions a day, but not every request over the OpSource network flows through the bus. In fact, a very very tiny percentage of them do, and they're not serving billions a day at any rate.

This is not to say that OpSource couldn't help Twitter, I'm sure they could. But it would not involve their ESB, and their ESB is not magic.

Also, JMS is not magic. As a performance engineer, I'm as big a fan of reading about Twitter's problems as the next guy, but the scaling problem they have would be hard to solve on ANY platform. They're going to have to relax some of their requirements, and move to a more asynchronous model to scale. JMS could help them do that, but they could also do it without JMS.

Mule also isn't magic. Unless you've load-tested Mule with millions of messages an hour, involving tens of thousands of queues, some small percentage of which should have thousands of subscribers, you haven't even done an order of magnitude simulation of the Twitter problem. I'm pretty sure Mule would have issues serving Twitter too, and it would take less than a day to implement the basic model and find out, provided you have the machines on hand to handle this load, and sufficient load generation capability. But I'm guessing you don't have either, so maybe OpSource can help you with *your* problem.

[shmooth2]

@perfeng: dude - take a chill pill. seriously. take a week off. just do something to take the edge off. i'm sorry you got canned from your last position, but it's not cnet's fault. you should have used Mule.

[syerramreddy]

I am not sure if ESB solution will prevent every website outage issues.. I hope it is that simple, I agree with the fundamental view that ESB/Messaging can relieve some load issues, but a blanket recommendation or view that ESB would have prevented the outage is kind of parochial and lacks intellectual assessment..