The ABC of the ESB

We’re a smart lot in technology - give us a complex business problem and a short amount of time and we can do wondrous things with some cryptic text and a compiler. We constantly develop new ways to structure intricate business systems and information. We can scale up, out, persist data as rows in a table or objects in a tree; we can build tiered architectures that limit rippling change and we can, if we put our mind to it, innovate to such an extent that small companies can come from nowhere to rival the biggest behemoths on the planet.

When the future’s architected
By a carnival of idiots on show
You’d better lie low*

But we are truly awful at recognising our mistakes, understanding them, responding appropriately and then remembering how not to make them again and again. What happens is we latch on to a tool or an approach as the next big thing, use it inappropriately, then cast it aside as not up to the job, just as the next big thing comes along. By the time we run out of new next big things, just enough time has passed for everyone to have forgotten the first one and we start around the cycle again.

Take the latest debates on ESB vs. SOA. Some people think they are the same, some people think that they are opposing forces, and others that they are unrelated concepts that can quite happily sit side by side. The answer to questions like this lie in the history of their development, which in turns gives you a sense of their respective ideologies.

Really there should be no debate. Except we can’t collectively seem to remember beyond last week, so there is. If you’re in the perplexed majority on this whole ESB SOA issue, this is for you.

In the late nineties companies like Tibco developed message bus technologies. A message bus works just like a highly efficient postal service (or email on steroids). You construct a message and hand it to the bus and, whoosh, it’s gone. Recipients (for there can be more than one) of the message then receive it. The routing of the message is handled by the bus software itself so there’s no need to worry about whether the sender and receivers are on the same machine or not. What’s great about messaging is that it’s very fast and loosely coupled, as long as you don’t tinker with the message format that much, your senders are receivers can adapt and roll with business demand.

About the same time, as operational architecture complexity increased, IT looked for ways to address the inherent limitations in just connecting anything to anything, as needed (the much maligned point-to-point approach) and came up with EAI - Enterprise Application Integration. EAI uses a message bus as its main enabler. The message bus sits in the middle of the enterprise architecture diagram and so it become known as middleware.

Middleware sounded cool and took off quickly. But of course you don’t want to just start turning your old point-to-point connections into messages (well .. you could, and I’ve certainly seen a few places do this to their cost, but if point-to-point didn’t work because of complexity, then point-to-message-bus-to-point surely can’t be less complex). What you want to do is define a message structure that’s both reusable and based upon some company-wide agreed data standards. That way the myriad point-to-point connections should boil down to many fewer message-based connections. And to make sure each application can produce and consume the required message format we’ll use adaptors to talk to the bus. So now we have point-to-adaptor-to-message-bus-to-adaptor-to-point connections, but that’s OK because with standardisation we can cut the total number of connections down and with “adaptor factory” development teams we can make the production of adaptors fairly slick. Oh and of course we’ll need some architects to design those enterprise data models, message repositories, flows, reuse, governance, etc etc.

So EAI became quite unwieldy quite quickly. As it turned out, each application needed a different adaptor to map it’s own data model to the enterprise model and it needed a different adaptor for different scenarios and message types so we quickly had to manage adaptor proliferation in addition to the other complexities. I saw one project have its timeline trebled because the architects insisted on all its integration being delivered by a bus. Not surprisingly, this gave the business a pretty dim view of architects.

And request-reply interfaces (those where the caller needs something very important in the return payload from the callee) don’t work so well over a message bus because you’ve got a two-way conversation effectively made up of two one-way conversations. A bit like asking for your latest bank balance through the mail when the telephone is a better option. What happens if your letter got lost, or arrived on a different day than your loan application? If you resent it what happens if the bank gets two letters and sends two responses? Which would you believe? And performance wasn’t that hot either. With all that transforming from databases to objects to messages and back again a lot of work was being done to no business benefit. Oh dear, oh dear.

It wasn’t that messaging was bad. It’s not. It’s that IT was adopting a one-size-fits-all approach to integration. Clearly there are lots and lots of ways we might want to make system A talk to system B (and C and D..) and messaging might be one of them. What we needed was a richer tool box, not a bigger hammer. But we dropped the hammer we had and went off in search of a bigger one.

Enter SOA. Service Oriented Architecture was not a new idea even when it was new. The idea, that rather than use adaptors to integrate disparate systems, you create a business service interface backing off to one or more existing systems, goes back to the nineteen-sixties at least. It’s rooted in the idea that if you model interface functionality on aspects of the business that don’t change much - you can deploy something that also won’t change much. So, rather than create a service request message that returns a service reply message you create a service that can be called directly. Well, not so much directly, because that’s not strictly SOA, the calling system requests access to the service, and discovers its existence in a registry, which it then uses to access it.

So the SOA bandwagon quickly gathered pace. With services being business-aligned and at the appropriate level of granularity to promote stability, it looked like just the answer to the EAI nightmare. Expensive architects are still required but now the business is involved so they can’t climb too far up their ivory tower. There are no adaptors, although of course the service requires its caller to comply with its interface contract (another feature of SOA). You can even, in theory, create composite applications that hand-pick the services they need and cobble them together through a bit of workflow to provide new business functionality with minimal effort.

So imagine you’re a marketing guy in a message bus company. You have a great product, but the whims of fashion mean that fewer and fewer people want to buy it. Customers either deployed your software really badly and blamed the tool, or they went to a conference and came back with the SOA T-shirt. Naturally, it’s time to do what all good techno-marketing people do, you get a presentation slot at the next conference, and you re-issue and re-brand your software. Except you still have a bus, and people want services. Hmmm. Bus.. Services… across the Enterprise.. wait a minute.. Enterprise Service Bus. That’s it. The ESB is born. And to make sure that nobody twigs it’s the same old stuff under the covers you create some adaptor design and management software in the bus itself and allow functionality to be exposed as services. Bingo. You’re SOA compliant.

There’s nothing wrong with an ESB, it’s an improvement on EAI for sure, but then there was nothing wrong with messaging in the first place. It’s a horses for courses thing. What most enterprise companies shy away from is the real truth - that they need to be managing integration as a competency and a capability just like they do (or should) applications. It’s complex and it’s hard and just like you can’t run all your applications in one app server, you can’t do all your integration in one style. Well, you can, but you probably won’t like the level of investment you need to put in to keep it going.

So the answer to the earlier question is that if you have EAI and it doesn’t work, then an ESB might be a step in the right direction. But it isn’t SOA. SOA isn’t about integration in the way an ESB is. It’s about creating business interfaces that relate, somewhat indirectly, to business processes. In an SOA you “integrate” by orchestrating processes. In a pure SOA, if such a thing were possible, everything would be achieved simply by coordinating existing services. There would be no bus.

But at least consider the heretical possibility that you might not even need an SOA in certain areas. A shocking statement perhaps in these SOA-centric days, but I’ve seen plenty of business units where “services” are so dynamic that there’s just not the case for it. You do need the choice though. If you go all out to implement an ESB you may not be able to do SOA at all. The important thing is that all of these approaches are wrong in isolation. What you need is a team of people focusing on the best, costed, ways to integrate applications, some messaging here, some services here and, because this is the real world, some good old point-to-point there. As someone with an architecture background of some years I get very twitchy at the mention of tightly-coupled point-to-point, and perhaps there’s an article in this, but it is quick, easy and cheap (at least initially) and if the change cycle is long and small the case for it is undeniable.

By way of prediction (and this isn’t much of a gamble at all) there are a number of next-generation middleware products reaching maturity now that tend to be based on grid technology. They are replacing what the message bus did with queues, with a shared-memory infrastructure, the currency of which is objects not messages. There’s a lot less translation overhead and yet it has the same, or better, scale-out capabilities. This type of middleware, where the new “bus” also happens to be a very sophisticated application server, is already de rigueur in leading investment banks and is making its way into more pedestrian enterprises at a pace.

Within a year to eighteen months it will latch onto it’s super cool moniker (EDA - Event-Driven Architecture, XTP - eXtreme Transaction Processing and CEP - Complex Event Processing are related and stand ready to be adopted and misused) and then we’ll mysteriously start to hear why SOA was a duff idea from the start and super-cool-oriented architecture (SCOA. You heard it hear first) is the real thing.

Yes. We are a smart lot in technology. But we never seem to learn to remember along the way.

* From the Coldplay track Violet Hill. How clever I thought I was in quoting this. Until I discovered around 500 blog articles also using it to make similar points.