The Event-Driven Architecture

Right here. Right now

12 minute read

Tags: , ,

Michelangelo famously said that he didn’t just take a piece of stone and sculpt it into the shape he wanted, but rather he believed that every hunk of rock already has a sculpture inside and the job of the sculptor is simply to remove all the pieces that aren’t it. He added that the way to distinguish between that which should go and that which should stay is to “obey intellect” and understand that even the greatest of artists cannot conceive of anything more beautiful than that which already exists within the stone. The quote is from one of his best known sonnets “Non ha l’ottimo artista alcun concetto”, written sometime around 1540.

The notion that perfection isn’t something designed but uncovered, provides a useful metaphor which applies to many things beyond art and sculpture - including how to sell architecture, which I’ll talk about in an upcoming article, and why thoughts of Complex Events and the Event-Driven Architecture (EDA) should be top of your New Year’s Resolutions.

The Accidental Tourist

Shortly after it came out in 2002, I accidentally ordered a copy of David Luckham’s book The Power of Events. I say accidentally because I meant to order a copy of The Power of Now by Tibco CEO Vivek Ranadive but it was very early in the morning and I was only half concentrating. Come to think of it, I still don’t have a copy of The Power of Now.

Anyway, slightly dismayed, I opened the Power of Events, Luckham’s introduction to the field of Complex Event Processing (CEP) in distributed systems, and experienced a mini revelation. One that has helped contextualise other architectural styles like SOA and also get across the real value of enterprise architecture to business leaders - the very people to whom EA is most important, but who rarely believe this to be the case.

Let’s start with the soup.

The Event Soup: Simple Events

Every second of every day you are talking about IT strategy, designing blueprints for the future, and selling your architectural vision, real business events are occurring. In the hour or so it takes you to make a pitch to management for investment in a new integration platform, orders are coming in, customers are leaving, money is being spent. All these exist right now in your present architecture, however poorly designed and non-strategic you may think it is.

Our beloved employer, Gristle and Flint, has a web site that sells different kinds of widgets. Twenty four hours a day customers place orders for them and somewhere out there, in a row in a database table, a message on a bus, a line in a CSV file or in a plain old email, we capture some data to represent this.

These are Simple Events and are represented by a collection of attributes and one or more time-stamps to record for posterity when the event occurred and/or was captured. A simple order-received event, in JSON, might be something as plain as:

	customerid: 71689,
	item: "widget1",
	quantity: 3,
	value: 45.00,
	channel: "web",
	time-ordered: "7/12/2008 11:34:52 AM",
	time-captured: "7/12/2008 11:35:01 AM"

The Event Soup: Complex Events

When we are alerted to multiple simple events occurring within some sort of relationship, we can deduce complex events - essentially new event information.

The example used on Wikipedia contains bells ringing, a man in a suit, a woman in a big white dress and rice being thrown - you can surmise that there’s a wedding taking place without being explicitly told so.

In fact if you further examined the attributes of these events and discovered the bells were of type church, the suit was of type tuxedo, you could additionally infer that this is a US wedding, whereas church + morning suit would imply UK wedding, stringed instruments + red and gold sari implies Indian wedding, white dress + white dashiki implies West African wedding, and so on.

The point is that businesses already have ready access to simple events, but if we combine them the results can be enormously powerful in two ways:

  • Firstly, we can track events that we know happen but which we cannot measure directly, so an order-received event plus a matching order-dispatch event could create an order-fulfilled event with attributes that tell us something about the whole order process (automatically separating out order-received events that were not completed, or order-dispatch events that don’t have matching order-received events for product replacements or perhaps a sign of fraudulent activity).

  • And, using basic inference rules, we can generate other events to suggest what might be happening.

The Event Soup: Composite Events

Composite events are a special subset of complex events. Once you have identified your primitive (simple) events you can combine them different ways to make Composite Events and use these to make great leaps in understanding what customers are up to. Let’s keep this straightforward and take a widget-purchase event, an account-closure event and a customer-complaint event.

There are many types of composite event, a few interesting examples for the Gristle and Flint management team might be:

  • The Sequence: Event A (customer bought a widget) completed before Event B (customer closed their account) OR Event C (customer complained)
  • The Conjunction: Event A (customer bought a widget of one type) happened in any relation to Event B (customer bought a widget of another type)
  • The Iteration: Event A (customer bought a widget) happened more than once
  • The Negation: Event A (customer bought a widget) did not happen between time T1 and time T2

Certain types of Sequence would alert us to the fact that there’s something not quite right with one of our widgets; Conjunctions identify both loyal customers and popular widgets combinations that might be ripe for promotion; Iterations tell us something about unit quantities and possibilities for multi-buy discounts; Negations can tell us about failing promotions.

Understanding Composite Events doesn’t just help us run our businesses better, we can feed back what we learn directly to customers. If you’ve ever looked at a product page on Amazon and seen the “customers who bought this also looked at..” or the “buy this and get this too..” messages, you have witnessed the results of composite event processing.

Composite events are a collection of simple events that map to some kind of pattern or regular expression.

The Event Spaghetti

For the purposes of an introduction to event-based thinking, knowing that there are simple events (stuff that happened) and complex events (stuff we can generate in various ways from stuff that happened) is enough to be going on with. I added the composite event examples because they’re a neat way to see direct business value and also easy to program inside a regular project, without needing to make a big deal about adopting EDA to a sceptical business sponsor.

If you want to read more on the taxonomy of events Opher Etzion has a nice Venn diagram in his post “On relationships among: derived event, composite event, complex event and situation” showing how complex and composite events relate to the class of derived events and topics like situation theory.

The concepts that drive this get fairly esoteric quite quickly. Once you start to ponder how events inform you of the what-is, and can be used to predict all that what-might-be malarkey, you’re into the philosophy of possible world semantics and fictionality and narrative theory before you know it.

Thankfully, the architecture behind it is relatively simple.

The Event-Driven Architecture

Before I start, let me quickly voice one of my top-five annoyances with the way corporate Enterprise Architecture has been perceived over the last ten years: the fallacy of the *BA/*DA/*OA. I’ll cover it in a later article but, please accept that I only refer to a something-based, something-driven or a something-oriented architecture because that’s the de facto convention. No enterprise architecture is comprised of just one something. EDA, SBA, SOA, WOA, ROA are just styles, not one-size-fits-all, next-big-thing, this-is-better-than-that replacements for the previous fad. EDA is not SOA 2.0. Like most styles they can compliment each other quite happily. Thank you for listening.

It’s easy for the CEO to get his hands on reports that summarise simple events, each month he sees a summary of widgets sold by type, by day and by sales channel. He sees how much they cost to make, how much money they made and the staff overheads in running the sales channels. And so do all his competitors.

What he doesn’t get is any insight into the complex events that surround these basic measures of performance. Even as he is reading about them, more events, good and bad ones, are occurring, and all the tools he has at his disposal to help understand just what the hell is going on in his market are retrospective.

He is, to use a narrative device, steering a very long oil tanker, from a wheelhouse right at the stern, hoping he doesn’t hit a sandbank. What he needs is:

  • To know what’s happening much closer to when it’s actually happening (a wheelhouse closer to the bow of the oil tanker). This implies real-time events and is technically straightforward and well-understood these days. Or at least talk about real-time business intelligence dashboards is fairly common, even if there’s not much evidence of them being used.

  • A way to add contextual detail to the basic information he can already see (side-scan sonar). It’s in this area where much of the value of the EDA is derived.

So let’s explore the possible value-add we might get from adopting an EDA style in those aspects of our design where it would be most useful. The very aspects in fact where, as Michelangelo would appreciate, there are complex events to be uncovered rather than designed-in.

Simple Events tell us something about how the business is operating and Composite Events start to tell us why it’s operating the way it is, but this is still an historically deterministic view of the world. The sooner we access this view (i.e. the more real-time it is) the quicker we can respond. To take this to the next level though we need to be able to move from responding to predicting.

Predictive models are nothing new, although a novel concept in a business that sells widgets on the web, they have been used for decades in the investment banking and trading sectors. When you have an amount of capital invested in a range of items with values that can change rapidly (currencies, futures contracts, shares, etc) having even half a clue as to what might happen in the future gives you a major advantage over anyone who has no clue. You can never have a full clue because the future (at least when dealing with market prices and customer behaviour) is stochastic, i.e. it is based upon some things that are predictable (if competitor prices drop for an identical item, your sales will fall) and some things that are random (fashion, the weather, events in the news).

The good news is that there are plenty of methods for dealing with stochastic processes that increase your level of having a clue, like simulated annealing, genetic algorithms and others I have only the barest understanding of. A good book on the subject, that relates this to customer web experiences, is Programming Collective Intelligence by O’Reilly.

However abstruse the mathematical theory that goes into predictive models, the resulting formulae are easily computable once you have your event data collected (usually in some series of matrices) and because we’re dealing with probabilities it’s also not hard to measure that against what did happen and tune the models over time. The architecture to support this moves us from basic EDA to the related discipline of Complex Event Processing.

The great thing about EDA and CEP is your basic unit of currency is still just a collection of attributes with one or more time stamps. At the data level there’s no real difference between a simple event and a composite event, and events that have been inferred are the same again but with, obviously, some level of associated probability or confidence.

Selling the EDA

I came across the EDA in 2002, and thought that within a couple of years it would be the biggest bandwagon in corporate IT, particularly as the EDA/CEP concepts in the book were originally kicked around by David Luckham and Brian Frasca in a paper (pdf) as far back as 1998.

That mantel seems to have been held by SOA, which surprises me because whatever the intrinsic benefits of service-orientation it’s not as directly relevant to business people as EDA - a perfect SOA would make new functionality ridiculously easy to deliver, primarily due to service stability and reuse, and though this is what the business want to hear, it still leaves them with the sticky problem of running the business.

An EDA sales pitch would go right to the heart of that. It’s almost impossible to give an example of an EDA that’s sounds like technology for its own sake. In 2005, with a couple of colleagues, I created a commercial pitch for a start-up in this area (details of which are excluded from this article, but don’t be shy if you’re an investor..) and I still get shivers when I read it.

If there were such a thing as the Architecture Manifesto (now there’s an idea.. except someone already thought of making one), akin to the Agile Manifesto, it should probably say something like:

Events and Inference over Services and Loose Coupling

That is, while there is value in the items on the right, we value the items on the left more.