The Secret Sauce

You've gotta tell them. Soylent Green is people. We've gotta stop them somehow.

By Julian Browne on May 29, 2011. Filed Under architecture, delivery, development, governance

Last time, I was talking about what I consider to be the general lack of a crisis in software development. And it got me thinking - if there is no crisis in software development, no inherent flaws in our tools or our methods, then there must somehow be a way to convey the appropriate use of these tools and methods in order that everybody could get it right every time.

Ah ha, I thought. A book. A book entitled "The Secret Sauce". A book that will make me one .. million .. dollars. I can buy an island, with my own mini-me, surrounded by frickin' lasers.

But of course there is no book. There's no book because the truth is if anybody wrote it, it would go something like this:

Hire a few smart people. Let them hire some other people they'd like to work with. You don't want a team full of rock stars, but you do want people with a mixed bag of ideas and experience. You want them to put the case for different ways of doing things and then go with the flow when one gets decided on. Most of all they need to be keen to learn something every day. You don't need to test for this - it's kind of obvious. In fact, if you start with the right people, the rest will take care of itself, because as the saying goes "A's hire A's and B's hire C's". So the main objective initially is simply to avoid the B's and the C's.

Tools and methods?

Oh shit. You want me to tell you about those too? Honestly, all I can say is that the right answer to pretty much every question you could ever ask about whether to use a tool or a method, or how to best use one you've chosen, is the same: "it depends".

That's the problem with a book: I can't ask you questions.

But here's the thing. That team you hired? They can. Talk to them and, trust me, everything else will be gravy.

Or the The Secret Sauce. Whatever.

You think Prentice Hall would publish that? Have you seen how big their books are? The O'Reilly In-A-Nutshell version would be a business card.

So there's no real secret sauce. Or rather there is a secret sauce, but it's pretty much just the people you hire. And you can't put that in a book because you'd have to first ask the reader whether they were an A or a B. And if they were a B you'd have to give them a refund or say "hire people way smarter that you, yah big galoot".

Unfortunately you can't deconstruct the secret sauce either because, as Po's Dad in Kung Fu Panda made clear "there is no secret ingredient". Each team will come up with its own recipe. Some of those ingredients will feel like they are special, and maybe a little secret, but that's a product of the people choosing it, not a property of the ingredient.

Is anyone else getting hungry?

I work in architecture. There are plenty of times I wish I didn't. But I go on doing it because I believe, despite the very many flaws in its real-world execution, it's still one of the best vehicles most companies have to do good long-term things with their software.

Enterprise software delivered well means building just enough to cover what you absolutely need, which keeps complexity down to a minimum. Good designs continue this theme. They provide flexibility, partly because there's less stuff to change, but mostly because the moving parts are isolated in ways that prevent change creating ripples through the system. Architecture folk (rather than say developers on a project) have the golden opportunity to conceive of ways to make this real because, like the business itself, they exist above and beyond any one technical initiative subject to the naturally limiting constraints of budget and time. To understand the business potential that good architecture can deliver one only needs to look at organisations that do it well - modern internet businesses. It is true that they had a big advantage in that they had no legacy to deal with, and generally they are small in comparison to corporate enterprises, but Amazon isn't small and appears to deal with more than ten years of legacy development without hindering business growth.

Reduced complexity plus flexibility is competitive advantage, i.e. if your competitors don't have an architecture with these features then you have something over them. Having a function whose role it is to get architecture right should be a no-brainer and sure enough most companies, above a few hundred employees, seem to have some sort of enterprise architecture practice.

What's kind of scary is that the majority of these companies also have a pretty crappy architecture.

So I suppose it's no surprise that Enterprise Architecture has seen more than average levels of criticism over the years. In 2009 even the British Computer Society asked "Whither Enterprise Architecture?" and a Linked In Discussion - "Is Enterprise Architecture Dying?" couldn't find an Enterprise Architect to disagree with the idea that EA is fundamentally shuffling off this mortal coil.

The criticism is unquestionably valid - if a corporation has both an architecture team and a crappy architecture it has an untenable situation. Something will change. Either the architecture will be made less crappy or the team will be displaced. If there's an area that needs a little secret sauce it's architecture. Architects often point to lack of budget and/or control for the crappiness that surrounds them: there's no funding to do it right and no power to fix projects that are doing the wrong thing.

I submit that both of these notions are false. Firstly, investment has nothing to do with quality. Another day I will expand on this but there's a good argument to say that the more you spend on software products the less (per monetary unit) you actually get in terms of delivered business value. Secondly, good architecture has nothing to do with control. It may come as a surprise to many but architecture practices, particularly governance, are directly responsible for a lot of the crappiness architects complain about. Yes. That's what I am saying - it's not that an architecture practice can be merely ineffective in some benign nice-but-useless fashion, architectural activities may actually be making a bad situation worse.

Let me explain how. You may notice that I mix up enterprise architecture, solution architecture and design here - it's deliberate. The job title is unimportant. What I want to do is make a distinction between technical people who should help developers deliver value and the development teams themselves who, ideally, would just happily hack away, making the value-delivering product.

An All Too Common Story

Typically, change starts with an idea. As that idea is discussed and elaborated it gathers detail. If architects are involved in the conversations this detail begins to include decisions on things like technology approaches, systems to be reused, information flows, patterns, products to be licensed, and so on. We call this detail 'design' (or, more ominously, 'the blueprint'). Early design work is there, to some extent, to give stakeholders confidence.

But this confidence is false. In the early stage of a project, many aspects of process, detail, and design are in fact constraints. Constraints that will hamper the development team when they come to build the solution - in the form of tools (or dependencies) that are not fit for purpose, patterns that do not allow for requirements to change, and so on. Essentially, the development team gets saddled with an architecture that is unable to emerge gracefully as more detail is known and piles of documentation describing this inflexible tower of dung that must be kept up to date. Add architecture governance to the mix and you have the perfect recipe for flapdoodle - enforcing the constraints that caused the problem, slowing everything down, pissing everyone off (regional variations of this dish include large meetings, emails with a lot of cc'ing, and extra helpings of that documentation mentioned earlier).

Eventually business stakeholders lose their misplaced confidence. This commonly occurs during the test phase when the end-date becomes more and more indeterminate. The build took longer than expected, which creates pressure to deliver more quickly, adding short-cuts, complexity and defects. Testing time is cut. Defects that are found are fixed too quickly so have a habit of introducing more defects. If the project does ever go live, it will do so with a great deal of design debt and overall the landscape and the relationship with the business is made worse, not better.

An All Too Common Response

Things are worse. The strategy says to make things better, simpler. Clearly nobody followed the strategy. Oh and did I mention the business think the IT department is retarded? The Enterprise Architects do some soul searching. How do we get confidence? How can we make sure we know what we're doing so that the business don't think we're just badly-dressed chimps? Need .. more .. confidence. Need. It. Earlier. Need ..... more ... process. More. Design. More control. More Guuuuvernance.

Is Anyone Seeing a Circle?

Yes. More design, governance, and general bureaucracy, ironically makes it less likely that subsequent projects will be fit for purpose. In CIO-speak this equates to more investment required for future change, higher TCO, more downtime, poorer reliability.

The Fifth Discipline

In 1990 Peter Senge published a book entitled "The Fifth Discipline: The Art and Practice of the Learning Organization". In it he adds to the four disciplines that healthy organisations need to adopt - team work, a shared organisational vision, a shared employee mental model and an environment that empowers personal mastery - a fifth discipline: Systems Thinking. You can find a good summary on the Edinburgh University MEAB web site.

The cornerstone of any learning organisation is the fifth discipline - systems thinking. This is the ability to see the bigger picture, to look at the interrelationships of a system as opposed to simple cause-effect chains; allowing continuous processes to be studied rather than single snapshots. The fifth discipline shows us that the essential properties of a system are not determined by the sum of its parts but by the process of interactions between those parts.

Sound a bit like what Enterprise Architecture is trying to achieve?

The interesting thing for me is that Senge goes on to define some laws that govern the fifth discipline. Laws that act as a useful warning to architects who find themselves cloistered in their ivory towers work-shopping ideas on how to gain more control over complexity. Here are two:

• Today's problems come from yesterday's solutions

  Uh huh. In other words all this shit you are dealing with is a product of the process you have. The thing you are giving more credence to is the thing that got you into the mess.

• The harder you push, the harder the system pushes back

  Oh yes. You know those exercise machines that give you a cheap workout by positively increasing their resistance in response to the force you apply to them? Well people in organisations are just the same. You are unlikely to see it directly because it mostly takes passive forms. If I regularly communicate with someone who adds no value to achieving the project goal (or makes that goal harder to achieve) I stop doing it. And disengagement on a project is both poisonous and infectious.

I mention Senge's Systems Thinking because, although it's difficult to accept that complexity is a by-product of flaws in the delivery process (too much arbitrary detail up-front and too much bureaucracy), it should be obvious that good enterprise architects can bring an enormous amount of value to projects, not via control but as a client representative for a vision of something. A vision that incorporates all the moving parts of the business problem and the "ability to see the bigger picture".

Control (and measuring and governance) is irrelevant. If I came up with my best shot at solving the business problem with Design A early on and the development team (with their unique insight into the most detailed level of the problem space) come up with Design B, that has twice the amount super-awesome, do I really want to control them towards Design A? Is anyone controlling anyone or is it just better to use the vision as a sounding board?

Now, it occurs to me that I have used vision a few times in the last couple of paragraphs. I get a bit twitchy whenever people use that word because it often indicates a lengthy speech is coming up derived mostly of bollocks. Let's talk specifics.

The Agile Webscale Architect (2.0)

One thing we must all surely have learned by now is that everything changes. Ten years of agile development experience has given us a way to deal with this - making decisions at the Last Responsible Moment. We are always better off delaying important decisions until the last responsible moment, that is to say late enough that we have the richest set of facts possible to help us make the best decision and yet early enough that's there's still an effective decision to be made.

Up-front architecture design (that is then governed) is making decisions at the Earliest Irresponsible Moment and that's not just wasteful it's arrogant - as Jeremy Miller points out in Leaky Abstractions and the Last Responsible Moment for Design:

"Architect Hubris" is the belief that the architect(s) can design ... an application framework upfront that will reduce the remaining development to a mere exercise in configuration

And another thing. We have a word for doing all the work for one aspect of delivery in sequential steps - waterfall. Too much work (on anything) too early leads to phasing - a requirements phase, a design phase, a build phase, a test phase. The major problem with phased software engineering is the 'more work now and more work later' overhead. When clarity emerges about something decided upon in an earlier phase, the more collateral there is to update and maintain, the more effort there is in communicating the changes, the more likely we are to deliver late, and the more likely we are to deliver unintended complexity.

Techniques such as those in the XP stable of thought have proven the value of gathering requirements (stories), building, integrating and testing continuously rather than in phases. Small units of work (sprints) comprise all these activities happening together (some in preparation for subsequent sprints) with a functionally-light, but technically sound, deliverable at the end of each sprint.

I'm not saying XP is perfect but it does represent a mature collection of practices that go a long way to making the unavoidable issues of software engineering (e.g. people always want more than can be delivered in the time available) transparent and obvious early, and the baggage that comes with waterfall reduced to within manageable tolerances - both of which leave time to focus on the value-adding stuff like agreeing what should be done and delivering working software.

Not a lot of room for a traditional EA to show up with his blueprint, governance clipboard and pen.

So the challenge is: get good architecture that does the kind of things we've been talking about to come out the other end, without control, without governance and with only a light touch idea of a blueprint (basically enough to get started).

Good Architecture without Governance

• Discipline

  Adopting software delivery practices that operate on a continuous basis requires discipline. Everyone on the team has to follow the discipline to make it work. Developers need ready access to suitably-sized stories, pre-packaged with decent acceptance criteria so they know what the satisfactory outcome is. These stories need to be a blend of most valuable and hardest first. The business can tell you the value, ongoing design helps highlight tricky decisions that need to be made.

• Test Driven

  Another day I'll write a whole thing on TDD and how it relates to the role of an architect. For me it's pretty much the missing link. Projects that adopt TDD often do so to support safe refactoring (i.e. continuously clean the code knowing you haven't broken anything) but because the tests are written first it's useful to think of it as much a design tool as it is an enabler of maintainable code (which should be the same thing anyway).

  Here's a nice quote from James Hunt on what goal of this process is:

  When adding a feature the goal isn't getting from A to B in the simplest way. The goal is to make the code at B look as simple as possible. You're not done with a feature when all your tests pass, you're done when all your tests pass and the code looks as if it was designed from the beginning with your new feature in mind.

  Developers write tests to model expected behaviour. The tests fail, because the solution isn't yet written, and now they have the latitude to write code to pass the tests. And latitude is important here in many ways - there's freedom to best fit a solution to something like an executable specification, freedom to tweak and re-tweak sub-optimal code, freedom from the (later) nightmare scenario of having running code in production that nobody understands and nobody wants to touch.

• Continuous Test

  Tests become part of the build environment run every time code is checked in. Over time the test base grows to hundreds and then thousands of tests.

• Continuous Deploy

  Taking all that tested code and building it into a new environment is really an extension of testing that flushes out things like hard-coded config settings or hand-crafted shims that might have been forgotten about. A build that's regularly deployed is much better understood by the team than one that isn't. This should also happen many times a day.

• Continuous Fun

  There are plenty of detractors when it comes to TDD and continuous-whatever. They'll say things like 'but it doesn't scale'. Personally I think it does. Most practices have scaling issues when the project, or problem, or team, is too damn big, which is an indication that restructuring is necessary, not that practices need to change. Plus I think the arguments are missing the point of all this: confidence. Remember I said that upfront design is one way people seek to give the business confidence? Well that's no bad objective. It's just that a better way to give the business confidence is to be confident yourself. And confidence has a useful side-effect we call fun.

  Something I missed in my "Ungoverning the Business" article about open-source vs. governance was fun. Eric Raymond in "The Cathedral & the Bazaar" had this to say about the value of fun:

  .. because we have fun doing what we do. Our creative play has been racking up technical, market-share, and mind share successes at an astounding rate. We're proving not only that we can do better software, but that joy is an asset.

  When considering doing anything that affects how people work, it's worth taking into account its impact on fun. Pursuit of one asset at the expense of another is a dangerous trade-off. In respect of governance processes it would be wise to remember another nice quote from Eric Raymond:

  A happy programmer is one who is neither under utilised nor weighed down with ill-formed goals and stressful process friction. Enjoyment predicts efficiency.

• Continuous Understanding

  Keeping the story engine stoked takes a lot of effort. There's a constant need to look some steps ahead to prime the pipeline with suitably sized, clearly defined (via acceptance criteria), suitably prioritised stories. Priority is based on a combination of what makes sense (logical dependencies), what adds value, and what is technically hard, so that difficult decisions aren't left too late (beyond the responsible moment). Technically hard problems often fall into the design domain of unknowable things.

• Continuous Learning

  Unknowable things (stories that have no obvious solution or which require more upfront thinking time) are spiked. A spike entails trying solutions outside of the disciplined build pipeline so that ideas can be explored without impacting other developers.

  The result of a spike is usually knowledge - a better understanding of the best way to approach the solution, which could be purely technical or could simply be a better way to phrase the story (scope, acceptance criteria, etc). The important thing is spikes are time-boxed and the resulting code is thrown away.

  When the story is replayed (or re-spiked) the knowledge gained informs a refined approach.

• Continuous Governance

  If it's not now obvious, there is no need for traditional code reviews or formal governance in a continuous world because it is happening all the time. But, as the client representative for that broader architectural vision mentioned earlier, the EA can still play a key role in all of this: stories represent the business view (which informs the EA when requirements shift or clarify), some stories and acceptance criteria come from the EA directly (notably those that represent non-functional requirements), the build process and applied tests are both clearly visible so the EA can see immediately how the code and the vision line up. Unknowable things can be an architects playground, bouncing ideas and researching potential solutions. The EA can also play an important role in helping identify the order it makes sense to play stories, so that value (which can manifest itself in systemic as well as functional forms) is being delivered early.

  EAs should be good at this because all these outputs feed into the post-project landscape - the thing the EA is accountable for. The currently crappy thing that they really don't want to be getting fired for. The thing that creates TCO, complexity, maintenance cost, and the ease or friction that dependent, or future, projects will experience. We are not at home to Mr Cock-up

  The prerequisites to make this role successful are simple, but not always an easy shift for the traditional EA mindset to make.

The Enterprise Architect and the Change of Mindset

• Developers do not need to be policed

  Good developers have no desire to build the wrong thing or the right thing badly. Given the right steer most developers will seek to make their lives easier by making code easy to maintain (because initially at least they are the ones maintaining it and they know they will be coming back to update it often).

• Fear not the detail

  Detail and code are nothing to be afraid of. Most of the value delivered by an IT project is in some technical form. Ensuring the realised value is the highest it can be requires engagement at a detailed level. An EA must be part of the process, not someone who drops in late in the day to check up on alignment. Whatever the fuck alignment is. Which reminds me ..

• Alignment is dead

  I say this all the time and I will not stop saying it until I am carted off in the back of a van for a long lie down. Forget .. and I mean .. disregard totally .. the concept of business and IT alignment. In fact you have my express permission to excuse yourself from any meeting where this is the subject of conversation. Everyone who comes to work in the morning, does something, goes home and gets their bills paid by the company is the damn business.

  A "business" is made up individuals with different roles. IT has some of those roles; other people have other roles. You have customers that are represented by your product owner for any one project. The whole business has certain objectives that are represented by Senge's second discipline comprising a shared organisational vision. You don't need to align to that because you'll hear all about it via the product owner (though politeness would suggest you make an effort to read about it so they don't have to repeat it). Alignment (delivering what's required in the best way) is just part of the process, not something that happens separately. If you want to align to something, align to making better software.

• Emergent Design is good design

  Finally. We're here. Architects will often give ground on many of the other points but emergent design seems to be the stickler. It's the one frightening thing that leads them back toward big-design up-front and all the time-wasting governance that ends in bad architecture.

  All I can say is that all projects are journeys of discovery, for everyone involved, including the business, it seems wrong-headed that design has some special immunity from this. Will you discover things along the way that blow the budget and the plan? Yes, quite possibly. That's life. Not an argument for more up-front design. If you could spot a problem six months in advance by designing it, and everything else, up front then it's true you have six months extra to address it. You also have the changing design for the other 99% of the solution to maintain. And if you can spot something really big and important six months in advance you can also do that without designing it all. If it's a minor problem them the six months you have bought yourself is outweighed by the ungainly mass your project is now carrying.

  That's why the secret sauce is both a real and an imaginary thing. It's real because it exists in a very emergent kind of way, as good teams form and work out how they're going to tackle things. It's imaginary because you can't just make it happen anywhere, with anyone, after reading a book. Just like great architecture in fact. My favourite quote of the year so far was from Craig Larman at QCon 2011. In his talk on "Scaling Lean & Agile: Large, Multisite or Offshore Delivery" he said:

  Architecture is a bad metaphor. We don't construct our software like a building, we grow it like a garden.

  It got tweeted a gazillion times so you probably heard it, but when you get what he means, you get it all. Including the secret sauce.

Notes

1. The Secret Sauce is a bit of an over used metaphor, but as I was talking Noodle Soup last time it seemed to fit. Incidentally, I found The Secret Sauce of Highly Productive Software Development on InfoQ quite interesting when I was researching just how widespread the phrase is.

2. The idea for this piece was sown during a conversation with Thomas Moore (his tweet feed is locked but it's worth asking to join his retinue: he does a good line in dramatic weather reports) to generate some ideas to create a logical narrative to help Seb Chakraborty put together some slides for a Gartner talk.

3. The saying "A's hire A's; B's hire C's" has been around for a while. I tried to track down where it originated, but it has so many variants I couldn't pin one source down. Donald Rumsfeld, apparently, said it but I think we can be fairly sure he never made it up (that feels to me like a known-known). My best guess is it originated in Apple. This is implied by Guy Kawasaki.

4. If you want to read more about the fifth disciple and how it relates to software development "11 Laws of The System Thinking in Software Development" by Andriy Solovey covers it quite nicely.

5. The painting is an excerpt from "The Marvelous Sauce" circa 1890, by Jehan Georges Vibert.