Scaling With Agility
Deceptively simple, yet endlessly complicated
Tags: agile , development
It’s fifteen years since the Manifesto for Agile Software Development was published. Since then it seems most large companies have made some attempt to make themselves more agile. Either by adding ‘agile’ as another methodology alongside waterfall in their process framework, or embarking on a Transformation Programme, involving plenty of highly-paid consultants who come with a certificate that says they can recite an agile process framework word for word.
Of the companies that have had a go at “agile”, some have had a good experience: a single project delivered without any nasty surprises, by an unusually small team and with little fuss (at least internally to the project team, it’s quite common for there to be great deal of fuss from the individuals who guard the incumbent process). But then that good experience proves impossible to repeat, or to scale up across the business.
Others had a bad experience and wrote agile off as a fad, or just an alternative delivery vehicle to waterfall, and went back to whatever process they had before, preferring the devil they know to something that ‘only works for start-ups’.
Either way, software delivery processes returned to something painful, unpredictable and generally unpleasant to work within, sometimes with a few fake-agile remnants like daily stand-ups and cards on the wall to make the place not feel quite so 1990s.
And yet, here’s the thing: Software Development is a Largely Solved Problem
Software developers who have real experience with agile practices would agree on that. Nobody’s saying software development is easy, or that there’s a magic recipe, just that we are pretty good nowadays at tackling complex problems in small pieces, designing as we go, working in small teams to test our software as it emerges, and communicating back to customers what they can and can’t have.
But I can tell you that every CIO I’ve talked to in the last ten years does not agree. There may be a lot now who’ve had some form of ‘agile experience’, but they all tell a similar tale: a few smart alecs turned up, did something innovative that lasted maybe a year or two, and for reasons that are hard to identify, either the project became another IT mess to clean up, or trying to repeat it just didn’t work out.
Yet the so-called Digital threat is real. CIOs are looking like dinosaurs compared to the fifty start-ups on a frightening trajectory to come and eat their lunch. These start-ups are not going away and with each passing year the barriers to market entry are getting lower. And these start-ups are agile in their bones. They don’t need a Digital Transformation initiative to become better at delivery. They chucked the rule book in the bin years ago. Enterprise managers may use the term ‘start-up’ to disparagingly imply amateur or inconsequential, but these companies are anything but. They’re valued at billions, well financed and mostly loved by their customers. The boogie man exists. If you’re a CIO, these are desperate times.
And desperation is a dangerous thing, because consultancies can smell it from miles away. Where there is fear and unease, there’s a glossy and expensive solution just waiting to be packaged and sold. And with agile and digital it’s the Scaled Agile Framework, and any of the other ‘structured’ methodologies that purport to turn your lumbering bureaucratic delivery structure into Facebook.
I am not opposed to making money from selling insight and advice - wrapping your head around some agile techniques can be a bit tricky and it’s certainly worth paying for the benefit of someone’s hard-won experience and delivery expertise. What I cannot accept is the greedy cynicism that underpins these commercially-structured schemes, selling what is ultimately common sense (albeit a sometimes a counter-intuitive common sense).
As customers, we’re buying into it too. It’s being aided and abetted by legions of failed project managers, popping out for scrum certification (all of 16 hours of training), and re-badging themselves as agile coaches. Unless you really get what it means to develop with agility, and you have done more than just read about it, you have no business selling advice to others. There’s a bit more to it than a big diagram and a manual. Or throwing terms like ‘epic’ and ‘velocity’ around without quite grasping what they mean. Or suggesting new tooling because Netflix does it. Your company may benefit from a chaos monkey, but you don’t have to be it.
The situation is so bad it prompted Dave Thomas, one of the signatories to the Manifesto for Agile Software Development (and not, as he says, the agile manifesto, or the manifesto that is itself agile), to declare Agile to be dead and reborn in the flames of common sense as agility, because you can’t sell an adjective, you can only sell nouns. You can watch him talk about it on YouTube.
So let’s talk a little bit about what agility really is and why you can’t buy it from Accenture or get certified in it over a few lunchtimes. Dave Thomas begins his argument with the plain statement that, when presented with a problem, none of us know what to do. That, to me, is the first step on the road to enlightenment - accepting that when we begin a project, or discuss a product idea, or a strategy, or indeed any endeavour, we don’t actually know what to do. We may have ideas. We may even have very strong ideas. Our ideas may be based on experience. But whatever these are, we do not know with 100% surety whether they will work. Or whether they will still be the correct answer six months from now.
And you know what? That’s actually fine. We are human and whilst we live in a world that changes with time, even the very best of us have to adjust our thinking as we go.
Hold hands and say it with me: We have not the faintest idea how we are going to develop that new digital big-data social-commerce machine-learning whatsit and the truth will set us free. Because now we are ready to find the right way. And that requires agility. Agility to change our solution as new evidence comes in. Agility to work with others as they add to the bigger picture. Even the agility to experiment with our approach to solving the problem. After all if we don’t know what the solution is, we can surely admit we don’t know the right approach to finding it either?
Dave Thomas suggests the following as his definition of agility in this situation:
- Find out where you are
- Take a small step towards your goal (and if there are multiple choices here, take the path of least regret, or the one that makes future change easier)
- Adjust your understanding based on what you learned
And if there was ever an example of the profound in the mundane then that’s it. Obviously there’s a bit more to what you might call the prevailing norms of agility than just that, otherwise we’d all be making everything up from scratch each time, but all the methods that make up standard agile techniques were derived this way. Kent Beck’s Extreme Programming comprises 12 practices that found their extreme incarnation by identifying something that was good (the goal) and taking steps towards it until the optimum version of the goal was achieved.
Code quality is a good thing (a goal) and we know that code reviews can help improve it. But formal code reviews are long and wasteful and often happen some time after the code was written (therefore losing context). Making the review closer to the point in time the code is created is a small step towards the goal. Having two developers collaborate on the problem means the review and the creation happen at the same time. Is it counter-intuitive? Yes, it certainly is for many, who see pair programming as paying two developers to solve the same problem. And for programmers not used to it, it can be a bit painful. But then we baby-step away from pairing all the time towards pairing when it’s useful. Or, use pull requests to enforce a continuous review process. Whatever works for the team.
There are no absolutes. Pair Programming can certainly be useful for improving code quality. Mocking those who don’t do it is not useful if the goal is to improve agility in development (though it’s very useful when the goal is pointlessly argue over who’s the smartest developer in the room). If a team are resistant to pairing in any form, then fine, choose something else to try instead.
When your development iterations are short, the next check-and-adjust cycle is only days away. Bad decisions can be rectified and good ones continued, or even improved upon. This is why agility is hard work. It requires continuous focus and rigorous discipline. I suspect that’s partly why it fails in some places - because there’s no time for slacking off, or having lots of meetings. A great scrum master I worked with once said: “the best feature of waterfall delivery is it gives you two years to find another job.” With short iterations, the questions may be smaller but they come thick and fast.
At no time does the answer involve looking in the book that tells us what to do. There is no book. There are plenty of books and views and blogs and articles to use as food for thought. But no single answer. And that should not be a scary prospect. At an average company get together, there might be over a hundred years of domain knowledge in the room. How could any outsider possibly think they can come in with the ready-made answer that fits your organisation?
Which is why it’s hard to explain this:
That diagram, copyright, and property of Scaled Agile Incorporated. Represents the Scaled Agile Framework. Take a moment to marvel at all the upfront design that went into that. All the correct words are in there, as used by programme management offices, financial planners and architects worldwide, which will surely minimise any resistance to implementing it. No new process would be complete without a profusion of training options, so that the anointed few can waft their certificates at the great unwashed.
And look at the big check mark at the end of the process that says “built-in quality.” Oh, quality is built in? Pardon me and send the invoice for Gold Partner Level 82.
Before we move on, consider that not one signatory to the Manifesto for Agile Software Development has approved the Scaled Agile framework. But quite a few people have had less than complimentary things to say about it:
Some have even come up with their own alternatives, like LAFABLE
But it’s not a laughing matter. Remember the desperate CIO we talked about at the beginning? They have to make tough decisions to change their organisation and they will likely only get one chance. It’s not much good dangling a manifesto in front of them and making fun of consultancies. Big consultancies have a lot of clout in enterprise organisations. There’s no deferential supplier/customer relationship for them. If they think the CIO won’t swallow their two-year SAFe training programme they only need to quietly whisper in the CEO’s ear “Are you sure you want to take that risk? We’re Gold Partner Level 82 SAFe Practitioners and your alternative seems to be a manifesto, a four step process, and some childish jokes.”
Hell no. You’d be a brave CIO to take the leap.
A comment I hear a lot is “but agile has nothing to say about scaling up.” And that’s true. It doesn’t. When those 17 people went away to debate better ways to create software, they were thinking about defeating the overbearing process and backwards thinking that were preventing individual teams getting things done. How that scales is a different problem entirely. The principles of scrum (accepting that scrum is not enough on it’s own) are about team working, not portfolio management, financial planning or inter-product integration.
How does agile scale? is the wrong question.
Think about cooking (I know, bear with me for one paragraph). A cheese sauce is made up of a roux (a half-and-half mix of butter and flour) that’s cooked for a bit, then some milk added and finally some cheese. But the recipe isn’t rigid. Some variants warm the milk with bay leaves. Some use mixed cheeses, or add white pepper, or black, or both. Take that roux and add some stock instead of cheese and milk and you have a Sauce Espagnol, one of Escoffier’s mother sauces, from which many French dishes are developed. Chuck a roux in a soup and you are half way to a stew. Swap the butter for bacon fat and you have a Cajun Roux. Cooking is made up of hundreds of small methods that can be combined and recombined to produce amazing things. You can buy recipe books to learn the techniques and ask experienced cooks for their opinions.
Why is that excursion into cooking relevant? Well because if our projects involved making the same fucking macaroni cheese every day then I’d be all up for a pre-defined process to standardise and optimise it. If you eat in McDonalds, that’s exactly what they have. They can continually optimise a process, and scale it, because every day they do the same thing. We don’t. As software creators (and I include all roles in that definition) we are asked to take on new and wondrous challenges all the time. Last year’s macaroni is dead, people. All you can do is agree the goal and take a small educated step towards it. The more skills and experience you have, the easier those steps will be. That’s why agile teams are self-organising, not because they need to be interrupted less often by other teams (though that’s a nice side effect) but because they have to constantly work out the next best thing to do, and their success or failure in that is shared. As soon as you ‘scale’ team practices, by smearing a sort of half-baked scrum up the org chart, the team at the bottom can no longer work with agility. That’s what SAFe does.
A few other nasties to look out for that scaled agile methodologies subvert:
A team should contain within itself all necessary roles that are required to keep work flowing. If the team has to meet with other teams to achieve that then the process is broken. That’s not to say some meetings aren’t necessary, of course they are, but not ones related to hourly decisions of what to do next.
A team should be able to deliver working software into production the moment they believe it is right to do so. Otherwise what you have is something closer to what Dave West called water-scrum-fall - where waterfall gates surround a scrum-like delivery team who will struggle to do the right thing or get the right thing into production. i.e. ‘done’ should really mean ‘done’
Talking of ‘done’ - an iteration is where done happens. There’s no special hardening/testing/reworking iterations that come later. Quality comes from finishing the work while you are still thinking about it. Calling something ‘potentially’ releasable is the same as saying you wasted your time, and the company’s money, on it. Yoda is your friend here: do, or do not, there is no try.
When building a product, the process (whatever it is) should support iterative development, not incremental. If your web site needs a login page then build the simplest one and iteratively make it better until it’s good enough for launch (which could be sooner than you think). Approaching the problem incrementally would mean finishing all the features of the login page before moving onto some other aspect of the site. I like Alistair Cockburn’s summary which is that incremental thinking is good for process improvement (which is why project managers are dangerous around agile teams) whereas iterative is best for product improvement. This diagram is old now but summarises the difference nicely.
Then there’s the concept of the Agile Release Train. I don’t know where to start with that. To be honest if anyone thinks 5-12 teams (that’s 50-125 people) working in a virtual organisation for 10 weeks, that’s then followed by more specialist iterations for hardening, releasing, testing or whatever, isn’t just waterfall in disguise then they are beyond salvation. To be fair, SAFe itself does not promote specialist sprints but it does allow for them. I’d rather teams set themselves the goal of releasing every month, then every week, then every day.
And finally a word about architecture, which truthfully is what irks me the most about SAFe. It’s the big-thinking metaphor. It’s the idea that the only way to get something ambitious done is for teams to work together on a big thing. No. Just, no. Are we so thick that anyone believes that’s ever worked? John Gall in 1975 said:
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
Real agile teams work because: a) they are forced into being small which means b) they naturally limit the complexity they can handle which means c) they can only deliver simple systems that work. I think I might be in the minority in believing that architects are still needed to support those teams in producing simple systems that co-operate, but whatever. The point is that it’s relatively easy to take on the ambitious thing without having teams report into some verbally mangled train metaphor.
Agile practices were famously inspired by lean manufacturing (a great and very readable book on lean by the way is “The Machine That Changed the World” by Womack, Jones, and Roos. It’s 26 years old but then so is lean). Lean methods are, not surprisingly, similar in nature to the description Dave Thomas used for agility:
- Never produce anything until the customer wants it
- Always define clearly what value the customer wants, not necessarily what they ask for
- Define the steps of the process flow that delivers the customer value and always keep the value stream moving
- Continuously improve the value stream and your understanding of the customer
Scrum and XP apply this thinking to teams (and we’ve had 15 years to challenge, improve and optimise that thinking) but it can also be applied to other parts of the organisation directly (subject to the same improvement cycle). Check out Lean Enterprise by Jez Humble, Joanne Molesky and Barry O’Reilly for more on how lean principles can be applied to larger businesses.
You want to scale ‘agile’ to portfolio management? Then stop planning things a year in advance. Stop estimating huge programmes of work you don’t understand (when, as we’ve already covered, the agile team doesn’t yet even know how to approach it). If it’s not clear what the customer wants then don’t do it. Do something that does have a clear value or economic benefit instead.
And remember one thing above all - the whole point of a consultancy business is to grow itself using the money it gathers from its clients. Process frameworks are seductive because they make us feel safe (no pun intended) as we attempt to change. There’s nothing wrong with wanting to feel in control. But stay in control. It’s your business, not theirs. When you see the house made of cakes and confectionery and the windows of sugar, don’t eat it. Yes, it looks delicious but that’s the whole point of it. Turn around and follow the breadcrumbs back home and try a different way.
The painting is Muchachos trepando a un árbol (boys climbing a tree) by Francisco de Goya y Lucientes
This article was originally published on LinkedIn
The subtitle is a paraphrased quote from Arnold Palmer who was describing golf. The full quote is “Golf is deceptively simple and endlessly complicated; it satisfies the soul and frustrates the intellect. It is at the same time rewarding and maddening - and it is without a doubt the greatest game mankind has ever invented.” He could so easily have been describing software development.