The Fallacies of Agile

A Project Management Office I used to work with recently put out some statistics on the current portfolio of active business change initiatives. Eighty-one percent of them are being delivered using an “agile” methodology. If you’re an agile proponent this could be taken as a measure of some success - clearly this is a culture that embraces a responsive and intimate relationship with its business. And yet, on closer inspection, not one of these projects is structured around fixed-length iterations, burn-down charts, velocity metrics, or an onsite customer. None of them favour test-first thinking, continuous integration, minimum viable product, or any aspect of that “most-value for least waste” sensibility that underpins the majority of agile practices and rituals.

And talking of rituals - when questioned, nobody could actually say which of the plentiful options (Scrum, XP, DSDM, etc.) they were subscribed to.

What we have here in fact is a clear case of Cowboy Coding - a misappropriation of the term agile as a kind of corporate camouflage to enable projects to just start doing whatever it is they are doing (completely different things in all cases) without any form of constant value for money questioning.

Now don’t get me wrong. The biggest early life risk to all projects I have ever known is “not starting”. Painful workshops and endless discussions are not productive, nor are they value for money. A short sharp inception that teases out just enough of the required value to get started is usually all that’s needed. I am a big fan of getting started.

But just-doing-it is not the same thing as agile. And what tends to happen when you masquerade one thing as another, and don’t know what you’re doing, is the tool get blamed rather than the operators. I am seeing many companies say things like:

We tried agile. It doesn’t work for us. We need a level of certainty it doesn’t provide. We need to know when we’re getting things and how much they’re costing. Agile felt too much like a chaos that IT enjoyed because they reduced bureaucracy but the business couldn’t get a straight answer to simple questions.

The first agile project I was involved with was not exactly what I would call a case-study for success. We got there in the end but, frankly, we were lucky that what we delivered was so good and so useful to the business it masked a lot of what went wrong in the delivery cycle. Our estimating and measuring was woeful. Our rituals were daily stand-ups and an excessive use of stickies.

But we all learned some very important lessons. After the next (better) project the business sponsor sent me an email saying:

I didn’t get about half the things I asked for, but we hit the date and I got what I needed. So I am happy.

That was something of a profound moment: “hit the date”, “got what I needed” and even a “happy” at the end.

But alas some time later, after more failures, that delivery organisation adopted a different tone and agile was dropped as a way of working because it was considered to be nothing more than “an excuse to cut corners” and not deliver “properly”. This experience is far from unique.

Agile’s name, if not yet ruined, is certainly heavily tarnished in organisations that have either wilfully abused it or not bothered to adopt the rigour needed to make it work effectively. If you came to this article looking for a little ammunition to support some corporate Agile bashing, then I’m afraid this isn’t it - though I have reached a point where I cannot comfortably use the word.

Agile as a mindset and a collection of good practices (the manifesto), applied in the extreme (XP), or in delivery (SCRUM), is superior to much of what we traditionally call the “waterfall model”. Waterfall is bad in its bones - it’s a staged approach of artificial phases that positively prevents value-first, test-first, quality-first thinking. How anyone, during an early analysis phase, can extract simultaneous value, priority and risk and know what will work, how it will impact other tasks, etc is beyond me. Their may be a race of super-beings out there that can make waterfall work but humans aren’t it.

So once in a while we try new things. And “Agile” is just the latest in a series of fads. Big on promises but, because of the discipline required, mostly low on delivery.

So, with profound apologies to Peter Deutsch (and Bill Joy, Tom Lyon and James Gosling..), here are my fallacies of agile computing. A collection of misunderstandings, fibs and sophisms that, like the original fallacies, lead to mischief and mayhem.

1. Fallacy: It’s Agile

   Let’s start by clearing up what Agile is. In 2001, seventeen luminaries of the software world descended on a ski resort in Snowbird, Utah and amongst other things agreed on something called the Agile Manifesto.

   We are uncovering better ways of developing software by doing it and helping others do it.
   Through this work we have come to value:

   Individuals and interactions over processes and tools
   Working software over comprehensive documentation
   Customer collaboration over contract negotiation
   Responding to change over following a plan

   That is, while there is value in the items on the right, we value the items on the left more.

   The manifesto itself is nothing more than a wake-up call to software engineers telling us we should be more human and empathetic and less prescriptive and procedural. This is essential because it’s important to recognise that the second two are merely how we get there whereas the first two reflect what we do - that is to say one is good for us and the other good for our customers. All lovely and good, if perhaps just short of being tree-hugging hippy crap.

   What’s important about the manifesto is it’s just that - a public declaration of principles and intentions. It’s is not a method for delivery. Kent Beck had been working on what we now call XP for five or six years before that, Scrum’s origins date back to the early nineties. What the manifesto did was provide a word to make software delivery sound cool. Just like Java did for OO development. Unfortunately for all of us, coolness is often the kiss of death to implementing anything effectively because it attracts all the wrong kind of attention.

   So agile in a way doesn’t really exist. You can certainly prioritise responding to changes from your customer over beating them on the head with the plan. It’s all about communication - letting them understand what their changes mean, giving them lots of options, and involving them in making the key decisions. The problem is that only tells you how to act, not what you should do. What’s missing are the good practices of software development.

   That leaves the rest of my list with something of an issue. If agile doesn’t really exist, it’s going to get pretty darn confusing discussing fallacies of a fallacy. So what follows is a list of things I have heard said by otherwise sane individuals about the silver bullet that is perceived to be agile. Some of these relate to practises within methods like XP and so I accept that supporters of these will not necessarily agree, others are more generic myths that seem to have sprung out of nowhere and are worth debunking.

   Fact: There is no such methodology as “agile”. There are techniques that can be applied in any methodology, and approaches that embody many of these techniques under one banner. All should depend on frequent communication with the customer and a responsive attitude to changing needs. Whatever banner you do decide to deliver under, it should be tailored to meet you needs because methodologies suck when you follow them without thinking. And no approach works without a determined focus on internal software quality.

2. Fallacy: Agile is Faster

   When a customer asks for something specific, in a certain context, with certain constraints, there’s very little you can do to make delivering it faster. All you can do is avoid delivering anything that isn’t absolutely necessary to meet the requirement. You can also try and change what’s delivered, or change the constraints. And again we’re back to communication. Nothing mind-blowing there. Nothing agile either. Just common sense.

   Certainly a good practice here is not to focus on getting a big list of requirements that are signed-off and change-managed right at the start. There’s a great benefit in realising early that you’ll never be able to deliver everything, so adopting techniques to make sure that the business get those things that they really need can only be a good tactic.

   Fact: Agile is not faster than other approaches. It seems to be sometimes, because it weeds out what’s not essential (waste) in favour of uncovering the underlying value to the business. Sometimes we’re told that everything’s valuable. That all requirements are mandatory. They’re not. It takes tact and insistence but usually there are plenty of requirements that fall into the “you wouldn’t go out of business if you didn’t have X” category. There are exceptions. Infrastructure projects for example, There’s less scope to negotiate your way to a half-built data centre that’s delivered on time.

3. Fallacy: Refactoring

   A lot of developers talk about refactoring these days. Refactoring is not rewriting, it’s restructuring and optimising existing code to perform the same task, only in a better, more maintainable, more readable, more efficient way. The reality is whenever you write code you learn something (and this is true no matter how good your design is). After a period of learning, you want to fix it up so that your code legacy reflects your professionalism, and to avoid sniggers by support staff when they see what a dolt you were.

   The problem is, it’s not always easy to refactor. Martin Fowler’s excellent book covers many of the code smells you might come across or inadvertently create yourself and how to fix them. The problem is that in a project situation it’s all too easy to leave that refactoring in favour of more ‘working’ software until such time that making those changes is not easy. And if the requirements have changed, you might have created a super-smell. It’s not your fault and, because you’re doing super-communication too, your customer is aware. But now you’re going to be changing code and not adding to your working software base.

   Fact: Proper refactoring is critical to delivering quality software. “It works” is not the same as “it lasts”. Therefore refactoring, within the safety of test coverage, is a necessary, and continuous practice. Not doing it, or using the term to cover other kinds of change, helps nobody.

4. Fallacy: Just Good Enough is Good Enough

   Neither of these are true:

• A solution has to have big up-front design, requiring scholarly thoughts, stakeholder buy-in, academic arguments, fanciful pictures and an architecture blueprint otherwise the lack of prior thought will cause confusion later.

• A solution requires only action. Fears about future problems equate to waste and therefore don’t belong in Agile. Tomorrow’s problems can be solved tomorrow. Whatever that additional feature is You Ain’t Gonna Need It.

  I confess to leaning slightly toward the second perspective, but why does it have to be one or the other? Designing and coding should adopt a story by story negotiated point on the spectrum between these two (or the story needs rewriting). Perspective One means you have hired the wrong architects. Perspective Two means you have hired the wrong developers. Faith that you can solve tomorrow’s problems tomorrow is an important team value, but so is faith that the way you build today will help with that. You will be wrong on both counts from time to time.

  Fact: A design that is only good enough, will fall apart under continuous change just as easily as will one that’s been over-thought. For a design to have flexibility built-in it requires some up front thinking, beyond just today’s immediate issues. Sometimes changes come out of left field. You can’t cater for them. That’s life. You do what you can.

1. Fallacy: Onsite Customer

   It’s great that software folk are getting into the idea that frequent and effective communication is crucial to success. It’s rare you get the same from the business. That’s not necessarily a criticism - there just aren’t that many ‘customers’ who have the time to dedicate themselves to one project.

   It’s also not often the case that there’s one customer, and where there’s more than one you can be sure there will be politics. That’s what the Business Analyst role was created for - to elicit requirements, resolve conflicts, facilitate workshops, propose solutions and ultimately get enough solidity to go and build things.

   Fact: customers are customers, usually, because they aren’t IT. They are busy with their day jobs and therefore sometimes can’t, or won’t, be prepared to collocate with an IT delivery team. More often than not, one IT project needs to please multiple customers, who have conflicting aims and different opinions on what the priorities are.

2. Fallacy: Pair Programming

   It’s a well known fact that good programmers can be ten times more productive than average programmers and all programmers create their best stuff when they have enough peace and quiet to get in the zone so that they can hold vast amounts of context in their head at one time. That’s why quiet areas are good for coders and rodent farms of noisy cubicles are not. I also accept that regular code reviews are good for quality, and that pair programming represents the extreme practice aspect of this.

   But everybody’s different. In Myers Briggs terms I am an INTJ, which means I tend to internalise my thoughts rather than share them, my preference when faced with a problem to solve is to go away and think about it for a bit. I don’t like thinking out loud very often. Pair programming, I suspect is better suited to people who are either complimentary in this respect (an Ixxx and an Exxx) or who are both comfortable externalising their thoughts. Others are more direct in their mistrust, whereas some see humour in the idea of two techies forced to work at one station.

   When it works though it works well. I read an excellent post once describing how, given two developers of vastly differing ability, pairing will deliver code to the quality of the more proficient coder. And this is surely better than them working separately, because any duff code will certainly need refactoring by someone else later.

   Fact: Regular code reviews are good, mentoring is good. The close cooperation required for pair programming is best agreed by the team. Pair all the time, pair in the mornings, pair for tough tasks, whatever. That’s a team value, never a corporate policy.

3. Fallacy: Documentation

   I shouldn’t have to write this. The idea that there’s little or no documentation in an agile project is not so much of a fallacy as a cop out for lazy developers. Of course documentation for its own sake is a pointless (wasteful) exercise.

   If you can’t say why you are writing something up and who’s going to read it, you probably shouldn’t do it, but designs, plans, release notes, budgets, requirements, installation and support documentation are unfortunately necessary.

   The directive is to do just enough not to create useless documentation, or worse, none at all.

   Fact: Lots of people will need to interact with your project in indirect ways. They may not come to the party until quite late in the process. Targeted, terse, documentation is the best method we have of exchanging information when there’s not enough of us to go around to explain things verbally. You might be ‘agile’ but the rest of the organisation may not be. To get on (and be left alone) you may need to accept that. That’s also a kind of value, not always a waste.

4. Fallacy: Architecture

   I get as mad as even the most gung ho developer when architects swan about occasionally releasing vague ivory-tower statements that are impractical, irrelevant, annoying and, often, plain wrong.

   Like documentation, if architecture isn’t relevant or useful you shouldn’t do it. But pragmatic architecture is relevant and useful. In fact it’s critical. If you don’t have an overarching idea of how your project will fit together and fit with it wider environment, and meet its operational qualities and, within reason, meet the challenges of next year, then all you have is a tactical sticking plaster that will cost a lot of money to remove later.

   Fact: Projects aren’t successful just because they meet functional requirements. Other projects, later projects, strategic aims do need to be taken into account.

5. Fallacy: You Ain’t Gonna Need It

   I like the YAGNI principle. It’s good to help clear the mind and focus on developing the important requirements first. YAGNI says that if you don’t need a feature today you shouldn’t implement it. And that means “even if you’re totally, totally, totally sure” you’ll need that feature later.

   And here’s where I disagree. If I’m totally, totally, totally sure I will need something, then I’m going to build it. If I’m not sure, or I only have a hunch I might need it, then I’ll leave it until I am totally, totally, totally sure. If I happen to be totally, totally, totally sure I need it, and I’ve just discovered that it’s going to take twice as long to build as first estimated, then it would seem sensible to sense-check again how much of it I am still totally, totally, totally sure about. Adversity is one of the best foundations for creativity. There are rarely no other options and one of the main benefits to very short iterations is that they create a kind of mild adversity that forces questions of sureness.

   Fact: YAGNI today doesn’t always mean YAGNI tomorrow. Sometimes you are going to need it. You ignore this at your peril.

In late 2006 I was at the DSDM-sponsored Agile Business Conference. Kent Beck presented and was asked at the end what he would change if he could go back and reinvent XP. Without a pause he replied that he would have focused more on the people side. He said that XP has always been about trying to find techniques that work at that level.

And that’s the real message in all this. Whatever Agile is, software delivery works when you remember that you’re only trying to make your customers happy. To do that you have to make your team happy and you have to constantly engage with all the other areas of the customer business that are involved (finance, programme management, procurement, operations, suppliers) and that’s a lot of work, especially for the project manager.

It’s always been a lot of work. Somewhere in the distant past people thought that we could find ways to avoid hard decisions by falling back to something called process. Then, more recently, unscrupulous consultants told us we could avoid hard work and process by adopting their hokey version of agile.

But the underlying movement isn’t dead and if it can be used to remind us that real success involves us getting out of our chairs, talking to people, arguing to uncover where real value lies, and excising waste, then that can only be a good thing.