Strong like Bull, Agile like Weasel
permalinkApril 26, 2008. Filed under: delivery, development
Update of original essay from April 15, 2007
Our Project Management Office recently put out some statistics on the current portfolio of active business change initiatives. Eighty one percent of them are being delivered using an "agile" methodology. If you're an agile proponent this could be taken as a measure of some success - clearly this is a culture that embraces a responsive and intimate relationship with its business. And yet, on closer inspection, not one of these projects has a plan of fixed-length iterations, burn-down charts or velocity metrics, an onsite customer, or indeed the adoption of any practices commonly seen in any of the flavours of agile methods. And talking of flavours - when questioned, nobody could actually say which of the plentiful options (Scrum, XP, DSDM, etc.) they were using.
What we have here in fact is a clear case of Cowboy Coding - a misappropriation of the term agile as a kind of corporate camouflage to enable them to just start doing whatever it is they are doing (completely different things in all cases).
Now don't get me wrong. On very small projects with a clear business problem to solve, sometimes it is just better to get started. But just-doing-it isn't agile. And what tends to happens if you choose to masquerade one thing as another, when what you actually do is inappropriate (because there are plenty more cases where just-doing-it is not such a good idea) is that the tools get blamed rather than the operators.
I unfortunately have some direct experience of this. The first agile project I was involved with was not exactly what I would call a case-study for success. We got there in the end but, frankly, we were lucky that what we delivered was so good and so useful to the business it masked a lot of what went wrong in the delivery cycle. We all learned some very important lessons and immediately followed it with a textbook agile delivery, which ended with the business sponsor writing to me saying:
I didn't get about half the things I asked for, but we hit the date and I got what I needed,
so I am happy.
A few months later the delivery organisation adopted a different tone and agile was dropped as a way of working because it was considered to be nothing more than "an excuse to cut corners" and not deliver "properly".
That was some years ago, but it seems my experience was far from unique. Agile's name, if not yet ruined, is certainly heavily tarnished in organisations that have either wilfully abused it or not bothered to adopt the rigour needed to make it work effectively. If you came to this article looking for a little ammunition to support some corporate Agile bashing, then I'm afraid this isn't it, but I have to say, as good as it is under the right circumstances, under the wrong circumstances it's a recipe for disaster.
Agile as a mind-set and a collection of good practices (the manifesto), applied in the extreme (XP), or in delivery (SCRUM), is superior, in my view, to much of what we traditionally call the "waterfall model". But that's where the fallacies begin - there's absolutely no reason that you can't be responsive and working-software focused within the waterfall model.
So, with profound apologies to Peter Deutsch (and Bill Joy, Tom Lyon and James Gosling..), here are my fallacies of agile computing. They are a collection of misunderstandings, fibs and sophisms that, like the original fallacies, lead to mischief and mayhem.
Fallacy: It's Agile
Let's start by clearing up what Agile is. In 2001, seventeen luminaries of the software world descended on a ski resort in Snowbird, Utah and amongst other things agreed on something called the Agile Manifesto.
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a planThat is, while there is value in the items on the right, we value the items on the left more.
The manifesto itself is nothing more than a wake-up call to software engineers telling us we should be more human and empathetic and less prescriptive and procedural. This is essential because it's important to recognise that the second two are merely how we get there whereas the first two reflect what we do - that is to say one is good for us and the other good for our customers. All lovely and good, if perhaps just short of being tree-hugging hippy crap.
What's important about the manifesto is it's just that - a public declaration of principles and intentions. It's is not a method for delivery. Kent Beck had been working on what we now call XP for five or six years before that, Scrum's origins date back to the early nineties. What the manifesto did was provide a word to make software delivery sound cool. Just like Java did for OO development. Unfortunately for all of us, coolness is often the kiss of death to implementing anything effectively because it attracts all the wrong kind of attention.
So agile in a way doesn't really exist. You can certainly prioritise responding to changes from your customer over beating them on the head with the plan. It's all about communication - letting them understand what their changes mean, giving them lots of options, and involving them in making the key decisions. The problem is that only tells you how to act, not what you should do. What’s missing are the good practices of software development.
That leaves the rest of my list with something of an issue. If agile doesn't really exist, it's going to get pretty darn confusing discussing fallacies of a fallacy. So what follows is a list of things I have heard said by otherwise sane individuals about the silver bullet that is perceived to be agile. Some of these relate to practises within methods like XP and so I accept that supporters of these will not necessarily agree, others are more generic myths that seem to have sprung out of nowhere and are worth debunking.
Fact: There is no such methodology as "agile". There are techniques that can be applied in any methodology, and approaches that embody many of these techniques under one banner. All should depend on frequent communication with the customer and a responsive attitude to changing needs. Whatever banner you do decide to deliver under, it should be tailored to meet you needs because methodologies suck when you follow them without thinking.
Fallacy: Agile is Faster
When a customer asks for something specific, in a certain context, with certain constraints, there's very little you can do to make delivering it faster. All you can do is avoid delivering anything that isn't absolutely necessary to meet the requirement. You can also try and change what's delivered, or change the constraints. And again we're back to communication. Nothing mind-blowing there. Nothing agile either. Just common sense.
Certainly a good practice here is not to focus on getting a big list of requirements that are signed-off and change-managed right at the start. There's a great benefit in realising early that you'll never be able to deliver everything, so adopting techniques to make sure that the business get those things that they really need can only be a good tactic.
Fact: Agile is not faster than other approaches. It seems to be sometimes, because it weeds out what's not essential in favour of meeting a date, or coming in under budget. Any approach can do this, and if everything that's being asked for is essential (typically infrastructure type projects), there's no weeding out to be done. A half-built data centre delivered on time and in budget does not constitute success.
Fallacy: Refactoring
A lot of developers talk about refactoring these days. Refactoring is not rewriting, it's restructuring and optimising existing code to perform the same task, only in a better, more maintainable, more readable, more efficient way. The reality is whenever you write code you learn something (and this is true no matter how good your design is). After a period of learning, you want to fix it up so that your code legacy reflects your professionalism, and to avoid sniggers by support staff when they see what a dolt you were.
The problem is, it's not always easy to refactor. Martin Fowler's excellent book covers many of the code smells you might come across or inadvertently create yourself and how to fix them. The problem is that in a project situation it's all too easy to leave that refactoring in favour of more 'working' software until such time that making those changes is not easy. And if the requirements have changed, you might have created a super-smell. It's not your fault, and because you're doing super-communication too your customer is aware, but now you're going to be changing code and not adding to your working software base. I've seen whole iterations pass with nothing but refactoring going on.
Fact: Proper refactoring is a critical to delivering quality software. "It works" is absolutely not good enough beyond the first five minutes of use (see next fallacy)
Fallacy: Just Good Enough is good enough
It feels to me that IT people fall too easily into the binary trap of thinking that either:
A solution has to have big up-front design, requiring scholarly thoughts, academic arguments, fanciful pictures and an architecture blueprint that crumbles at the first sign of a change in requirements, or an undiscovered dependency
A solution completes just enough design to get the next bit of development started and worries about each problem only as it arises, in a solve-tomorrow's-problems-tomorrow style.
Why does it have to be one or the other? Option one, to me, means you have hired the wrong architects and option two means you have hired the wrong developers.
Fact: A design that is just good enough, will fall apart under continuous change just as easily as will one that's been over-thought. For a design to have flexibility built-in it requires up front thinking, beyond just today's immediate issues, sometimes though changes come out of left field and you can't cater for them. That's life. You do what you can.
Fallacy: Onsite Customer
It's great that software folk are getting into the idea that frequent and effective communication is crucial to success. It's rare you get the same from the business. That's not necessarily a criticism - there aren't that many 'customers' who have the time to dedicate themselves to one project.
It's also not often the case that there's one customer, and where there's more than one you can be sure there will be politics. That's what the Business Analyst role was created for - to elicit requirements, resolve conflicts, facilitate workshops, propose solutions and ultimately get enough solidity to go and build things.
Fact: customers are customers, usually, because they aren't IT. They are busy with their day jobs and therefore sometimes can't, or won't, be prepared to collocate with an IT delivery team. More often than not, one IT project needs to please multiple customers, who have conflicting aims and different opinions on what the priorities are.
Fallacy: Pair Programming
It's a well known fact that good programmers can be ten times more productive than average programmers and all programmers create their best stuff when they have enough peace and quiet to get in the zone so that they can hold vast amounts of context in their head at one time. That’s why quiet areas are good for coders and rodent farms of noisy cubicles are not. I also accept that regular code reviews are good for quality, and that pair programming represents the extreme practice aspect of this.
But everybody's different. In Myers Briggs terms I am an INTJ, which means I tend to internalise my thoughts rather than share them, my preference when faced with a problem to solve is to go away and think about it for a bit. I don't like thinking out loud very often. Pair programming, I suspect is better suited to people who are either complimentary in this respect (an Ixxx and an Exxx) or who are both comfortable externalising their thoughts. Others are more direct in their mistrust, whereas some see humour in the idea of two techies forced to work at one station.
Fact: Regular code reviews are good, mentoring is good, but the close cooperation required for pair programming is best dealt with on an individual by individual basis. It should not be policy.
Fallacy: Documentation
I shouldn't have to write this. The idea that there's little or no documentation in an agile project is not so much of a fallacy as a cop out for lazy developers. Of course documentation for its own sake is a pointless exercise.
If you can't say why you are writing something up and who's going to read it, you probably shouldn't do it, but designs, plans, release notes, budgets, requirements, installation and support documentation are necessary.
The directive is to do just enough not to create useless documentation, or worse, none at all.
Fact: Lots of people will need to interact with your project in indirect ways. They may not come to the party until quite late in the process. Targeted, terse, documentation is the best method we have of exchanging information when there's not enough of us to go around to explain things verbally.
Fallacy: Architecture
I get as mad as even the most gung ho developer when architects swan about occasionally releasing vague ivory-tower statements that are impractical, irrelevant, annoying and, often, plain wrong.
Like documentation, if architecture isn't relevant or useful you shouldn't do it. But pragmatic architecture is relevant and useful. In fact it's critical. If you don't have an overarching idea of how your project will fit together and fit with it wider environment, and meet its operational qualities and, within reason, meet the challenges of next year, then all you have is a tactical sticking plaster that will cost a lot of money to remove later.
Fact: Projects aren't successful just because they meet functional requirements. Other projects, later projects, strategic aims do need to be taken into account. Also see Fallacy 4.
Fallacy: You Ain't Gonna Need It
I like the YAGNI principle. It's good to help clear the mind and start developing the important requirements first. YAGNI says that if you don't need a feature today you shouldn't implement it. And that means "even if you're totally, totally, totally sure" you'll need that feature later.
And here's where I disagree. If I'm totally, totally, totally sure I will need something, then I'm going to build it. If I'm not sure, or I only have a hunch I might need it, then I'll leave it until I am totally, totally, totally sure. If I happen to be totally, totally, totally sure I need it, and I've just discovered that it's going to take twice as long to build as first estimated, then it would seem sensible to check with the business that they are still totally, totally, totally sure they need it and that there aren't any alternatives that might be quicker to build (in fact the benefit of short iterations is that you are frequently asking if everyone is still totally, totally, totally sure).
Fact: YAGNI today doesn't mean YAGNI tomorrow. Sometimes you are going to need it even if you think you don't. You ignore this at your peril.
In late 2006 I was at the DSDM-sponsored Agile Business Conference. Kent Beck presented and was asked at the end what he would change if he could go back and reinvent XP. Without a pause he replied that he would have focused more on the people side. He said that XP has always been about trying to find techniques that work at that level.
And that's the real message in all this. Whatever Agile is, software delivery works when you remember that you're only trying to make your customers happy. To do that you have to make your team happy and you have to constantly engage with all the other areas of the customer business that are involved (finance, programme management, procurement, operations, suppliers) and that's a lot of work, especially for the project manager. It's always been a lot of work; it's just that somewhere in the distant past we thought that we could find ways to avoid it by falling back to something called process. If the agile movement has reminded us that we also need to get out of our chairs and talk to people then that can only be a good thing.