Is Agile Killing the Architect?

is agile killing the architect?

The formally recognised discipline of software architecture is 20 years old but it’s also a bit of a conundrum: opinions about the questionable value architects add to projects are strongly held and yet job vacancies for architects are as abundant as ever. So which is it? Is architecture in the enterprise misunderstood? or is it living on borrowed time?

On 19th May, La Fosse Associates hosted an event at Runway East in London centred on a contentious question:

Is the adoption of agile methods in large businesses killing off the traditional role of the IT architect?

Unsurprisingly, there were some strong views on both sides of the argument and some nuanced discussions around what exactly it means to be agile in a big enterprise company. And, for that matter, what it means to be an architect.

Is part of the problem not the role but the grandiose associations that comes with the title?

In the last week, around a thousand IT architect vacancies, with various specialisms from enterprise, to business, to data, to security and infrastructure, appeared on Internet job boards. That’s a very nice statistic to focus on if you’re an architect. But is it a useful measure of the health of architecture roles in large enterprises?

One constant refrain from the boardroom is the need to modernise software delivery to become more agile, to address the threat from smaller competitors, who are able to release new, high quality code into production at ever increasing speeds, whilst spending only a small fraction of the budgets enterprises have to muster to get anything done. And many of these niftier companies don’t have architects, not even one in an oversight role.

Where small high-growth companies are shedding traditional roles, enterprises seem to be creating new ones. The rise of the Chief Digital Officer being one of many signs that the effectiveness of the corporate CIO isn’t all it should be.

Add to this picture a view held by many delivery and product owners that architects spend a lot of time drawing nice pictures and talking a good talk, whilst bring very little in the way of tangible value to the table, and you might find yourself wondering what kind of longevity those thousand architects can expect in their new roles.

And there’s plenty of precedence for roles disappearing from the market quickly. The number of DBA roles has halved in the last ten years according to IT Jobs Watch. It used to be that every software project had one (or had to deal with one) and then NOSQL databases and agile practices meant the responsibilities were incorporated into just another software development specialism. Database administration as a skill hasn’t gone away, but many companies do not see that skill as distinct enough to warrant the overhead of recruiting an entirely separate person to perform it.

So, could architecture go the same way? All the ingredients are there: the role is hard to define, there is a perceived lack of value in some quarters and individuals are expensive to hire and train just at a time when there’s an increased sensitivity to enterprise bloat.

Ignoring the controversial role title for a moment, perhaps the key question is this:

is there a need for structural software design work to be done above, around and outside projects of a kind that’s valuable and therefore not a dying?

Any delivery organisation is really just a group of people performing tasks. In an optimised structure, those tasks are well coordinated between individuals.

Movements like Programmer Anarchy (essentially a hyper-distilled version of agile) lean towards folding most of those tasks into the software developer role, thus negating the need for business analysts, scrum masters, and even QAs.

The opposite point of view, seen in very large enterprise programmes, is that tasks are more widely distributed across people, increasing the number of roles and the lines of communication (creating a negative network effect). Clearly, large teams, and the associated need for endless meetings, are an inefficient way to get things done. On the other hand if developers have to do everything then they won’t have much time left for coding, which is what they come to work for in the first place.

An architecture team is nothing special - it too needs to justify the value of task specialisation to the degree that a role performed by a separate person is necessary. It also needs to be able to show that this role can execute these tasks in such a way that the software team, and the organisation as a whole, is better off than it would otherwise be. That is to say it needs to assist progress in answering the three eternal questions of delivery: What do we need to do? How are we going to do it? What shall we do next?

A fact that should be obvious, but isn’t always, is that in an unpredictable business world, with high levels of software complexity and choices to be made, nobody could possibly know the answers to these questions ahead of time. Competitive advantage and developer effectiveness comes from choosing just the right technology for the job. Big enough that it does the job, small enough that it doesn’t impose on the flexibility of the developers, simple enough that it’s easy to test. Anyone that’s had to sit through preordained architecture standards, governance boards, gates and bureaucratic processes knows that they certainly do presume to know the answers ahead of time.

There’s simply no requirement for policing in modern software development. If there is a relationship between rules and speed and quality, it’s that the more rules there are, the slower teams go, and therefore the higher the risk that corners will be cut when the project is late.

Policing assumes that architects know better than delivery teams and yet, even when it comes to design debt, they can’t possibly have the context to know what the trade offs are. Intentional design debt is actually a good thing (i.e. debt that is both prudent and deliberate) according to the Technical Debt Quadrant.

So without governance processes and standards to police, is the distinct role of architect disappearing? Architects themselves might complain, but their opinions in this matter are mostly diversionary. There’s a subtle irony in the fact that, to design this optimised organisation, with or without architects, you can’t actually ask an architect to help. The need for architecture has to be clear from the delivery process itself. Architects have a conflict of interest in designing an organisation that might not need them.

But perhaps the notion of conflict of interest itself holds the answer.

Conflicts exist throughout the delivery process. For example, if a product team delivers a CRM system, who owns it? Sales? Marketing? Customer Support? If all of them want changes, who decides which ones go first? In fact who decided that a CRM system was required in the first place? Not the product team because they only exist after delivery is funded. It’s something of a self-referential paradox. In order to design an organisation without architects, one has to design the organisation as the architect. Someone needs to decide where new capability is built. And once delivery is underway, they need to support the team in making good decisions about how to build it. These decisions will sometimes challenge and undo assumptions made before the team existed. That’s a good and healthy thing.

The process of laying out the landscape and constantly sense checking that it’s still valid is not unlike programming. It’s what Frank DeRemer and Hans Kron in their 1975 paper called “programming in the large.” It can also be entirely compatible with agile if it’s done in such a way as to minimise upfront (wasteful) activities. One upfront wasteful activity is letting suppliers or large software packages dominate the direction taken. That’s presumption of the most expensive kind and is usually impossible to reverse, making it one of the reasons some architecture teams look limp and ineffectual in agile environments.

So is “programming in the large” worthy of being referred to as “architecture”? Maybe, maybe not. What enterprises call architecture is really a support function for projects, to help them answer their own questions and to leave behind something that doesn’t come with operational regret.

It’s less about oversight than it is about insight, i.e. providing an extra pair of eyes so that software developers can develop with full context and attend fewer meetings. No doubt some individuals in this project support function could be worthy of the title architect but, if it’s not to die a slow and ignominious death by devaluation, it’s a designation to be earned not automatically assigned.

You can find the takeaways from the event on the LaFosse web site.

Notes

• The subtitle is, of course, a quote from Monty Python

• This article was originally posted on LinkedIn as part of the after-event follow up. Due to popular demand, the event was run again on 5th October. The takeaways from the first one are here.

• I made the ‘case for the defence’ on the night, one the one hand acknowledging the very strong (and frankly justified) anti-architect views that exist in the developer community whilst also trying to make the point that I’ve certainly managed to play a value-adding role in companies which is architectural in nature. I don’t much care what the role is called but I’ve seen the effectiveness of development teams and transformation efforts supercharged when it’s done well.

• The slides are on SpeakerDeck