The Grand High Order of the Easter Bunny

I’ve talked, rather a lot, about my views on the use of open source software in the big fat corporate enterprise. Because these rambles have been at conferences or in occasional blog posts, the pitch has been aimed at enterprise developers:

Use open-source where you can, but do not think of it simply as a “free” resource; give something back to community through promoting it, patching it, extending it and paying for support.

Everyone wins.

The argument is partly idealistic (make software free), partly ethical (respect others work and don’t abuse their efforts) and partly financial (exchange of labour is cheaper than the hefty license cost of commercial software).

But there’s another reason. One that I’ve thought about in abstract terms a lot, but never really been unable to explain. There’s something that using open source software can do to an organisation that’s only indirectly related to it’s freeness or the openness of its source code. Something that manifests itself both politically and in an organisation’s ability to deliver.

It’s worth exploring because it tells us something about the common mistakes big companies continue to make when they pick software packages. And by thinking about these processes differently I think it’s possible to mitigate many of the risks, even when the software’s not open source.

When I was writing The New New Tool I was thinking about the many Request For Information (RFI) and Request For Proposal (RFP) processes I have been involved in and how they all start out with good intentions and how they all, in big enterprises, end up making really bad expensive product choices. This happens for many reasons: people involved not technical enough, insufficient due diligence, real users not consulted, political interference, trying to solve a problem that doesn’t exist, I could go on.

RFI and RFP processes are an insurance policy against spending company money on The Wrong Thing. And The Wrong Thing isn’t just, or even mostly, about functional fit, it includes the prevention of fraudulent activity, such as buying software at an over-inflated price from your mate Dave down the pub. They are open and transparent processes designed to make sure everything is auditable, fair, and above board. It’s for that reason I assume that they’re never enacted for selecting open source products - Dave may be the main committer to your chosen repo, but he won’t be charging you a few million quid to access it. The worst that could happen is you’d commit yourself to some painful extra development hours sorting out Dave’s mess but that’s operational expenditure, not capital investment, and procurement don’t care so much about that (unless Dave starts a lucrative consultancy to help you sort the problem out).

So, for the majority of product selections, we rely on an RFP process that begins with a short list of expensive tools and, through an open process, chooses one that’s a poor fit for the organisation’s needs. And poor old Bob and Alice in The New New Tool story end up having to work with it.

RFP exercises are time consuming, and not a little boring. Lots of different people have to be involved. There are a lot of meetings and presentations. Not much of that time is spent knee-deep in technical due diligence for good reasons - you can’t get source code from commercial vendors so any assessment of technical quality has to done a step or two removed. A bit like buying a car without being able to drive it. Sometimes it’s a bit like buying a car from brochure shots only. And when you do finally make a decision your problems have only just begun.

The selected product will be expensive. They always are. RFPs aren’t run for buying text editors. And by expensive I mean hundreds of thousands, maybe millions, for a license. This cost brings with it a sense of reverence. People must go on a training course to know how to use the product. These training courses are also expensive, meaning only a select few can take them. You might even need to hire a team of Delphic experts who have already done the training. Only the select indoctrinated few can manage the product. The product requires expensive hardware to run on so it sits in a data centre somewhere far away. Now nobody can see the product running. Like that final scene at the end of Raiders of the Lost Ark it becomes a much talked about, almost mythical, object sitting in a nondescript box alongside many other nondescript boxes. Legends are created about its genesis.

Once it’s finally in production (many months and project managers later) the product can’t simply be activated all willy nilly. No sir. A service brought into being by such divination requires policy and a process. These must be meditated on at length, then ornately crafted and written down so that they might be ministered by the trained ones.

The end result is like some butt-crazy Knights Templar conspiracy theory, except it’s true. And what’s even more odd is the fact that The Product is protected by The Select Few and The Process and housed in The Faraway Place is not even considered unusual. That’s just how big businesses roll. Anyone who questions this is seen as ‘difficult’.

In season eleven, episode five, of South Park, Stan asks his Dad why on earth people paint eggs for Easter. Like what, exactly, has that got to do with the Christian story? In his subsequent investigations he uncovers a secret society - The Hare Club for Men - who protect a rabbit descended from the original St Peter (Peter Rabbit) who is the right and true Pope.

I bet there were a lot of people who watched that thinking it was just a slightly more surreal version of their day job.

And why is open source different?

Because it’s much easier to make it pervasive and ubiquitous. It’s free to access, so anyone can get it. You can buy training courses for the more popular packages if you want to, but that’s not a barrier to learning about them. You can install most packages anywhere and if it doesn’t install on your chosen platform you can, if you have the chops and the time, build your own version. Once some helpful soul has done this, even if it’s a bit ropey, it can be hosted somewhere and improved. Tools that are everywhere like this can’t hide their faults. They can’t be protected by The Select Few in the enterprise. The Select Few are nearly always non-technical managerial types anyway so in good meritocratic fashion, experts can challenge anyone who seeks to create power based on sand.

I came across a post the other day talking about how 10gen’s approach to bringing MongoDB to market has been so successful. It’s true. I’ve been using MongoDB for a couple of years now and 10gen’s curation of the open source package has a lot to do with why I’m working with it. They made it easy to get, easy to get started with, and easy to learn about. It’s not the only NoSQL database on the market by a long stretch. For some uses cases it’s obviously not going to be the best fit either. And yet MongoDB is almost synonymous with NoSQL in big enterprises. That’s both a testament to some great management, for 10gen, and a real shame for the others. But, a bit like Microsoft’s early policy of pushing for horizontal markets for their products (begrudgingly noted in F. W. van Wensveen’s “Why I Hate Microsoft”) these are the policies that work.

And that’s the sense I had but found hard to put into words. It’s simple.

When you look for a new product, make sure it can be woven into the very fabric of the way you want to work. Make sure everyone who even shows an interest can get a copy and use it. Make sure it’s free to do that. Vendors who don’t want their software loose all over your estate should be viewed with some suspicion. Mandatory training courses, chaperoned demonstrations, guided tours, hard to reach support consultants, all smell to me like bad software.

I like open source because it goes further than even that and makes it’s very DNA visible to look at. For better or worse, it’s honest. Sometimes that’s really useful when you’re working on a problem, but it’s surprising how few the occasions are when the openness of the source code is important, which makes me think a lot of these lessons can be applied to closed-source packages. Because no software should ever hide behind protected processes and selected people, even during the sales process. And until companies stand up and demand that kind of transparency nobody should be complaining that in order to get ordinary shit done you have to deal with the arcane processes of the unaccountable Grand High Order of the Easter Bunny.

Notes

• The subtitle is from the South Park episode mentioned in the article

• The picture is from the original illustrations of Alice in Wonderland