Embracing the Third Way

I had the honour recently to be invited to sit on the judging panel of the OpenSpaces Developer Challenge, an international competition to encourage the creation and open-source sharing of innovative ideas in grid, cloud computing and next-generation middleware, based around the Gigaspaces eXtreme Application Platform. If the objective of the organisers was to illustrate just how diverse the uses are for this kind of technology it certainly succeeded in that respect.

Unfortunately, for the judges, that meant facing the unenviable task of choosing one winner from fourteen finalists, with submissions ranging from a BPEL engine, an automated test framework, a grid configurator, a persistence model, to a full PHP space-connected application. I have to say though, it was personally satisfying to see some functional programming concepts coming though in a couple of the entries, notably in the clever use of Ruby and JSON. Last year I wrote a Rails plug-in for Gigaspaces and there were moments when I wished I could be on the other side of the table joining in the fun, rather than having to undertake the serious task of critiquing the entries. Still, there’s always next year.

And a serious task it was. The judges agreed on a reasonably sophisticated criteria/weighting system to score against, there were video presentations, and then the top five projects had a thorough code review, which was followed by more emails and a final phone conference. The worthy eventual winner was GoDo, a goods donation system written by Leonardo Goncalves, nicely combining GWT, JSON and a space-based architecture in addition to having a decidedly altruistic objective.

The judging debate was fascinating. There’s a jokey comment on the openspaces site saying the judges “were not always in agreement with one another” but in fact it was all very good natured, and I for one was surprised at how much consensus there was, given we were all very different individuals. Possibly even more surprising was that despite vastly different perspectives (after a slightly mercenary comment from me, for example, we were duly reminded that “value to corporate environments” was not what the competition was about) even the scoring was pretty consistent. If there was a strong difference of opinion, it was over which of Java One, TSSJS or OSCON was the best place to announce the results (proximity to judges home, and consequent likelihood of hospitality beers, being the prime consideration here, naturally).

The experience sparked off a train of thought that’s been with me ever since. One that throws into sharp relief a perceived dichotomy of modern software. Or rather it suggests that attempts to divide the software world into two camps creates a classic false dichotomy. I am talking of open-source vs. commercial software. Not the argument about which is “better” - that’s one that’s been made for both sides far more elegantly than I could ever attempt. Not that I believe the question of betterness is even relevant - packages like Rails would not exist if it were not for the open source movement, and many corporate environments have neither the patience nor the will to do anything other than pay top dollar for solutions that precisely meet their immediate requirements or to purchase COTS packages to fulfil their commodity needs.

However, it seems to me that innovation comes largely (though not entirely) from the open-source world whereas commodity needs are met largely (though not entirely) by commercial software. Where communities create packages such as Ruby-on-Rails, the commercial world comes up with Dreamweaver. A comparison in terms of betterness is nonsensical. Not just because Rails is a framework which promotes convention over configuration, and Dreamweaver is a site and page editor (i.e. you could use both together) but because, without open-source existing, we’d just get better and richer versions of Dreamweaver until the product bloated to the size of a Microsoft Office application and a smaller, nimbler, tiny bit more innovative, newcomer appeared. Adobe would buy them out and we’d start the cycle again.

Open-source disrupts that cycle by having the freedom to ask “what can I be?” rather than “what specific requirement do you need met right now?”. Put another way, Dreamweaver solves a vertical requirement, whereas Rails takes on an entire horizontal aspect of web development. Of course Adobe could have come up with something like Rails but, while developing it, competing products like FrontPage might steal market share. There’s also the risk that nobody would want a framework which may be seen as proprietary. I don’t meant to pick on Adobe, clearly with products like AIR, which I quite like, they are in the framework business too.

The owners and committers of open-source software experience the double edged sword of having greater freedom on the one hand, but also the lack of a demanding/immediate user base steering the product on the other. There are exceptions to this, notably products like Firefox, but for the vast majority without users or awareness, many open source products wither and die, making them, curiously, a better model of survival of the fittest than commercial software (which can always be pushed beyond it’s useful date by some clever marketing).

This commercial vs. open-source conundrum is perfectly illustrated by the rise and fall of the Chandler project, the story of which is told with great sensitivity and insight by Scott Rosenberg in his book Dreaming in Code. Chandler was (is) the brainchild of Mitch Kapor, designer of Lotus Notes, squillionaire, interesting blogger and all round nice chap (he allowed pet dogs in the office, which automatically makes him a great guy in my book). Intended as a Microsoft Outlook Killer, utilising peer-to-peer technology (at least in the early days) and written largely in Python, it had innovation written all over it. I (along with, I’m sure, countless others) waited with great excitement for the day when I could ditch my bloated email/calendar managers and switch to Chandler. That day never came. To say Chandler failed would be too harsh. Others have certainly made this statement but I feel too much for the people who worked on it to be so blunt. It may not have toppled Outlook, but I’m pretty confident that much of the experience and knowledge gained during the development process is being successfully and creatively used in other places right now.

What Chandler didn’t have, in the way that Outlook has in abundance, is user-driven requirements. Microsoft’s product enhancements are relentless. I hate Outlook but I have to admit it does the job. It just also does a lot of other jobs I don’t need it to. But in the corporate world, email and calendar management is a commodity. Nobody won market share buy booking meetings faster (now that I think about it there may be an argument to improve competitive advantage by making it harder to book meetings..). Corporate communications are simple - you have requirements, Microsoft meets them, and you’re happy to pay. You don’t want innovation in email. It wasn’t that Chandler had no user requirements, it was that they were trumped by the vision of innovation.

So where does the open model fit? I mean there aren’t many companies who don’t want innovation anywhere, just perhaps not in email and diary management. The bewildering range of licensing models for software has perhaps more than anything shown the bifurcated view to be a fallacy. From hard-core commercial, to completely free, shareware, free as in speech, free as in beer, free-but-credit-me, free for you but not your company, pay if you like it, pay if you feel like it, pay and you also get the source code. The situation is far from clear. There’s software with source code you can access, some where you can access part of it, and some where you can’t, but these don’t map one-to-one with payment structures. No wonder people like Richard Stallman get cranky. It seems like whatever anyone believes the rights and wrongs of software distribution, use, access and redistribution are, the reality is a bit of mess.

My mercenary, corporate, view on this is pretty simple. Coders need money to pay their bills. Companies need commodity code. The world needs creativity and innovation. There’s no conflict in these facts. What they say to me is that coders have the choice as to whether they provide commodity code to companies (boring, predictable, usually pays well) or innovation to a wider audience (interesting, more risky, rewards might be something other than financial) or both. Nothing wrong with both. And certainly the evidence is that people within open-source communities these days have a fairly healthy attitude to mixing progressive techniques and earning a living. Where I don’t see so much openness is in the commercial world itself.

I started my career in the kind of world that open-source thrives in. The first proper IT book I ever read was Goodheart and Cox’s “The Magic Garden Explained” a fairly scary text on the internals of Unix System V Release 4. Subnets, IRC, VI and a general eschewance of all things Microsoft (ideally with a $ where the S should be) were the hallmarks of the true believers. Heck, until not so long ago I used to pick people up if they used the word internet when they meant the web. I ran a dial-up bbs, gophered, wrote complex C code with linked lists and pointer arithmetic and freely shared anything that was useful. By rights I should be an out-and-out open-source freak. But here’s the thing. Somewhere along the way I found that just writing software wasn’t for me. And the reason wasn’t the code, but the users. Making all that complexity mean something became more interesting than the nuances of the complexity itself.

Finding meaning in software entails an acceptance that a lot of normal people hate it. They see IT as a clumsy and inept money pit, sucking in profit and delivering systems that get in the way of what they really want to do. I might enjoy the thought-provoking writings of Richard Stallman on why there is no such thing as intellectual property but users don’t. They want to pay for products that deliver email. Not because they want to hand over money, but because payment necessitates a contract, and with a contract they can vent feelings of frustration through it. Is it then strange that the provider under that contract will focus on requirements, not innovation, and protect the source of that income through something called intellectual property? Of course not. You can argue that the situation isn’t morally right, or stifles creativity, and you’d be right, but that doesn’t change how users feel.

Changing how software is perceived won’t come from the creators of software. It will have to come from the organisations that use it, via a new breed of IT leadership. One that judiciously applies innovative software where it can make a real impact whilst accepting that some boring, predictable needs are better met with boring, predictable, commercial applications. Applying innovative software doesn’t mean hopping on the open-source bandwagon and extracting financial advantage from the honourable intentions of the open-source community. It means using it and actively participating in it, putting back at least what you take out. That gives those communities a similar advantage to the one that commercial software has.

Companies then have three options open to them: commercial paid-for software, in-house developed software and open-source. Open-source and commercial software won’t negate the need for in-house because there will always be operational techniques and know-how developed within companies that they don’t want to share, or have built via a third-party they maybe can’t trust. Where companies seek competitive advantage, they should never limit their choices to the oversimplified buy vs. build.

There are plenty of articles around suggesting that the new breed of CIO/CTO should be more business-savvy and less of a technologist. I can see why this might be desirable but I worry about the “less of a technologist” aspect. Of course the business wants IT leaders that understand their predicament, but surely they hire these leaders to address this predicament with technology and solutions. So I’d say business-savvy, yes, but don’t sacrifice the technology specialism, otherwise the business will have to get advice on innovation from outside.

The fact is that all forms of software creation, licensing, and sharing and all forms of commodity and innovative models can coexist and thrive if budget-holding IT leaders embrace their technologist roots and accept the responsibility to wield their knowledge with a duty of care through three simple steps:

• Research

  There are many sacrifices one has to unconsciously make in terms of hands-on knowledge lost, or never gained, in living the corporate life. You just can’t be an expert in everything all the time. But you can make a point of reading a few academic articles each week, being at least aware of what the leading edge looks like, and maybe even coding a little something from time to time, just to get the feel for where all that applied strategy ends up.

  It’s important to constantly grapple with the fundamentals of what software actually is. Why? Well because all leaps forward tend to be in closing the gap between requirements and code. MDA, DDD, SOA all have their roots in ideas decades old, but they seek the same thing: to encapsulate a model of how the business really works into code that faithfully represents it and changes easily with it - the so-called supple architecture.

  It’s easier said than done. Many’s the weekend I’ll sit with a pile of research papers, knotted forehead, brain feeling like Udi Dahan’s Concept Map, trickle of blood seeping from each ear, and nobody to talk to except my three-year-old son, who when asked whether the concept of Fluid AOP, where one component might be exercised within the context of different aspects, depending on circumstances, might, in some sense, be analogous to the execution of a closure which, despite its local bindings, is in fact still a dynamic function, running in a chosen context. To which he, quite rightly, asks whether the fact that Davros is king of the Daleks, and therefore analogous to a super villain, makes Dr Who a superhero. The world is indeed a complex place.

• Understand

  Although the research effort is time consuming, and mind-achingly difficult sometimes, it’s surprising how many seemingly arcane ideas do break down to similar, simple, and overlapping ideas. Armed with this sense of knowledge and purpose, it’s only a minor challenge to see where to be leading-edge/innovative, where to be strategic, operational and just keep the lights on. Techniques like the McFarlan Matrix are great for categorising this.

  Like any software, there’s more to it than install-and-go. Commercial software comes with the requirement to engage in contractual negotiations and service-level agreements, in-house software comes with a requirement to build an effective change and support structure and open-source should similarly be seen as coming with a requirement to accept and respect the culture from which it comes.

  This step is crucial, and absolutely the responsibility of IT management, because as Geoffrey Moore said in Crossing the Chasm innovative ideas, of the sort that move things forward in a discontinuous way, can only be sold to the technologists who will understand them. I think this is why the competition started me thinking about this, because grid concepts are entering common business parlance, and getting confused in the process. Lots of vendors are noticing this but it hasn’t yet crossed the chasm. Clever IT leaders can make much of this by researching and understanding the facts before any money is spent and decide which model best suits their needs.

• Execute

  Then, as always, it has to be a business call as to how the potential rewards from innovation stack up against the risks in not having service contracts and lawyers to fall back on. It’s surprising how frequently the contract-driven approach wins through, despite the fact that commercial suppliers are rarely taken to task when they don’t deliver.

So where do we end up? With individuals free to choose to distribute the fruits of their labours in the way they see fit and taking the risks associated with their choice. With corporate entities choosing to meet their needs as they see fit, but understanding that all sources come with an obligation. With innovation part-sponsored, but not controlled, by corporates who wish to share and contribute to the works of others.

And with IT leaders understanding technology options, and applying them wisely, supporting vibrant commercial and open models for those that stand on their own merits.