The Retep Principle

Crossing the chasm

By Julian Browne on March 11, 2008. Filed Under business

My first job after university was as a unix performance consultant working on now-defunct product called Officepower for a now defunct company called ICL. Being largely inexperienced on day one, I had an intellectual mountain to climb to be effective in the role. But persevere I did. I printed out every page of the unix online manual, read every book I could find on the operating system internals, learned to program in C, got to grips with scheduling, partitioning, subnets and all manner of arcane concepts so that when presented to a customer I would be comfortable in solving their office automation issues.

It was tough, but I loved it - there's something inordinately satisfying about sitting in front of a green-screen display, late at night, in a basement, methodically working through problems and knowing that, by the time that last slice of pizza has been eaten, you'll be well on the way to enabling your widget-making customer to make widgets just a little bit more efficiently.

The principles I learned then have influenced or informed nearly everything I have done since. The unix shell is a domain-specific language whose components can be chained together to provide quite sophisticated functionality in much the same way as SOA promises to do (albeit without contracts and discovery).

I can't say in the early days I was ever really all that aware of 'projects'. My job meant that I was either operating in a pre-sales capacity, never really knowing whether bids were won or not, or I would turn up at an annoyed customer's site to fix a problem after a project had delivered. If I wasn't doing either of these I would be working on something in a lab to support product development.

What my job lacked was visibility and projects were where you were visible. Visibility meant promotion, and promotion meant bigger jobs, more control and more money. I got plenty of recognition from customers, but it didn't take long for me to hanker for something more fulfilling. There's only so much tracing, restructuring and fixing you can do before you start to ask why the problem arose in the first place, and playing the role of mute-geek, second-fiddle, to the sunbed-tan salesman gets old even faster.

Maybe it was youthful impatience, but I found I quickly developed a simmering resentment to my lack of progress. Here I was making bad systems work, helping win new business, and fast developing a reputation as a font of knowledge in many software related areas. I would look around and see others caring less, doing less, and yet getting more because of some aspect of their role, or personality, that I didn't have.

As I scanned the org chart it seemed that people very different to me were sitting in the higher-up positions. In fact the higher up I looked, the greater the difference was.

And it was about that time that I first heard of the Peter Principle - the adage that each person is promoted until they reach their particular level of incompetence. So I consoled myself with the notion that everyone was an incompetent, except me. After all, there's an undeniably neat logic to it: you do a job well, you get promoted. That's the natural order. Eventually you won't do so well and you won't get promoted. And there you'll be. Stuck doing a bad job.

I eventually left and got a job where I could work on proper projects, and write software, and get the visibility that I wanted, but a few things about the Peter Principle continued to bother me.

Firstly, I wasn't getting promoted. The obvious response to that is that I had already reached my level of incompetence. A scary thought, but one I could discount because of the sheer volume of success I seemed to be having.

Secondly, incompetent people (at least according to my inexperienced jellybean definition as someone who knows a lot, does a lot, and optionally cares a lot) were getting promoted. Not just my cohorts, but all the way to the top.

And thirdly, there were plenty of cases where idiosyncratic competent techno-gurus had achieved super-recognition: Gates & Allen, Jobs & Wozniak, Bricklin & Frankston, McCarthy, Dijkstra, Thompson & Ritchie.

My conclusion, many years later, is that whilst the Peter Principle certainly exists, it's counteracted in technology fields by the Retep Principle, a force that has precisely the opposite affect.

Retep Principle

The basic principle goes like this: the chances of promotion of an individual in a technological role is inversely proportional to the level of their skill and focus in doing that role.

The Retep Principle is made up of three factors:

• Irreplaceably useful

  Every decent manager knows that having even one good programmer in your team, who is productive and diligent, can make or break a project. It's often said that a good programmer can be ten times as effective as an average one (although in my experience that is quite a conservative measure). Let's say you pay that programmer 50k per year, and the average programmers get 30k. If you promote your star coder away from direct programming, you will either need to find a like-for-like replacement on the open market (and given the vagaries of the hiring process, this is not easy) or hire 300k worth of extra heads just to stay still, which of course wouldn't work even if you could afford it, because with 10 new people all that extra communication overhead would probably kill your next project.

• Competence vs. Attitude

  And there's another feature that makes promotion difficult. Eric Raymond, in his essay How To Become A Hacker, said "learn to distrust attitude and respect competence of every kind", meaning that proper hackers respect knowledge, concentration and problem-solving skills of the highest order over anything that smells like attitude ("hackers won't let posers waste their time").

  Now I'm not about to defend posers, nor would I disagree that competence is everything when you are actually writing software, but software is not written in a bubble, and career changing promotions beyond writing software clearly depend on different techniques and skills. I've worked with individuals who were exceptional at delivering software projects or managing technical teams, but who would think the singleton pattern must be a way of explaining why some programmers don't get married. What they could do though is explain, cajole, communicate and generally make good decisions (or know who they could use to make good decisions).

  Explaining things, for example, is a vastly underrated skill. There's nothing in technology so complicated that you shouldn't be able to break it down into something basic enough to discuss with your grandmother.

• Truth to Power

  There's a great phrase used in American politics called truth to power that is used to point to why bad decisions get made (i.e. because nobody was brave enough to speak the truth to the people with the power). If you're in the school of competence over attitude, then you're more than likely to also think speaking truth to power is a good thing. After all it's only the truth and the truth is what you need to make good decisions, right?

  Well, maybe not so much. Software people can be an idealistic bunch and, particularly at the code level, they deal in very black and white (binary) concepts so often that it can dominate their world-view a bit too much. You only need to spend an afternoon on any of the popular software and architecture discussion sites to see what a dogmatic lot we can be.

  The fact is your harsh truth is not always what's needed all the time. It's called politics, and many times it's not what you say, but when and how you say it, that matters. I've sat and listened to developers almost on the verge of tears because so-and-so project manager or business sponsor won't listen to them, and the project is doomed, and it will never work. And sometimes they are right. They've just mistaken their truth for the politic truth. Software, and especially the wider enterprises that make use of it, can be pluralistic universes. There's no point telling the Finance Director that JEE is overkill when Spring would work, they don't care. Their truth is couched in different terms.

Beating the Retep

If you want to stay a hacker then, honestly, you've just wasted your time when you could have been reading something much more apposite like an API spec. But wanting to remain one is a noble pursuit and you will have more direct fulfilment than many in doing so.

If you want to try something else, then you have to accept that whilst technical competence is valuable to you, you need to guard against being too valuable to others. Being irreplaceable in your manager's eyes is not an easy thing to reverse. The saying dress for the job you want, not the one you have can be adapted to behave for the job you want, not the one you have too.

My years cutting code and tuning kernel parameters is still an invaluable asset in most things I do, but I can't look to it for advancement any more than I can my studies of eighteenth dynasty ancient Egypt. If I don't succeed as I would like at something these days, rather than fret about it, as I may have done in my impetuous youth, I have to look first at attitude not knowledge.

And when a project isn't going as I think it should I, most of the time, keep my mouth shut and find ways to influence it rather than bring in a copy of Ed Yourdon's Death March and talk about the last project that I saw fail that way. I say most of the time because sometimes it's my can to carry and using influence is too meek an approach to solve the issue.

And let's face it, if you're smart enough to master software development, you should be able to turn your hand to anything.