Agile is the opposite of waterfall – no

The Iguazu Falls (healthy/agile)

The Department for Work and Pensions have written off millions of pounds spent on developing IT for Universal Credit and we expect the write-off to rise into the hundreds of millions.

How can we stop intelligent organisations from wasting money like this?

Over and over again we are told that the answer is "agile".

Use "agile" software engineering methods and the waste will be minimised.

How? What problem is "agile" solving?

Over and over again we are told that "agile" is to be contrasted with "waterfall". Waste is endemic in "waterfall" software engineering methods. That's the problem. And "agile" will solve it – that's the suggestion.

That's the suggestion made by ex-Guardian man Mike Bracken CBE, for example, when he was over in the US telling the Americans how to do government IT last October:

What is your reaction to HealthCare.gov and what you're reading and seeing regarding failures of what was meant to be an Expedia shopping for health coverage? Yeah ... I'll say this with no sense of enjoyment whatsoever, but it feels a bit like Groundhog Day to where we were three or four years ago. Hundreds of millions of dollars, large-scale IT enterprise technology, no real user testing, no real focus on end users, all done behind a black box, and not in an agile way but in a big waterfall way, which is a software methodology. And basically not proven good value, and I'm afraid to say I've got example after example in the U.K. in the past where we've had that experience. So it looks just like one of those.

As a further example, that's the suggestion in Richard Bacon MP and Christopher Hope's Conundrum: Why every government gets things wrong and what we can do about it, pp.240-1:

The traditional approach to software development is often known as 'waterfall' development: that is, you plan, build, test, review and then deploy, in a relentless cascade. But some IT industry players regard this practice as the chief problem ...A rather different answer which has emerged in the last ten to fifteen years has been what are called 'Agile Systems', perhaps best described as a philosophical movement in action within the software industry.

Niagara Falls, January 2014 (unhealthy/DWP)

The suggestion is nonsense. There's nothing wrong with the "waterfall" method. You can't get away from the "waterfall" method. All the "waterfall" method says is that you can't deploy a system until you've coded and tested it and you can't code and test it until you've designed it and you can't design it until you've analysed the requirements.

If you reject the "waterfall" method, you must believe that you should start designing a system before you know what's required and if you believe that then you're no use to the benefit claimants who need Universal Credit.

All that its proponents tell us about "agile" is that it's not "waterfall". But the "waterfall" method is right. It's the only method there is. It must be. There it is right bang in the middle of the "agile" method professed by ex-Guardian man Mike Bracken CBE's Government Digital Service – first you have a discovery phase, then an alpha and a beta, then you go live. That's a waterfall.

You can iterate, of course. You can have one release/deployment after another as the system is maintained and enhanced. But what you're iterating is a waterfall. There is no escape from the waterfall. And no need to escape from it. What would you be escaping from? From the belief that you must analyse before you design? But that's madness.

Madness won't solve DWP's problems. And those problems are not diagnosed simply by saying "waterfall" as though that's all bad and "agile", whatever it is, as though that's all good.

This distinction being touted between "agile" and "waterfall" is false.

----------

The picture of Mr Boehm's spiral has been added, as have the hyperlinks in the four citations at the end. but otherwise unchanged here's an extract from someone's essay written 12 years ago in February 2002 as part of his MSc in software engineering taken after 25 years of doing the job. For what it's worth:

Waterfall ... After that, it is pleasant to come back to the chirpy school magazine style of Requirenautics Quarterly [[1]]. There is not a megalomaniac or religious fanatic in sight and it is packed with sensible articles: a quick review by Ian Alexander of Barry Boehm's WinWin including a copy of every IT man's favourite cartoon; a long article by Ralph Young on how to help everyone; a good contrarian contribution by Richard Veryard in praise of scope creep; some helpful thoughts on abstraction and scenario-building by Ian Alexander; and a long series of book reviews contributed by the same man. One of the book reviews covers Ralph Young's book, Effective Requirements Practices. The review is witty, sensible, fair and acute but there is one sore thumb sticking out of it. Quite out of the blue, Mr Alexander writes (p.12): ... Something that they both agree on [Michael Jackson and Ralph Young] is that the waterfall model is inadequate: engineering development certainly does not follow a straight line path. Why does he say this? It adds nothing to his review. It is gratuitous cruelty. The waterfall model is like some unfortunate dog that no-one can pass without kicking. Some people will even go to the trouble of crossing the road just so that they can kick it: Mr Alexander, for example; and Messrs Bowen and Hinchey in their Ten Commandments of Formal Methods [[2]] – you would think that they had enough on their plate but, no (8th commandment): ... System development is by no means a straight-forward one-pass process. Royce's 'Waterfall' model of system development [[3]] was abandoned because of the simplistic view it held of system development .... People have been kicking it for years; kicking it seems to be an 11th commandment; but there's something funny about this dog – it's still there, it's always there, it doesn't matter how much you kick it, it just won't go away. Why? The waterfall method is supposed to be all wrong. How silly to think that a specification can ever be finished or that development can ever end! Well, yes, it is silly, but who ever thought that? The merest charwoman knows that she must dust over and over again, the dust keeps coming and she must keep dusting it away. Are we to suppose that once upon a time, in some dark age or some foreign country, users and system developers didn't realise this, unlike charwomen, and thought that requirements could be fixed finally and forever? Show me one of these system developers. Introduce me to one of these mythical users who would sign off a specification and then wait patiently and confidently for six years to have the perfect system delivered. I don't think they ever existed. Attacks on the waterfall method are attacks on an Aunt Sally. "What about Government contracts, then", you may ask? "They always insist on quoting a fixed fee for a fixed specification." Do they? Show me. I'd be interested. I've never seen a government contract. Perhaps they are as stupid as you suggest. Silly old government. But perhaps they aren't. Perhaps the contracts do allow for change. If you attack the waterfall method, does that mean that you think that you shouldn't analyse and design before you start coding? No, it is generally recognised that actually that is a rather sensible order to perform these tasks in. You can play games with the topology, of course: ·  You can say that the evolutionary method is better than the waterfall method. But all you've done then is to string out system development into a long series of ... waterfall cycles, each iteration with a bit of requirements capture followed by a bit of design followed by a bit of implementation and, yes, we'd probably better have a spot of acceptance testing before going into production. You're still using the waterfall method and, frankly, it would be surprising if you weren't. ·  Alternatively, if you really can't stand straight lines, spirals may be more your bag, with quick access from one iteration to the next, optionally cutting out a few phases of the waterfall cycle. But you're still actually acknowledging that the waterfall model is there, wrapped up in one of Mr Boehm's spirals. You can't get away from it and there is no reason to try. What does this achieve? It reinforces the point that all the processes involved in system development have to be performed iteratively and that progress is incremental. That should always have been clear anyway. It doesn't add anything to the original model and it doesn't make the original model wrong. So, I think that it may be time now for us to stop kicking the poor old dog. We should give it a shower, start feeding it properly and take it out for walks with us. The waterfall model should be treated as the long-suffering, worthy and faithful member of the software engineering family that it has always been. This dog deserves our warm regard and respect. After 30 years of this sort of vilification Mr Royce, the dog's breeder, has probably suffered his own Darkness At Noon [[4]] and brain-washed himself into believing that the waterfall model is a legitimate target for any passing boot. He should be rehabilitated. [1] Requirenautics Quarterly, BCS RESG, Issue 24, July 2001. [2] Bowen, Jonathan P. & Hinchey, Michael G., Ten Commandments of Formal Methods. [3] Royce, W.W., "Managing the Development of Large Software Systems", Proc. WESTCON'70, August 1970. [4] Koestler, Arthur, Darkness At Noon, Jonathon Cape, 1940. __________________________________________________________________