Agile is the opposite of waterfall – no

The Iguazu Falls (healthy/agile)

The Department for Work and Pensions have written off millions of pounds spent on developing IT for Universal Credit and we expect the write-off to rise into the hundreds of millions.

How can we stop intelligent organisations from wasting money like this?

Over and over again we are told that the answer is "agile".

Use "agile" software engineering methods and the waste will be minimised.

How? What problem is "agile" solving?

Over and over again we are told that "agile" is to be contrasted with "waterfall". Waste is endemic in "waterfall" software engineering methods. That's the problem. And "agile" will solve it – that's the suggestion.

That's the suggestion made by ex-Guardian man Mike Bracken CBE, for example, when he was over in the US telling the Americans how to do government IT last October:

What is your reaction to HealthCare.gov and what you're reading and seeing regarding failures of what was meant to be an Expedia shopping for health coverage? Yeah ... I'll say this with no sense of enjoyment whatsoever, but it feels a bit like Groundhog Day to where we were three or four years ago. Hundreds of millions of dollars, large-scale IT enterprise technology, no real user testing, no real focus on end users, all done behind a black box, and not in an agile way but in a big waterfall way, which is a software methodology. And basically not proven good value, and I'm afraid to say I've got example after example in the U.K. in the past where we've had that experience. So it looks just like one of those.

As a further example, that's the suggestion in Richard Bacon MP and Christopher Hope's Conundrum: Why every government gets things wrong and what we can do about it, pp.240-1:

The traditional approach to software development is often known as 'waterfall' development: that is, you plan, build, test, review and then deploy, in a relentless cascade. But some IT industry players regard this practice as the chief problem ...A rather different answer which has emerged in the last ten to fifteen years has been what are called 'Agile Systems', perhaps best described as a philosophical movement in action within the software industry.

Niagara Falls, January 2014 (unhealthy/DWP)

The suggestion is nonsense. There's nothing wrong with the "waterfall" method. You can't get away from the "waterfall" method. All the "waterfall" method says is that you can't deploy a system until you've coded and tested it and you can't code and test it until you've designed it and you can't design it until you've analysed the requirements.

If you reject the "waterfall" method, you must believe that you should start designing a system before you know what's required and if you believe that then you're no use to the benefit claimants who need Universal Credit.

All that its proponents tell us about "agile" is that it's not "waterfall". But the "waterfall" method is right. It's the only method there is. It must be. There it is right bang in the middle of the "agile" method professed by ex-Guardian man Mike Bracken CBE's Government Digital Service – first you have a discovery phase, then an alpha and a beta, then you go live. That's a waterfall.

You can iterate, of course. You can have one release/deployment after another as the system is maintained and enhanced. But what you're iterating is a waterfall. There is no escape from the waterfall. And no need to escape from it. What would you be escaping from? From the belief that you must analyse before you design? But that's madness.

Madness won't solve DWP's problems. And those problems are not diagnosed simply by saying "waterfall" as though that's all bad and "agile", whatever it is, as though that's all good.

This distinction being touted between "agile" and "waterfall" is false.

----------

The picture of Mr Boehm's spiral has been added, as have the hyperlinks in the four citations at the end. but otherwise unchanged here's an extract from someone's essay written 12 years ago in February 2002 as part of his MSc in software engineering taken after 25 years of doing the job. For what it's worth:

Waterfall ... After that, it is pleasant to come back to the chirpy school magazine style of Requirenautics Quarterly [[1]]. There is not a megalomaniac or religious fanatic in sight and it is packed with sensible articles: a quick review by Ian Alexander of Barry Boehm's WinWin including a copy of every IT man's favourite cartoon; a long article by Ralph Young on how to help everyone; a good contrarian contribution by Richard Veryard in praise of scope creep; some helpful thoughts on abstraction and scenario-building by Ian Alexander; and a long series of book reviews contributed by the same man. One of the book reviews covers Ralph Young's book, Effective Requirements Practices. The review is witty, sensible, fair and acute but there is one sore thumb sticking out of it. Quite out of the blue, Mr Alexander writes (p.12): ... Something that they both agree on [Michael Jackson and Ralph Young] is that the waterfall model is inadequate: engineering development certainly does not follow a straight line path. Why does he say this? It adds nothing to his review. It is gratuitous cruelty. The waterfall model is like some unfortunate dog that no-one can pass without kicking. Some people will even go to the trouble of crossing the road just so that they can kick it: Mr Alexander, for example; and Messrs Bowen and Hinchey in their Ten Commandments of Formal Methods [[2]] – you would think that they had enough on their plate but, no (8th commandment): ... System development is by no means a straight-forward one-pass process. Royce's 'Waterfall' model of system development [[3]] was abandoned because of the simplistic view it held of system development .... People have been kicking it for years; kicking it seems to be an 11th commandment; but there's something funny about this dog – it's still there, it's always there, it doesn't matter how much you kick it, it just won't go away. Why? The waterfall method is supposed to be all wrong. How silly to think that a specification can ever be finished or that development can ever end! Well, yes, it is silly, but who ever thought that? The merest charwoman knows that she must dust over and over again, the dust keeps coming and she must keep dusting it away. Are we to suppose that once upon a time, in some dark age or some foreign country, users and system developers didn't realise this, unlike charwomen, and thought that requirements could be fixed finally and forever? Show me one of these system developers. Introduce me to one of these mythical users who would sign off a specification and then wait patiently and confidently for six years to have the perfect system delivered. I don't think they ever existed. Attacks on the waterfall method are attacks on an Aunt Sally. "What about Government contracts, then", you may ask? "They always insist on quoting a fixed fee for a fixed specification." Do they? Show me. I'd be interested. I've never seen a government contract. Perhaps they are as stupid as you suggest. Silly old government. But perhaps they aren't. Perhaps the contracts do allow for change. If you attack the waterfall method, does that mean that you think that you shouldn't analyse and design before you start coding? No, it is generally recognised that actually that is a rather sensible order to perform these tasks in. You can play games with the topology, of course: ·  You can say that the evolutionary method is better than the waterfall method. But all you've done then is to string out system development into a long series of ... waterfall cycles, each iteration with a bit of requirements capture followed by a bit of design followed by a bit of implementation and, yes, we'd probably better have a spot of acceptance testing before going into production. You're still using the waterfall method and, frankly, it would be surprising if you weren't. ·  Alternatively, if you really can't stand straight lines, spirals may be more your bag, with quick access from one iteration to the next, optionally cutting out a few phases of the waterfall cycle. But you're still actually acknowledging that the waterfall model is there, wrapped up in one of Mr Boehm's spirals. You can't get away from it and there is no reason to try. What does this achieve? It reinforces the point that all the processes involved in system development have to be performed iteratively and that progress is incremental. That should always have been clear anyway. It doesn't add anything to the original model and it doesn't make the original model wrong. So, I think that it may be time now for us to stop kicking the poor old dog. We should give it a shower, start feeding it properly and take it out for walks with us. The waterfall model should be treated as the long-suffering, worthy and faithful member of the software engineering family that it has always been. This dog deserves our warm regard and respect. After 30 years of this sort of vilification Mr Royce, the dog's breeder, has probably suffered his own Darkness At Noon [[4]] and brain-washed himself into believing that the waterfall model is a legitimate target for any passing boot. He should be rehabilitated. [1] Requirenautics Quarterly, BCS RESG, Issue 24, July 2001. [2]Bowen, Jonathan P. & Hinchey, Michael G., Ten Commandments of Formal Methods. [3]Royce, W.W., "Managing the Development of Large Software Systems", Proc. WESTCON'70, August 1970. [4]Koestler, Arthur, Darkness At Noon, Jonathon Cape, 1940. __________________________________________________________________

Agile is the opposite of waterfall – no

The Iguazu Falls (healthy/agile)

The Department for Work and Pensions have written off millions of pounds spent on developing IT for Universal Credit and we expect the write-off to rise into the hundreds of millions.

How can we stop intelligent organisations from wasting money like this?

Over and over again we are told that the answer is "agile".

Use "agile" software engineering methods and the waste will be minimised.

How? What problem is "agile" solving?

Over and over again we are told that "agile" is to be contrasted with "waterfall". Waste is endemic in "waterfall" software engineering methods. That's the problem. And "agile" will solve it – that's the suggestion.

What do you think about Parliament holding the government to account? It would be a good idea

The Blunders of Our Governments by Anthony King and Ivor Crewe

If you have to choose between reading this book and Conundrum by Richard Bacon and Christopher Hope, don't. Read them both.

Messrs King and Crewe start, like Bacon and Hope, with the descriptions of a dozen examples of UK government blunders. They then diagnose the problems and write out their eminently sensible prescriptions.

Private Eye get just one citation in Blunders and no citation at all in Conundrum. Which is odd – not much seems to get past Private Eye. If there's a bit of public maladministration going on, they seem to hear about it and they report it.

The same cannot be said of other media outlets, whether printed, broadcast or on-line. Many blunders simply do not get picked up. They fail to become scandals, as Messrs King and Crewe point out, even when they're huge.

Parisians call it the "Metro" and new Yorkers call it the "Subway". We Londoners call it the "tube". And when Labour came to power in 1997 the tube was falling to pieces. Gordon Brown and John Prescott set about fixing it.

They wanted to outsource the risk to the private sector. And they didn't want the cost appearing in the national accounts. "The magic you want in that case", they were told, "is a public-private partnership, a PPP". They were told this by armies of bankers and lawyers and accountants and consultants and contractors, all of whom proceeded to run rings round Brown and Prescott and, as Messrs King and Crewe tell us in Chapter 14, Down the tubes (p.221):

Although the figures are open to dispute ..., the PPP blunder certainly cost UK taxpayers not less than about £2.5 billion and probably far, far more, possibly in the region of £20-£30 billion.

£30 billion (maybe), and the media never got this scandal – the Metronet scandal – into the public consciousness.

But so what? You don't imagine that the perpetrators suffer, do you, even if their blunder does become a scandal? Surely you don't still believe that the buck must stop somewhere, mustn't it? With Whitehall officials, whose job it is to advise ministers and to implement policy? Or with ministers?

No. There's a lot of talk about accountability and responsibility. Baroness Jay once managed to lay on two hours of entertainment provided by not one, not two, not three but four former Cabinet Secretaries talking about these strange private sector concepts. And only the other day the current Cabinet Secretary said it was "unfair" to blame the Permanent Secretary at the Department for Work and Pensions for the Universal Credit blunders and what's more it was "harming civil service morale".

There's a lot of talk but as things stand Whitehall officials can't be penalised effectively and neither can their ministers, however much their blunders cost the taxpayer. Messrs King and Crewe are adamant (p.348):

... the idea that individual ministers should be held accountable still has a hold on the British psyche. People want to believe it. They half do believe it. They imagine that from time to time, even if not often, ministers who make serious mistakes are held accountable and are punished accordingly. Whenever a cabinet minister is in trouble, the House of Commons resounds with opposition cries of "Resign!", and editorials in newspapers often call for resignations. Whenever an individual minister has signally failed in either policy or administrative terms, it is somehow felt that he or she ought to resign, that he or she might possibly do the honourable thing and actually choose to resign. It is also felt that, if the individual in question does not resign voluntarily then the prime minister should step in and sack him or her. Sadly, our study of blunders points in the opposite direction. It suggests that the doctrine of individual ministerial responsibility is even more mythical than most people realise. It suggests that ministers guilty of even the grossest biunders are almost never held to account or punished in any way. On the contrary, most minister-blunderers, even if they can be clearly identified, survive politically and prosper ...

... and, à propos, Gordon Brown put in ten years as Chancellor of the Exchequer and three years as Prime Minister despite Metronet.

What do you think about Parliament holding the government to account? It would be a good idea. As Messrs King and Crewe say, Parliament has become "peripheral" and (p.366):

More MPs could usefully see themselves as real legislators and not just as social workers, cheerleaders, askers of questions and occasionally – once blunders have already been committed – conductors of belated post-mortem inquests.

Their book is a plea for Parliament to do its job of deliberation properly. At the moment, too often, unnecessarily and expensively, it fails. That's the final point they make in the main body of the book before a Postscript covering the coalition government to date, deliberation – there are more effective ways of deliberating than are currently used at Westminster or in Whitehall.

One quibble.

The notion of ministers, officials and their consultants and contractors being accountable the way the rest of us are does not exist solely in the "British psyche". It's also on the statute book. There is an offence known as "misfeasance in public office".

There are sound arguments that you don't want to push this sort of thing too far because democracy might shatter. Be careful what you wish for, and all that.

There are also sound arguments that in a democracy no-one is above the law. Not even the legislators.

Suppose that, in connection with Universal Credit, someone brought a charge of misfeasance in public office against Iain Duncan Smith, Robert Devereux and the UK heads of Accenture, IBM, HP and BT. Would democracy shatter, do you think, and let in some latter-day Stalin? Or would the effect of the law being enforced be more likely to cause Parliament in future to deliberate more effectively?

Answers in the second edition of this excellent book, please.

----------

Updated 13.1.14
DMossEsq, the lawyer? No.

Misfeasance in public office is not on the statute book. Joshua Rozenberg tells us that: "Misfeasance in public office is a tort - a civil claim that does not rely on the existence of a contract between the parties. It was not created by legislation: instead, it forms part of the common law, the distilled wisdom of judges through the ages".

In The English Law, chapter 6 of his England: An Elegy, Roger Scruton argues that common law is distinctive of the peculiar English character, it is an expression of the national psyche. To describe the idea of individual ministerial responsibility as having a hold on the British psyche, as Messrs King and Crewe do, is in that case profoundly accurate. No quibble there after all.

Further, as Duncan Fairgrieve tells us: "Misfeasance in public office is the only specifically ‘public law’ tort, and provides a remedy for citizens who have suffered loss due to the abuse of power by a public officer acting in bad faith". Since the UK heads of Accenture, IBM, HP and BT aren't public officers, even if they are the agents of public officers, it might be hard to bring a case of misfeasance in public office against them.

That said, there remains in this British psyche, at least, an ineradicable belief that there's something wrong with government being a responsibility-free zone.


Updated 13.1.17

RIP:
• Times obituary
• Telegraph

What do you think about Parliament holding the government to account? It would be a good idea

The Blunders of Our Governments by Anthony King and Ivor Crewe

If you have to choose between reading this book and Conundrum by Richard Bacon and Christopher Hope, don't. Read them both.

Messrs King and Crewe start, like Bacon and Hope, with the descriptions of a dozen examples of UK government blunders. They then diagnose the problems and write out their eminently sensible prescriptions.

Private Eye get just one citation in Blunders and no citation at all in Conundrum. Which is odd – not much seems to get past Private Eye. If there's a bit of public maladministration going on, they seem to hear about it and they report it.

The same cannot be said of other media outlets, whether printed, broadcast or on-line. Many blunders simply do not get picked up. They fail to become scandals, as Messrs King and Crewe point out, even when they're huge.

Parisians call it the "Metro" and new Yorkers call it the "Subway". We Londoners call it the "tube". And when Labour came to power in 1997 the tube was falling to pieces. Gordon Brown and John Prescott set about fixing it.

A Conundrum for the Electoral Commission

This morning the UK woke up to be told that the Electoral Commission proposes to introduce photo-ID for voting. It's one of those ideas that sound sensible until you investigate them.

How best to get this point across?

What better than a gift to the Chair of the Commission?

A gift with a message: "Dear Ms Watson Re PhotoID to vote, to avoid starring rôle in 2nd edition of Bacon&Hope's Conundrum (attached, pls enjoy), suggest large-scale independent trial first before reluctantly abandoning apparently good idea. Best wishes, DMossEsq".

---

From: auto-confirm@amazon.co.uk [mailto:auto-confirm@amazon.co.uk]
Sent: 08 January 2014 16:34
To: David Moss
Subject: Your Amazon.co.uk order

January Deals | Your Account | Amazon.co.uk Order Confirmation Order #204-7077346-0733905 Hello, Thanks for your order. We’ll let you know once your item(s) have dispatched. Your estimated delivery date is indicated below. You can view the status of your order or make changes to it by visiting Your Orders on Amazon.co.uk. Your estimated delivery date is: Tuesday, January 14, 2014 - Thursday, January 16, 2014 Your delivery option: Super Saver Delivery Your order will be sent to: Jenny Watson Chair Electoral Commission 3 Bunhill Row London, EC1Y 8YZ United Kingdom This order contains one or more items that you've marked as a gift. Order Details Order #204-7077346-0733905 Placed on January 08, 2014 Conundrum: Why every government gets things wrong and what we can do about it Sold by Amazon EU S.a.r.L. Gift wrap: No Gift Message: Dear Ms Watson Re PhotoID to vote, to avoid starring rôle in 2nd edition of Bacon&Hope's Conundrum (attached, pls enjoy), suggest large-scale independent trial first before reluctantly abandoning apparently good idea. Best wishes, DMossEsq £12.23 Item Subtotal: £12.23 Delivery & Handling: £0.00 Order Total: £12.23 Selected Payment Method: Visa Need to make changes to your order? Visit our Help page for more information and video guides. We hope to see you again soon. Amazon.co.uk Unless otherwise noted, items sold by Amazon EU Sarl are subject to Value Added Tax based on country of delivery in accordance with the EU laws on distance selling. If your order contains one or more items from a seller other than Amazon EU Sarl, it may also be subject to VAT, depending upon the seller's business policies and the location of their operations. Learn more about VAT and seller information. This e-mail is only an acknowledgement of receipt of your order. Your contract to purchase these items is not complete until we send you an e-mail notifying you that the items have been dispatched. Please note: This e-mail was sent from a notification-only address that can't accept incoming e-mail. Please do not reply to this message.

---

Dispatched:

From: Amazon.co.uk [mailto:auto-shipping@amazon.co.uk]
Sent: 08 January 2014 22:53
To: David Moss
Subject: Your Amazon.co.uk order of "Conundrum: Why every..." has been dispatched

January Deals | Your Orders | Amazon.co.uk Dispatch Confirmation Order #204-7077346-0733905 Hello, We thought you'd like to know that we've dispatched your item(s). Your gift order is on the way, and can no longer be changed. If you need to return an item or manage other orders, please visit Your Orders on Amazon.co.uk. Your estimated delivery date is: Tuesday, January 14, 2014 No tracking information. Why? Your order was sent to: Jenny Watson Chair Electoral Commission 3 Bunhill Row London, EC1Y 8YZ United Kingdom This delivery contains one or more items that you've marked as a gift Your item(s) is (are) being sent by Royal Mail. This order does not have an associated tracking number.

A Conundrum for the Electoral Commission

This morning the UK woke up to be told that the Electoral Commission proposes to introduce photo-ID for voting. It's one of those ideas that sound sensible until you investigate them.

How best to get this point across?

Bacon and Hope's faith is a mystery

Conundrum: Why every government gets things wrong and what we can do about it
by Richard Bacon MP and Christopher Hope

As a member of the Public Accounts Committee, Richard Bacon has been an observer for years of the scandalous failures of our government in the UK. Not just an observer. An energetic and noble investigator as well.

In the first 12 chapters, he and Mr Hope tackle the gruesome Child Support Agency, the UK Passport Agency that couldn't issue passports, HM Treasury's tax credits fiasco, and nine more government failures.

They write clearly and authoritatively and it would be a pleasure to read their prose if it weren't for the fact that what we're reading is the story of how billions of pounds of public money have been wasted by the Executive – by Whitehall and the Ministers in political charge of Whitehall.

With 12 sets of raw material to work on, they then give themselves five chapters to do what it says in the title. That is, to explain why governments get things wrong and to suggest what we can do about it.

Messrs Bacon and Hope quote from a large number of studies of the problem. Again, they write very well. And it's a valuable service, hugely appreciated, to bring together so much of the literature in one place.

The many solutions proposed over the past 30 years or so are analysed with philosophical rigour, touching on the constraints of politics in a democracy. None of these proposals has worked – the same lurid mistakes carry on being made, Whitehall remains too often unbusinesslike and irresponsible.

Can Messrs Bacon and Hope succeed where everyone else has failed?

No. Regrettably.

In Chapter 13, which is devoted specifically to the failures of government IT, they tentatively suggest that "agile" software engineering methods might work better than "waterfall" methods. No. They might do better to consider Professor Sir Martyn Thomas's advice to the House of Commons Science and Technology Committee – formalised languages.

And at the end of their tether , in Chapter 17, they assert that advances in behavioural psychology will improve the record of delivery by government. This desperate gesture is based on the fact that we humans share 98 percent of our DNA with chimpanzees and on the success of Whitehall's Behavioural Insights Team in improving the rate at which people in Devon pay tax by "nudging" them.

The Behavioural Insights Team (RIP) have made no suggestions what to do about the 12 chapters of Whitehall's delivery failures. No suggestions, at least, recorded by Messrs Bacon and Hope. Nor have any of the other behavioural psychologists they cite, the Thalers, Sunsteins and Kahnemans of this world. Bacon and Hope's faith is a mystery.

The National Audit Office reported that Her Majesty's Revenue and Customs pay their contractors' invoices on the ASPIRE project even though they often don't know what the invoices are for. The contractors have been asked for a breakdown but they refuse to provide it. You don't need to be a behavioural psychologist nor a Nobel Prize-winning economist to know that this practice is unbusinesslike and irresponsible.

Arguably this practice, like the scores of derelictions in Bacon and Hope's first 12 chapters, amounts to misfeasance in public office. That is an offence. And prosecuting one or two of these offences might have a salutary effect while we're waiting to see what the chimpanzees can teach us.

Bacon and Hope's faith is a mystery

Conundrum: Why every government gets things wrong and what we can do about it
by Richard Bacon MP and Christopher Hope

As a member of the Public Accounts Committee, Richard Bacon has been an observer for years of the scandalous failures of our government in the UK. Not just an observer. An energetic and noble investigator as well.

In the first 12 chapters, he and Mr Hope tackle the gruesome Child Support Agency, the UK Passport Agency that couldn't issue passports, HM Treasury's tax credits fiasco, and nine more government failures.

They write clearly and authoritatively and it would be a pleasure to read their prose if it weren't for the fact that what we're reading is the story of how billions of pounds of public money have been wasted by the Executive – by Whitehall and the Ministers in political charge of Whitehall.

With 12 sets of raw material to work on, they then give themselves five chapters to do what it says in the title. That is, to explain why governments get things wrong and to suggest what we can do about it.

Messrs Bacon and Hope quote from a large number of studies of the problem. Again, they write very well. And it's a valuable service, hugely appreciated, to bring together so much of the literature in one place.

The many solutions proposed over the past 30 years or so are analysed with philosophical rigour, touching on the constraints of politics in a democracy. None of these proposals has worked – the same lurid mistakes carry on being made, Whitehall remains too often unbusinesslike and irresponsible.

Can Messrs Bacon and Hope succeed where everyone else has failed?

RIP IDA – individual electoral registration

The key to success with regard to IER lies in being boring.

The more boring the better.

No need to say it, it goes without saying, it should be obvious to all but, just in case it isn't obvious to all, IDA is dead.

IDA is the Cabinet Office Identity Assurance programme. And it's dead.

----------

If all goes well, the media will pay not the slightest attention to the changes promised for 2014 in the way the electoral register is compiled in Great Britain.

Beginning on 10 June 2014, England and Wales will switch from compiling the electoral register on a household basis to individual electoral registration (IER). In Scotland, the equivalent date is 19 September 2014 – the delay there is to cater for the referendum on Scottish independence.

IER will be a yawn and a bore. That's if all goes well. The new electoral register will be ready for the 2015 general election and it will be complete enough and accurate enough not to impugn the legitimacy of the election result.

The Electoral Commission published a readiness report back in October 2013. They've got the forms ready and they just need political approval before Electoral Registration Officers (EROs) send them out to individuals to register. That will be in July 2014 and there will be an accompanying public awareness campaign.

It is to be hoped that that public awareness campaign will be workmanlike, clear, simple and above all uncontroversial. Dull. Worthy. Yawn-inducing, as befits a highly respected, confident and mature democracy.

There are a few worryingly interesting bits of IER.

Confirmation
There is the "confirmation" element, for example:

The Government’s plan for the introduction of IER includes the intention to compare existing electors’ names and addresses on the electoral registers with records held by the Department for Work and Pensions (DWP) in order to verify the identity of people currently on the registers. This process is known as ‘confirmation’.

EROs will be provided with reports comparing their electoral register records with records on DWP's Customer Information System database (CIS). On the basis of trials already carried out, it is expected that about 75% of the electoral register can be "confirmed" in this way.

How good is CIS?

Back in April 2007 there were about nine million records on CIS that no-one could account for, please see Fraud fear as millions of NI numbers are lost. Surely, you may say, DWP could clean up their data? They already had. Before the clean-up, there were more like 20 million unaccountable records according to David Blunkett.

Back in the old days of the National Identity Scheme/Service (2002-10 RIP) when we were all going to have government-issued identity cards (RIP), the Identity & Passport Service (RIP) were going to build a brand new National Identity Register (RIP). Then they decided to use CIS instead ...

... and then the National Audit Office (para.4.13, p.23) pointed out that they could only use CIS if it met the security standards laid down by CESG, the information assurance arm of GCHQ. Which it didn't, please see the Public Administration Select Committee report Good Governance – effective use of IT, pp.295-304, particularly para.24ff. At which point the whole ID cards project collapsed into obvious chaos.

The Electoral Commission may say that CIS confirmation will "verify the identity of people currently on the registers" but they're just being polite. If there's a mismatch between CIS and the electoral roll, which database is right? Neither of them? Is it the EROs' job to clean up the CIS? No. It's all too interesting. No more chaos. Expect CIS confirmation quietly to disappear.

Verification
In one sense there's nothing new about "verification". It's always been the EROs' job to verify that people are who they say they are and that they are allowed to vote. EROs know how to do that and they will continue to verify the entitlement to vote, boringly it is to be hoped, without fuss and behind the scenes.

But there is supposed to be a new element in 2014, an IER digital system for verification, please see para.1.12 onwards in the readiness report:

1.12 ... [the] Commission has some remaining concerns around the timetable for developing the other significant element of the system - for verifying electors’ personal identifiers under IER ... 1.13 We are aware that some testing of the system has recently taken place ... We understand that there are further tests on the algorithm to be completed ... 1.15 ... the system for verification has not yet been fully tested, and according to the current plans will not have been fully tested until March 2014 ... We understand that the testing programme will be conducted on a rolling basis between now and next March, but the key risk is that it will not be fully clear until then whether the system is fully robust ... 1.17 We have not yet seen a detailed plan for the full testing process, although we understand from officials that this will be shared shortly ... 1.18 It will be important for this testing to demonstrate the ability of the system to cope with the volume of registrations ... We (and EROs) ... await reassurance on this point. 1.19 It is also important for effective and realistic contingency plans to be put in place in the event that problems with the verification system do arise ... We have not yet seen any detailed plans although we are aware that the [Electoral Registration Transformation] Programme team are working on them. We would welcome sight of them when they are available ...

It's not clear from the quotations above what digital identity verification is. The only thing that is clear is that the Commission's welcome for this new component of IER is heavily qualified. That is only to be expected after their experience of the dog's dinner served up by the data-mining pilots.

The suspicion is that what is intended here by "digital identity verification" is something to do with the Government Digital Service's identity assurance scheme, IDA.

IDA was meant to provide us with an "ecosystem" of competing private sector "identity providers". Philip Virgo tells us that there were initially "80 expressions of interest" in joining the IDA framework, please see Who won the battle between DWP and Cabinet Office over ID Policy?. 80 became eight a year or so later in January 2013. By September 2013, eight had become five. And now we're down to two, please see Beta launch for identity assurance this year:

... an official from the IDA programme ... explained that the first two identity providers will start supporting the scheme from the end of November ... These two providers come from a pool of five companies- Digidentity, Experian, Mydex, The Post Office and Verizon- who have signed contracts to deliver IDA services, out of a total of eight companies who were originally on the framework. The official said that they are hoping for new providers to join in and start working on the programme next year ...

We do not know which two "identity providers" now occupy the shrinking "ecosystem".

And we do not know which brave "new providers" might join where 78 have already pulled out.

Perhaps the government could lean on the two banks which it controls, Lloyds and RBS? Let's hope not. We've been here before. There's no upside. It's all risk.

Perhaps the government could invite Facebook to help. Or Google:

Andrew Nash, Google’s Director of Identity, ran us [Francis Maude and ex-Guardian man Mike Bracken] through the current issues facing identity.He explained how Google aim to grow and be part of an ecosystem of identify providers, and encouraged the UK Government to play its part in a federated system. The UK ID Assurance team and Google agreed to work more closely to define our strategy – so look out for future announcements. Andrew also took the opportunity to walk the Minister through the Identity ecosystem.

All far too interesting for the Electoral Commission to let them become involved. Expect digital verification politely, quietly and firmly to be swept under the confirmation carpet.

On-line registration portal
Ditto the on-line registration portal, para.1.21 onwards in the readiness report:

1.21 ... we are concerned that the website which will enable online registration in Great Britain has also not yet been fully tested. 1.22 The current plan indicates that while the user-facing part of the application ... will be developed for the majority of users by the end of October 2013, testing the process that takes place ‘behind’ the screen ... will not be completed until later (likely to be March 2014). As with the verification development work more generally, this is a tight timetable given the intended IER start date in June 2014 and we have not seen a detailed timetable for this testing ... 1.23 We also understand that the technical development work required to allow use of the online application system by certain important groups of electors ... will not be completed until March 2014. 1.24 The [Electoral Registration Transformation Programme] team have assured us that this development work has been fully scoped and timetabled and that they are confident of delivering the work to time. However, ... this remains an important area of concern.

The Electoral Commission hardly need reminding that an on-line registration portal with no identity assurance and no "ID hub" is an invitation to electoral fraud. Unicorns. Too interesting. Drop it. Keep it boring.

Cloud computing
Always keen to follow the latest fad, the Government Digital Service want to store all our data "in the cloud", as they say. They have chosen to use Skyscape and Carrenza.

Storing data in the cloud is the most efficient way of losing control of it.

The Commission may care to look into the current practice of using the cloud for electoral rolls. One company, Halarose, which provides electoral registration services to 80 local authorities, runs its services on Amazon's cloud servers located in the Republic of Ireland.

Do the Commission agree that this raises interesting questions whether our data is properly under the control of the people who owe us voters a duty of care? If the questions are too interesting, perhaps the Commission would look into changing the current lenient procedures which countenance use of the cloud, not just by Halarose but throughout the electoral registration system.

The key to success with regard to IER lies in being boring. The more boring the better.

----------

Updated 8.1.14
Keep it boring. A simple enough suggestion. So what do the Electoral Commission do? They only go on the radio this morning and announce that we're all going to need photo-ID to vote. That's what.

Far too interesting.

What photo-ID? Passports and photo-ID driving licences. Or a special voting ID card for people who don't have a passport or a photo-ID driving licence. The special voting ID card will be free. "Free"?

If you need photo-ID to vote, why don't you need it to register?

The credit card companies rejected photographs on credit cards in the UK because, based on tests with supermarket staff, that would increase fraud and not reduce it. How would the people manning polling stations fare any better? What happens when you are refused your right to vote because one of these people says you don't look like yourself?

How long before someone points out that if you can register on-line you ought to be able to vote on-line? Perhaps proving your identity on-line using GDS's non-existent identity assurance system? Or, for old-timers, using biometrics.

How long before someone points out that if you need photo-ID to vote, then you must need it to get married? Or to get your children into state education? Or to be given non-emergency state healthcare?

It is strongly suggested that the Electoral Commission conduct trials to see if the benefits of photo-ID voting outweigh the costs. If not, the initiative is counter-productive and disproportionate and should be dropped. There is no loss of face in acknowledging the authority of a large-scale independent trial. The larger and the sooner, the better.

Then, let's hope, we can get back to boring. Please. Boring, boring, boring.

Updated 10.6.14

Today is the first day of individual electoral registration in England and Wales. There is not a single press release about it. Anywhere. Not even on the Electoral Commission's website.

Updated 28.6.14

UK should consider e-voting, elections watchdog urges Rowena Mason, political correspondent The Guardian, Wednesday 26 March 2014 18.31 GMT ... the head of the Electoral Commission, Jenny Watson, warned that the ... long-term trend of falling voter turnout was particularly marked among young people ... ... the election watchdog would examine a range of ways to make voting more accessible, including the "radical" option of internet voting and US-style same-day registration for those not on the electoral roll ... ... "we plan to look at a variety of options, assessing how they will help citizens engage more effectively" ... more could be done to make the system more reflective of wider society ... "an increasingly disenfranchised younger generation" ... ... "Unless our electoral system keeps pace with the way many voters live the rest of their lives – where the way they bank and the way they shop has been transformed – it risks being seen as increasingly alien and outdated, particularly to young voters as they use it for the first time" ...

The claims Jenny Watson makes for on-line voter registration and on-line voting are hypotheses. The potential benefits are great. It is worth testing these hypotheses. And they have been.

Estonia allow internet voting. And the University of Michigan discovered that the system is open to being hijacked – the result of the election may not be decided by the voters. The university had previously discovered the same fault in a proposed eVoting system in Washington DC. And now we hear that Norway have given up on eVoting after some careful testing:

E-voting experiments end in Norway amid security fears BBC News – Technology 27 June 2014 Last updated at 12:12 ... voters' fears about their votes becoming public could undermine democratic processes. Political controversy and the fact that the trials did not boost turnout also led to the experiment ending ... ... criticism was levelled at the encryption scheme used to protect votes being sent across the net ... ... there was no evidence that the trial led to a rise in the overall number of people voting nor that it mobilised new groups, such as young people, to vote ... ... there was also some evidence that a small number of people, 0.75% of all voters, managed to vote twice in 2013 ...

The result of these on-line voting tests is to cast doubt on the hypothesis. It seems to be wrong. On-line voting doesn't boost participation and it introduces dangerous features which undermine the trustworthiness of the election. Conclusion: it is irresponsible to assume that on-line voting is a cure-all.

What about on-line registration to vote?

The Government Digital Service (GDS) introduced an on-line system on 10 June 2014 in pursuit of individual electoral registration in England and Wales. The system collects application details and forwards them to electoral registration officers (EROs) who have to decide whether to register the applicant to vote.

How do the EROs know whether the applicant is who they say are?

GDS have provided a check based on use of the applicant's National Insurance number. That is the same check they use in DVLA's view-driving-record application. And what do the Driver & Vehicle Licensing Agency have to say about it?

Access to the service is currently allowed by matching the user’s data to the driving licence number. We also use an existing link to the Department for Work and Pensions (DWP) to check if the National Insurance Number (NINO) provided matches details held by DWP and HM Revenue & Customs. Whilst this authentication process is fairly quick and straightforward, there are some downsides ... it does not provide us with the level of confidence the user is who they say they are in order to offer them more information such as their photo image or allow them to link to a transactional service.

Again, a perfectly sensible hypothesis, but the test results suggest that it would be irresponsible to rely on on-line voter registration.

Updated 3.7.14

All this febrile raving about photo-id and electronic voting and same-day registration? Too exciting. We need boring.

The Electoral Commission have started their public awareness campaign for individual electoral registration, hat tip Halarose:



30 seconds of total inanity. The postman delivers a letter. Men turn into women half way downstairs to the accompaniment of irritating music and then read the IER information leaflet while they enjoy a cup of tea.

The shoutline? "Make sure you're in".

Congratulations to the Electoral Commission. Perfect. A collector's item. More like that, please.

RIP IDA – individual electoral registration

The key to success with regard to IER lies in being boring.

The more boring the better.

No need to say it, it goes without saying, it should be obvious to all but, just in case it isn't obvious to all, IDA is dead.

IDA is the Cabinet Office Identity Assurance programme. And it's dead.

----------

If all goes well, the media will pay not the slightest attention to the changes promised for 2014 in the way the electoral register is compiled in Great Britain.

Beginning on 10 June 2014, England and Wales will switch from compiling the electoral register on a household basis to individual electoral registration (IER). In Scotland, the equivalent date is 19 September 2014 – the delay there is to cater for the referendum on Scottish independence.

IER will be a yawn and a bore. That's if all goes well. The new electoral register will be ready for the 2015 general election and it will be complete enough and accurate enough not to impugn the legitimacy of the election result.

The Electoral Commission published a readiness report back in October 2013. They've got the forms ready and they just need political approval before Electoral Registration Officers (EROs) send them out to individuals to register. That will be in July 2014 and there will be an accompanying public awareness campaign.

It is to be hoped that that public awareness campaign will be workmanlike, clear, simple and above all uncontroversial. Dull. Worthy. Yawn-inducing, as befits a highly respected, confident and mature democracy.

There are a few worryingly interesting bits of IER.