GaaP has its own blog ...
... and the other day Graham Bleach published Incident Report: Platform as a Service for government:
Take another look. Just to check: "a program to delete ... development environments was accidentally run in the production environment. As a result, there was a complete outage to the platform".
This post is about an incident in the 'Platform as a Service for government' production environment for hosting applications.
What happened
At 1.30pm UTC on Friday 3 June 2016, a program to delete Cloud Foundry (CF) development environments was accidentally run in the production environment. As a result, there was a complete outage to the platform.
Accidentally?
This accident should be almost impossible.
Cast your mind back 37 years to 1979, when DMossEsq had to make an amendment to some data on the Lloyds Bank International (LBI) live production database. Programs acting on the production system all had the prefix X. Y was used for test development systems only. So he couldn't use YDBAXS, it had to be XDBAXS. The operators wouldn't run an XDBAXS job without the signed authorisation of the deputy head of the computer department. And he wouldn't sign without a convincing explanation.
No-one questioned this procedure. It was just obviously prudent.
It still is. But it appears nevertheless to have eluded GDS. They have only just learned this basic element of prudence. A test data job was allowed to delete the production environment. "... there was a complete outage to the platform". On their core system. GaaP. The one that Whitehall and everyone else is supposed to feel total confidence in. And invest in.
If GDS are still learning that lesson, what else don't they know yet? That we at LBI and everyone else already knew 37 years ago and more.
----------
Updated 10.8.16
Helen Margetts is the "Professor of Society and the Internet and Director, Oxford Internet Institute, University of Oxford" and she "sits on the Advisory Board of the Government Digital Service [GDS]".
"The largest government departments have begun to reassert their authority over GDS expert advice", she tells us, without telling us what GDS's expertise is, "and digital government looks likely to be dragged back towards the deeply dysfunctional old ways of doing things", like keeping your production environment safely separated from the development environment, please see above.
"GDS isn’t perfect, but to erase the progress it has put in place would be a terrible loss". What is this logic? GDS has failed to see off the deeply dysfunctional old ways of doing things and that is progress, the loss of which would be terrible?
GDS has 700 staff according to Professor Margetts and a budget of £450 million to spend on unspecified UK government contracts by March 2020. Just like half a dozen other systems integrators (SIs), the villains of the deeply dysfunctional old ways of doing things.
They have performed just as badly as the other SIs. See for example Government Digital Service “hindered delivery” of rural payments programme, Public Accounts Committee says and Student Loans Company burns £50 million in IT project superfail and Electoral Commission warns of ‘lost’ voters and ...
What went wrong? Was it a deeply dysfunctional leadership? Or a deeply dysfunctional Advisory Board?