Tuesday 14 May 2013
The unqualified success of the Government Digital Service
Comment submitted to the UK Constitutional Law Group in response to a post on their blog about the perils of GOV.UK:
The unqualified success of the Government Digital Service
Comment submitted to the UK Constitutional Law Group in response to a post on their blog about the perils of GOV.UK:
midata is an attempt to get us all to embrace PDSs (personal data stores)
Comment submitted to Craig Belsham's midata blog:
Mr Belsham
My objections to midata are set out in my response to last year's BIS consultation and I shan't repeat them all here.
None of midata's claims to empower the consumer and to expand the economy is even remotely convincing. Which leaves me asking, like Paul Clarke, why?
One hypothetical answer is that midata's sole purpose is to encourage people to maintain PDSs (personal data stores).
That hypothesis is consistent with William Heath being a member of the midata strategy board and the chairman of Mydex – a PDS company – which is, in turn, one of the UK's eight appointed identity providers. It makes midata part of the Government Digital Service's Identity Assurance Programme (IDAP).
It doesn't excuse the mendacious marketing. But at least it explains why Whitehall takes the trouble to promote this otherwise fatuous initiative.
What do you think, Mr Belsham?
midata is an attempt to get us all to embrace PDSs (personal data stores)
Comment submitted to Craig Belsham's midata blog:
Mr Belsham
My objections to midata are set out in my response to last year's BIS consultation and I shan't repeat them all here.
None of midata's claims to empower the consumer and to expand the economy is even remotely convincing. Which leaves me asking, like Paul Clarke, why?
One hypothetical answer is that midata's sole purpose is to encourage people to maintain PDSs (personal data stores).
That hypothesis is consistent with William Heath being a member of the midata strategy board and the chairman of Mydex – a PDS company – which is, in turn, one of the UK's eight appointed identity providers. It makes midata part of the Government Digital Service's Identity Assurance Programme (IDAP).
It doesn't excuse the mendacious marketing. But at least it explains why Whitehall takes the trouble to promote this otherwise fatuous initiative.
What do you think, Mr Belsham?
The historically inevitable triumph of midata
Many of us find the Department for Business Innovation and Skills's initiative, midata, perplexing. With the passing of the Enterprise and Regulatory Reform Act 2013, midata now has the statutory powers needed. But why?
An explanation is available.
It nestles in one of the comments on Craig Belsham's first post on his new midata blog.
The midwife to our understanding is one William, who opens his comment (May 8, 2013 at 9:47 am) with a dazzling and surely incontrovertible exposition of the economic benefits of midata and then adds:
An explanation is available.
It nestles in one of the comments on Craig Belsham's first post on his new midata blog.
The midwife to our understanding is one William, who opens his comment (May 8, 2013 at 9:47 am) with a dazzling and surely incontrovertible exposition of the economic benefits of midata and then adds:
Thank goodness, you may say, that there are all-powerful, benevolent scholars out there who understand economics and who will help us to overcome our miserable inertia.
It’s a rational thing for businesses to accept and make a virtue of going along with the inevitable. But inertia tales a lot of overcoming, and it’s understandable to see the element of regulation in Midata ...
The historically inevitable triumph of midata
Many of us find the Department for Business Innovation and Skills's initiative, midata, perplexing. With the passing of the Enterprise and Regulatory Reform Act 2013, midata now has the statutory powers needed. But why?
An explanation is available.
An explanation is available.
Sunday 12 May 2013
Biometrics: will the Center for Global Development reconsider?
A recently published report on India's identity management scheme says that: "accurate, biometric-based, identification is quite feasible for large countries, including the US".
The suggestion below is that the conclusion should read: "subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register and the fact that the discipline of biometrics is out of statistical control".
Will the authors consider issuing a revised version of their report?
On the rare occasions when trials have been conducted, the performance of biometrics technology has been disappointing. For example, when 10,000 of us took part in a UK government-run trial in 2004, about 20% of participants couldn't have their identity verified by their fingerprints.
That's useless. For example, the plan at the time was to use biometrics to confirm people's right to work in the UK. You can't tell 20% of the working population that it's illegal for them to work.
Ever optimistic, the biometrics industry is always announcing that the corner has been turned and that it's safe now to believe their promises. Is that true at last?
Consider Performance Lessons from India’s Universal Identification Program [Updated 13.12.18, change of web address, now here], a 12-page report by Alan Gelb and Julia Clark (Gelb and Clark, G&C).
It's about India's Unique Identification project (UID, also known as "Aadhaar") which relies on biometrics. UID/Aadhaar is the brainchild of UIDAI, the Unique Identification Authority of India. UIDAI are currently trying to register the biometrics of all 1.2 billion Indians.
G&C conclude that:
If they're correct.
But are they?
Why do G&C conclude that biometrics is now ready for large-scale deployment?
They have identified "160 [biometrics] programs in 70 countries that together cover over 1 billion people and include a wide range of applications – financial access, public payroll management, social transfers [?], health insurance and tracking and voter rolls – as well as national identification systems" (p.1).
Do they say that biometrics is ready for the big time because UIDAI have successfully implemented financial access systems which depend on biometrics? Or public payroll management systems? Or ...
Certainly not.
In fact G&C are at pains to say that:
What G&C base their conclusions on is the performance of biometrics in the compilation of the Indian population register so far. If we are to answer the question whether their conclusions are correct, we need to look at the UIDAI statistics which measure the reliability of biometrics.
Before we do that, we need to update G&C's conclusions. There's a rider to add. Their p.2 warnings about spoofing and eavesdropping on telecommunications and burgling the population register need to be incorporated – the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack.
Slide rules ready? G&C say (p.5):
Think in terms of ice cream. How many unique combinations of two ice cream flavours can you make from a choice of five flavours (A, B, C, D and E)? G&C suggest that the answer is 25, "the square of the population". It isn't. It's 10 (AB, AC, AD, AE, BC, BD, BE, CD, CE and DE), 5!/((5-2)! x 2!).
G&C have a peculiar habit. They're talking about India, with its population of 1.2 billion, but half the time when they use statistics they apply them to Ughana, a country they have invented. Why?
It confuses the readers. It may also confuse the writers. Forswearing Ughana and sticking to India, how many comparisons would have to be made to compare each one of 1.2 billion sets of biometrics against all the rest? Answer, 719,999,999,400,000,000 and not G&C's implied answer 1,440,000,000,000,000,000, which is out by a smidgeon over 100%.
New rider on the conclusions – the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right.
It was in the midst of a discussion about false accept rates and false reject rates.
Aadhaar is all about comparing the biometrics captured by a fingerprint scanner or an iris scanner with the biometrics stored in the population register. Either they match or they don't.
Say your Aadhaar number is 782474317884, that there's an election on and that you have turned up at a voting centre. The biometrics associated with 782474317884 are retrieved from the population register and checked to see if they match your freshly scanned biometrics. If they do, you can vote. It's a one-to-one comparison, an "authentication" process.
Two ways the process can go wrong (among others):
Take a look at UIDAI's 27 March 2012 report on authentication (p.4). Using one or two fingers to authenticate yourself, UIDAI expect the Aadhaar system to be between 93.5% and 99% accurate. I.e. FRR will be between 1% and 6.5%. That's with a FAR of 0.01%. FRR is high, FAR is low(ish).
Varying FAR from high to low and FRR the other way is achieved by changing the matching threshold. You can set the system to insist on a very high score before asserting that President Lincoln's freshly scanned fingerprints match the set already stored on the population register. That would give a low FAR and a high FRR. Or you can set a very low threshold and achieve the opposite. And all points in between.
This is odd.
In the world we're used to, if you are President Lincoln then you are President Lincoln and that's all there is to it. It doesn't depend on the matching threshold set by some state functionary.
In the world of Aadhaar, depending on the threshold chosen, sometimes you will be President Lincoln (low threshold, easy to achieve a match, low FRR, high FAR) and sometimes you won't (high threshold, hard to achieve a match, high FRR, low FAR). It all depends. At the limit, the functionary could fix it so that no-one was President Lincoln. Or that everyone was.
When we said above that "either they match or they don't", that was a tease. That's the way people imagine biometrics systems to work. All cut and dried. In fact, it's discretionary.
The concept of identity in Aadhaar is different from the concept in the real world. Identity becomes discretionary, something that can be granted or revoked by twiddling the dial on a gizmo.
There's another rider to add to G&C's conclusions – the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed.
Identification is different.
Identification is the process you go through when you are enrolled into Aadhaar. Before identification, you don't exist as far as Aadhaar is concerned. If public services in India ever start to depend on Aadhaar and you don't have an Aadhaar number, you won't get any public services. Why would the state provide benefits to someone who doesn't exist? At the very least you will look very suspicious.
Identification is a one-to-many process. When you first enrol someone in the register, their biometrics have to be checked for uniqueness. Instead of checking them against just one set of biometrics, they have to be checked against every set already registered.
The errors that can be made by the biometrics system are very similar (among others, yes you are already enrolled when really you're not or no you're not already enrolled when really you are) but the process has such existential consequences that it's normal to talk of false negative identification rate (FNIR) and false positive identification rate (FPIR), rather than FAR and FRR, to distinguish it from mere quotidian authentication.
UIDAI talk of FRRs between 1% and 6.5% for authentication using fingerprints whereas, when it comes to identification, their FPIR figure is 0.057%. That's two orders of magnitude different. Identification is a strict process and, by comparison, authentication is flabby.
G&C unfortunately use FAR and FRR for both identification and authentication which obscures the important distinctions between the two processes.
How good are the biometrics UIDAI are using at creating a reliable population register?
It's a problem Professor John Daugman has looked at. Not in connection with Aadhaar in particular. But in general. For any biometrics-based identity management scheme.
Remember, to establish uniqueness for every one of the 1.2 billion sets of biometrics on India's population register, you have to make 719,999,999,400,000,000 comparisons.
Suppose, says Professor Daugman, that there's a mistake 1 time in a million such that a false positive identification is made. Then Aadhaar will throw up 719,999,999,400 false matches.
These can't be resolved by the computer – it's the computer that threw up the false matches in the first place. They have to be resolved by human investigations.
Humans aren't going to complete 719,999,999,400 investigations. It's impractical. The identity management scheme will drown in a sea of false positives, as the professor puts it.
Is there a one-in-a-million chance of a mistake?
Professor Daugman thinks that it's a lot worse than that if you rely on face recognition as a biometric. There's far too little randomness in faces, there are far too few degrees of freedom, for face recognition to support enormous numbers like 719,999,999,400,000,000. (Never mind Ughana, that doesn't stop the UK government wasting money on face recognition.)
Fingerprinting is better in this sense than face recognition, but still not good enough to avoid drowning in a sea of false positives. (That doesn't stop the UK government wasting money on glitzy new fingerprinting systems.)
Irises on the other hand do have enough randomness, he says, there are enough degrees of freedom to stay afloat. Which is good news for UIDAI – Aadhaar uses a combination of both fingerprints and irises.
According to UIDAI's report on identification (p.4), on 31 December 2011 when there were 84 million sets of biometrics on the population register, the FPIR was 0.057%, the FNIR was 0.035% and "it is unnecessary and inaccurate to attempt to infer UIDAI system performance from other systems which are ten to thousand times smaller".
It may be unnecessary and it may be inaccurate but it's impossible to resist the temptation – compared to any other biometrics-based scheme known to man, these figures for Aadhaar are astonishing. Certainly no salesman worth his or her salt will ignore it.
It looks as if there would be only 684,000 false positive identifications to investigate by the time the population register is full, and not 719,999,999,400.
684,000 is manageable. As UIDAI say (p.18):
Presumably they had recorded 47,880 cases of false positive identifications to date.
You'd think that. But you'd be wrong. UIDAI tell us that (p.18):
Funny way to do it.
Perhaps we shall be told that there's an agreed protocol in the biometrics industry such that this is an acceptable way of determining FPIR. Even so, why not report the actual number of false positive identifications recorded?
That statistic should be available in the case of Aadhaar – G&C tell us that (p.2):
The questions mount and the answer gradually comes into focus – in order to inspire confidence, UIDAI's figures need to be audited by independent experts and certified like a set of company accounts.
And, like company accounts, they should be audited every year. These figures from 31 December 2011 are getting very long in the tooth.
Another rider – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed.
G&C have this footnote, #7, on p.5 of their report:
What does that mean? In order not to drown in false positives, UIDAI will let false negatives go up? UIDAI have got to get the population register completed and if that means tolerating lots of duplicate entries, too bad, so be it, let uniqueness go hang? If that isn't what it means, then what?
How relaxed? Very relaxed? What level does FNIR have to rise to, to keep FPIR down at 0.057%? Do UIDAI even know? Should they change their name to the Multiple Identification Authority of India?
"It is unnecessary and inaccurate to attempt to infer UIDAI system performance from other systems which are ten to thousand times smaller"? On the contrary, it is only sensible to question UIDAI's performance claims.
The riders are piling up now – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register.
It's different with false negative identification. If an impostor turns up at the centre and his or her earlier registration is not detected by Aadhaar, then they're not going to tell you. You won't know. Impostors don't have the same desire to keep the performance statistics up to date as upright people do.
The upshot is that you can't measure FNIR. Not in the field.
You can submit a batch of sample biometrics and see how well the system performs. How successful it is at finding these deliberately seeded duplicates on the register. And that's what UIDAI did (pp.18-19):
What we do know – G&C tell us – is that UIDAI have been "relaxing" the FNIR to keep FPIR low. The confidence we can have in UIDAI's figure for FNIR is severely limited.
G&C tell us on p.1 of their report that:
As G&C tell us, the experts conclude that technology tests and scenario tests tell us nothing about how well or how badly a biometrics system will perform in the operational environment. As they put it, biometrics is out of "statistical control".
To put it another way, UIDAI's FNIR and FPIR test probes are a waste of time.
Tony Mansfield is the UK's top biometrics authority and Jim Wayman is the US's. And Antonio Possolo is the top man on measurement at the US National Institute of Standards and Technology (NIST). They're practitioners. They have decades of experience. They advise governments. Their own and others. They know what they're talking about.
And what they're talking about is biometrics being out of statistical control.
That implies many things. Among others, consider the following.
Messrs Wayman, Possolo and Mansfield refer to the USA PATRIOT Act in their paper (p.20). By law, NIST have to certify biometrics systems before they are deployed in the national defence.
That may be the law but, if the technology is out of control then NIST have a problem obeying the law. They could refuse to certify any biometrics systems and then none would be deployed. That's one option. They have chosen another option. The certificate they issue says:
Final rider – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register and the fact that the discipline of biometrics is out of statistical control.
G&C work for the Center for Global Development, a Washington-based think-tank and lobbyist which aims to "reduce global poverty and inequality through rigorous research and active engagement with the policy community to make the world a more prosperous, just, and safe place for us all".
It's hard work finding good homes for aid money. There are legitimate doubts about the reliability of biometrics. Aid money isn't necessarily well spent on biometrics systems.
Michela Wrong, a journalist who has covered Africa for two decades, reported on the March 2013 elections in Kenya complete with biometric registration of electors and electronic voting. She had this to say:
That was some time ago. They remain dazzled by technology to this day: "India has registered 275m of its 1.2 billion people in one of the world’s most sophisticated ID schemes (it includes iris scans and fingerprints)". Why do they think that the inclusion of biometrics is ipso facto "sophisticated"?
They should talk to Michela Wrong.
The biometrics salesmen won't like that conclusion of G&C's and they won't mention it, please see UIDAI and the textbook case study of how not to do it, one for the business schools. (Neither will the UK government.)
All that healthy scepticism, and yet G&C conclude that biometrics is ready for large-scale deployment:
No.
It shows nothing of the sort.
Is there any chance of G&C reissuing their report with revised conclusions?
The suggestion below is that the conclusion should read: "subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register and the fact that the discipline of biometrics is out of statistical control".
Will the authors consider issuing a revised version of their report?
----- o O o -----
On the rare occasions when trials have been conducted, the performance of biometrics technology has been disappointing. For example, when 10,000 of us took part in a UK government-run trial in 2004, about 20% of participants couldn't have their identity verified by their fingerprints.
That's useless. For example, the plan at the time was to use biometrics to confirm people's right to work in the UK. You can't tell 20% of the working population that it's illegal for them to work.
Ever optimistic, the biometrics industry is always announcing that the corner has been turned and that it's safe now to believe their promises. Is that true at last?
Consider Performance Lessons from India’s Universal Identification Program [Updated 13.12.18, change of web address, now here], a 12-page report by Alan Gelb and Julia Clark (Gelb and Clark, G&C).
It's about India's Unique Identification project (UID, also known as "Aadhaar") which relies on biometrics. UID/Aadhaar is the brainchild of UIDAI, the Unique Identification Authority of India. UIDAI are currently trying to register the biometrics of all 1.2 billion Indians.
G&C conclude that:
Those conclusions are electric.
UID’s performance suggests that accurate, biometric-based, identification is quite feasible for large countries, including the US (p.8).
UID shows that countries with large populations can implement inclusive, precise, high-quality identity systems by using existing technology (p.9).
If they're correct.
But are they?
Why do G&C conclude that biometrics is now ready for large-scale deployment?
----- o O o -----
They have identified "160 [biometrics] programs in 70 countries that together cover over 1 billion people and include a wide range of applications – financial access, public payroll management, social transfers [?], health insurance and tracking and voter rolls – as well as national identification systems" (p.1).
Do they say that biometrics is ready for the big time because UIDAI have successfully implemented financial access systems which depend on biometrics? Or public payroll management systems? Or ...
Certainly not.
In fact G&C are at pains to say that:
Their logic doesn't depend on any practical successes of Aadhaar. There aren't any.
UID is still at a relatively early stage, and links to the delivery of public programs are only now getting under way (p.2).
It remains to be seen how robust the system is against active efforts to spoof it by providing faked fingerprints or iris images, to capture biometric data in transmission or to penetrate the database (p.2).
Having a unique Aadhaar number issued by UIDAI itself entitles the holder to no specific privileges or programs (p.3).
UID is still at an early stage. Only one fifth of the population has been enrolled and the linkage to public programs is just beginning (p.8).
What G&C base their conclusions on is the performance of biometrics in the compilation of the Indian population register so far. If we are to answer the question whether their conclusions are correct, we need to look at the UIDAI statistics which measure the reliability of biometrics.
Before we do that, we need to update G&C's conclusions. There's a rider to add. Their p.2 warnings about spoofing and eavesdropping on telecommunications and burgling the population register need to be incorporated – the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack.
----- o O o -----
Slide rules ready? G&C say (p.5):
Wrong.
How many people would be denied enrolment because of a wrong determination that they had already enrolled? The False Rejection Rate (FRR) of the identity system is critical, especially with a large population. Since every new enrollment has to be checked against every existing enrollment, the number of comparisons increases with the square of the population ... Extrapolating this to our hypothetical Ughana population ...
Think in terms of ice cream. How many unique combinations of two ice cream flavours can you make from a choice of five flavours (A, B, C, D and E)? G&C suggest that the answer is 25, "the square of the population". It isn't. It's 10 (AB, AC, AD, AE, BC, BD, BE, CD, CE and DE), 5!/((5-2)! x 2!).
G&C have a peculiar habit. They're talking about India, with its population of 1.2 billion, but half the time when they use statistics they apply them to Ughana, a country they have invented. Why?
It confuses the readers. It may also confuse the writers. Forswearing Ughana and sticking to India, how many comparisons would have to be made to compare each one of 1.2 billion sets of biometrics against all the rest? Answer, 719,999,999,400,000,000 and not G&C's implied answer 1,440,000,000,000,000,000, which is out by a smidgeon over 100%.
New rider on the conclusions – the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right.
----- o O o -----
How did G&C get themselves into this bind?It was in the midst of a discussion about false accept rates and false reject rates.
Aadhaar is all about comparing the biometrics captured by a fingerprint scanner or an iris scanner with the biometrics stored in the population register. Either they match or they don't.
Say your Aadhaar number is 782474317884, that there's an election on and that you have turned up at a voting centre. The biometrics associated with 782474317884 are retrieved from the population register and checked to see if they match your freshly scanned biometrics. If they do, you can vote. It's a one-to-one comparison, an "authentication" process.
Two ways the process can go wrong (among others):
- Either the process says the biometrics don't match, you are not who you claim to be, you are not President Lincoln according to Aadhaar, even though you are in reality. That's a false reject.
- Or, alternatively, the process can say that, yes, you are who you claim to be, you are President Lincoln, when, in fact, you're not, you're an impostor. That's a false accept.
Take a look at UIDAI's 27 March 2012 report on authentication (p.4). Using one or two fingers to authenticate yourself, UIDAI expect the Aadhaar system to be between 93.5% and 99% accurate. I.e. FRR will be between 1% and 6.5%. That's with a FAR of 0.01%. FRR is high, FAR is low(ish).
Varying FAR from high to low and FRR the other way is achieved by changing the matching threshold. You can set the system to insist on a very high score before asserting that President Lincoln's freshly scanned fingerprints match the set already stored on the population register. That would give a low FAR and a high FRR. Or you can set a very low threshold and achieve the opposite. And all points in between.
This is odd.
In the world we're used to, if you are President Lincoln then you are President Lincoln and that's all there is to it. It doesn't depend on the matching threshold set by some state functionary.
In the world of Aadhaar, depending on the threshold chosen, sometimes you will be President Lincoln (low threshold, easy to achieve a match, low FRR, high FAR) and sometimes you won't (high threshold, hard to achieve a match, high FRR, low FAR). It all depends. At the limit, the functionary could fix it so that no-one was President Lincoln. Or that everyone was.
When we said above that "either they match or they don't", that was a tease. That's the way people imagine biometrics systems to work. All cut and dried. In fact, it's discretionary.
The concept of identity in Aadhaar is different from the concept in the real world. Identity becomes discretionary, something that can be granted or revoked by twiddling the dial on a gizmo.
There's another rider to add to G&C's conclusions – the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed.
----- o O o -----
That's authentication.Identification is different.
Identification is the process you go through when you are enrolled into Aadhaar. Before identification, you don't exist as far as Aadhaar is concerned. If public services in India ever start to depend on Aadhaar and you don't have an Aadhaar number, you won't get any public services. Why would the state provide benefits to someone who doesn't exist? At the very least you will look very suspicious.
Identification is a one-to-many process. When you first enrol someone in the register, their biometrics have to be checked for uniqueness. Instead of checking them against just one set of biometrics, they have to be checked against every set already registered.
The errors that can be made by the biometrics system are very similar (among others, yes you are already enrolled when really you're not or no you're not already enrolled when really you are) but the process has such existential consequences that it's normal to talk of false negative identification rate (FNIR) and false positive identification rate (FPIR), rather than FAR and FRR, to distinguish it from mere quotidian authentication.
UIDAI talk of FRRs between 1% and 6.5% for authentication using fingerprints whereas, when it comes to identification, their FPIR figure is 0.057%. That's two orders of magnitude different. Identification is a strict process and, by comparison, authentication is flabby.
G&C unfortunately use FAR and FRR for both identification and authentication which obscures the important distinctions between the two processes.
----- o O o -----
FNIR and FPIR are inversely proportional, like FAR and FRR.How good are the biometrics UIDAI are using at creating a reliable population register?
It's a problem Professor John Daugman has looked at. Not in connection with Aadhaar in particular. But in general. For any biometrics-based identity management scheme.
Remember, to establish uniqueness for every one of the 1.2 billion sets of biometrics on India's population register, you have to make 719,999,999,400,000,000 comparisons.
Suppose, says Professor Daugman, that there's a mistake 1 time in a million such that a false positive identification is made. Then Aadhaar will throw up 719,999,999,400 false matches.
These can't be resolved by the computer – it's the computer that threw up the false matches in the first place. They have to be resolved by human investigations.
Humans aren't going to complete 719,999,999,400 investigations. It's impractical. The identity management scheme will drown in a sea of false positives, as the professor puts it.
Is there a one-in-a-million chance of a mistake?
Professor Daugman thinks that it's a lot worse than that if you rely on face recognition as a biometric. There's far too little randomness in faces, there are far too few degrees of freedom, for face recognition to support enormous numbers like 719,999,999,400,000,000. (Never mind Ughana, that doesn't stop the UK government wasting money on face recognition.)
Fingerprinting is better in this sense than face recognition, but still not good enough to avoid drowning in a sea of false positives. (That doesn't stop the UK government wasting money on glitzy new fingerprinting systems.)
Irises on the other hand do have enough randomness, he says, there are enough degrees of freedom to stay afloat. Which is good news for UIDAI – Aadhaar uses a combination of both fingerprints and irises.
----- o O o -----
Is Aadhaar in the clear? Which is it? Sink or swim?According to UIDAI's report on identification (p.4), on 31 December 2011 when there were 84 million sets of biometrics on the population register, the FPIR was 0.057%, the FNIR was 0.035% and "it is unnecessary and inaccurate to attempt to infer UIDAI system performance from other systems which are ten to thousand times smaller".
It may be unnecessary and it may be inaccurate but it's impossible to resist the temptation – compared to any other biometrics-based scheme known to man, these figures for Aadhaar are astonishing. Certainly no salesman worth his or her salt will ignore it.
It looks as if there would be only 684,000 false positive identifications to investigate by the time the population register is full, and not 719,999,999,400.
684,000 is manageable. As UIDAI say (p.18):
How do UIDAI know that the FPIR was 0.057% when the register had 84 million entries?
... at a run rate 10 lakhs enrolments a day, only about 570 cases need to be manually reviewed daily to ensure that no resident is erroneously denied an Aadhaar number. Although this number is expected to grow as the database size increases, it is not expected to exceed manageable values even at full enrolment of 120 crores. The UIDAI currently has a manual adjudication team thatreviews and resolves such cases.
[1 lakh = 100,000 and 1 crore = 10,000,000]
Presumably they had recorded 47,880 cases of false positive identifications to date.
You'd think that. But you'd be wrong. UIDAI tell us that (p.18):
They did a test. They probed the gallery with 4 million sets of biometrics and they got 2,309 false positive identifications.
An FPIR of 0.057% was measured when the gallery size was 8.4 crore (84 million) and probe size was 40 lakhs (4 million). The false rejects (legitimate residents who are falsely rejected by the biometric system) were a count of 2309 out of the 40 lakh probes
Funny way to do it.
Perhaps we shall be told that there's an agreed protocol in the biometrics industry such that this is an acceptable way of determining FPIR. Even so, why not report the actual number of false positive identifications recorded?
That statistic should be available in the case of Aadhaar – G&C tell us that (p.2):
Why not tell us how many false positive identifications there were as well as the result of the test probe? Why were there 4 million sets of biometrics in the probe and not 5 million, or 3 million? How were the 4 million chosen?
UIDAI places a heavy emphasis on data quality throughout the process. It collects as much operational data as possible, including on the details of each individual enrolment as it is carried out, process by process. This is included, together with biometric and demographic data, in the packet of information sent from the enrollment point to the data center.
The questions mount and the answer gradually comes into focus – in order to inspire confidence, UIDAI's figures need to be audited by independent experts and certified like a set of company accounts.
And, like company accounts, they should be audited every year. These figures from 31 December 2011 are getting very long in the tooth.
Another rider – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed.
----- o O o -----
UIDAI say that the incidence of false positive identifications is manageable and that they expect it to remain manageable. I.e. they're not drowning in a sea of false positives.G&C have this footnote, #7, on p.5 of their report:
"UIDAI plans to contain the numbers by ... relaxing the [FNIR] if needed to further reduce the [FPIR]". What? "Relaxing the [FNIR]"?
For a huge population like India’s, even this small level of error would result in some 3.1 million false rejections if continued through the program. UIDAI plans to contain the numbers by eliminating some sources of error unearthed by the initial study, and also by relaxing the [FNIR] if needed to further reduce the [FPIR]. Handling false rejections has reportedly been a manageable problem to date.
What does that mean? In order not to drown in false positives, UIDAI will let false negatives go up? UIDAI have got to get the population register completed and if that means tolerating lots of duplicate entries, too bad, so be it, let uniqueness go hang? If that isn't what it means, then what?
How relaxed? Very relaxed? What level does FNIR have to rise to, to keep FPIR down at 0.057%? Do UIDAI even know? Should they change their name to the Multiple Identification Authority of India?
"It is unnecessary and inaccurate to attempt to infer UIDAI system performance from other systems which are ten to thousand times smaller"? On the contrary, it is only sensible to question UIDAI's performance claims.
The riders are piling up now – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register.
----- o O o -----
If a supplicant turns up at an Aadhaar registration centre and is the victim of a false positive identification, you're going to know about it. They're going to demand their Aadhaar number and they're going to stay there and jump up and down until they get it. At least they will if they're legitimate and not impostors.It's different with false negative identification. If an impostor turns up at the centre and his or her earlier registration is not detected by Aadhaar, then they're not going to tell you. You won't know. Impostors don't have the same desire to keep the performance statistics up to date as upright people do.
The upshot is that you can't measure FNIR. Not in the field.
You can submit a batch of sample biometrics and see how well the system performs. How successful it is at finding these deliberately seeded duplicates on the register. And that's what UIDAI did (pp.18-19):
That's fine. But if the actual number of "duplicate submissions" is higher than UIDAI assume and the "false acceptances" are more numerous than they expect, no-one will know. All UIDAI can say is, when we did this test, we got this result. Whether that is an accurate measure of FNIR out there in the operational system in the real world, nobody knows.
To compute FNIR, 31,399 known duplicates were used as probe against gallery of 8.4 crore (84M). The biometric system correctly caught 31,388 duplicates (in other words, it did not catch 11 duplicates). The computed FNIR rate is 0.0352%. Assuming current 0.5% rate of duplicate submissions continues, there would only be a very small number of duplicate Aadhaars issued when the entire country of 120 crores is enrolled. Aadhaar expects to be able to increase the quality of all collections as the system matures. Consequently, we expect the potential number of false acceptances to decrease further below this already operationally satisfactory number.
What we do know – G&C tell us – is that UIDAI have been "relaxing" the FNIR to keep FPIR low. The confidence we can have in UIDAI's figure for FNIR is severely limited.
----- o O o -----
It's worse than that.G&C tell us on p.1 of their report that:
The paper they cite, Fundamental issues in biometric performance testing: A modern statistical and philosophical framework for uncertainty assessment, is written by three world-class experts – James L. Wayman, Antonio Possolo and Anthony J. Mansfield.
Although there has been extensive laboratory testing of different hardware and software for a variety of biometrics, including fingerprints, iris, face and voice, testing under carefully controlled conditions does not provide adequate information on real-world performance, which can be affected by many factors (Wayman et al 2010).
As G&C tell us, the experts conclude that technology tests and scenario tests tell us nothing about how well or how badly a biometrics system will perform in the operational environment. As they put it, biometrics is out of "statistical control".
To put it another way, UIDAI's FNIR and FPIR test probes are a waste of time.
Tony Mansfield is the UK's top biometrics authority and Jim Wayman is the US's. And Antonio Possolo is the top man on measurement at the US National Institute of Standards and Technology (NIST). They're practitioners. They have decades of experience. They advise governments. Their own and others. They know what they're talking about.
And what they're talking about is biometrics being out of statistical control.
That implies many things. Among others, consider the following.
Messrs Wayman, Possolo and Mansfield refer to the USA PATRIOT Act in their paper (p.20). By law, NIST have to certify biometrics systems before they are deployed in the national defence.
That may be the law but, if the technology is out of control then NIST have a problem obeying the law. They could refuse to certify any biometrics systems and then none would be deployed. That's one option. They have chosen another option. The certificate they issue says:
That's the best they can manage in the circumstances. The result of the test is the result of the test and that's all we know. How the system will perform in the field is anyone's guess. According to three world-class experts, in biometrics, that is the state of the art.
For purpose of NIST PATRIOT Act certification this test certifies the accuracy of the participating systems on the datasets used in the test. This evaluation does not certify that any of the systems tested meet the requirements of any specific government application. This would require that factors not included in this test such as image quality, dataset size, cost, and required response time be included.
Final rider – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register and the fact that the discipline of biometrics is out of statistical control.
----- o O o -----
It is premature to conclude that biometrics have proved themselves in Aadhaar:- Let's wait and see if any bank is confident enough to authorise payments on the basis of biometrics alone. No password. No PIN. No token. Just biometrics.
- Let's wait and see if legitimate voter participation is increased by Aadhaar.
- India's various food and fuel distribution programmes and its temporary employment programmes for the long-term unemployed are plagued by large-scale corruption. Let's wait and see if Aadhaar reduces the level of corruption or simply automates it.
- And let's wait for an independent audit of UIDAI's results.
G&C work for the Center for Global Development, a Washington-based think-tank and lobbyist which aims to "reduce global poverty and inequality through rigorous research and active engagement with the policy community to make the world a more prosperous, just, and safe place for us all".
It's hard work finding good homes for aid money. There are legitimate doubts about the reliability of biometrics. Aid money isn't necessarily well spent on biometrics systems.
Michela Wrong, a journalist who has covered Africa for two decades, reported on the March 2013 elections in Kenya complete with biometric registration of electors and electronic voting. She had this to say:
The Economist magazine have let down their scepticism guard and become active in the unsolicited testimonials market – please see The Economist magazine sticks its nose into Indian politics, comes away with egg on its face and The Economist magazine's chickens, now on their way home to roost.
I suddenly realised I was watching a fad hitting its stride: the techno-election as democratic panacea ... EU and Commonwealth election monitors hailed the system as a marvel of its kind, an advance certain to be rolled out across the rest of Africa and possibly Europe, too. The enthusiasm was baffling, because almost none of it worked.
That was some time ago. They remain dazzled by technology to this day: "India has registered 275m of its 1.2 billion people in one of the world’s most sophisticated ID schemes (it includes iris scans and fingerprints)". Why do they think that the inclusion of biometrics is ipso facto "sophisticated"?
They should talk to Michela Wrong.
----- o O o -----
G&C have spotted what the Economist have missed:- The Wayman, Possolo and Mansfield paper.
- UIDAI relaxing the FNIR.
- The element of smoke and mirrors in biometrics – they talk about the "fiction of infallibility" (p.9) and the "pretense of uniqueness in the ID system" (p.10) and the possibility that "in the longer run, as its mystique evaporates, the identity system will no longer be trusted by anyone, eliminating any value" (p.10).
The biometrics salesmen won't like that conclusion of G&C's and they won't mention it, please see UIDAI and the textbook case study of how not to do it, one for the business schools. (Neither will the UK government.)
All that healthy scepticism, and yet G&C conclude that biometrics is ready for large-scale deployment:
- Did they check with NIST or the FBI before publishing their report? Those organisations know quite a lot about biometrics and might have provided some useful input.
- Did they contact Messrs Wayman, Possolo and Mansfield? If G&C believe them when they say that biometrics is out of statistical control, then there's not much point filling up their report with useless statistics, is there? If they don't believe them, why not?
- Would G&C be so generous with their testimonials if Aadhaar was an aeroplane safety system, for example?
- Would they feel qualified to comment if they were dealing with the pharmaceutical industry rather than the biometrics industry?
- Would they be more sceptical if they were dealing with research funded by the tobacco industry?
- Why does biometrics get the kid gloves treatment?
- And what is this fake distinction G&C make between countries with a large population and a small one? The biometrics tested in the UK failed with a trial population of 10,000 participants. Biometrics is a technology. At least it's supposed to be. Either it works or it doesn't. Cars work in the US. And they work in India. If biometrics isn't good enough for the US, it's not good enough for India. Or Uganda or Ghana. Which are two different countries. Ask Michela Wrong.
No.
It shows nothing of the sort.
Is there any chance of G&C reissuing their report with revised conclusions?
Biometrics: will the Center for Global Development reconsider?
A recently published report on India's identity management scheme says that: "accurate, biometric-based, identification is quite feasible for large countries, including the US".
The suggestion below is that the conclusion should read: "subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register and the fact that the discipline of biometrics is out of statistical control".
Will the authors consider issuing a revised version of their report?
On the rare occasions when trials have been conducted, the performance of biometrics technology has been disappointing. For example, when 10,000 of us took part in a UK government-run trial in 2004, about 20% of participants couldn't have their identity verified by their fingerprints.
The suggestion below is that the conclusion should read: "subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register and the fact that the discipline of biometrics is out of statistical control".
Will the authors consider issuing a revised version of their report?
----- o O o -----
On the rare occasions when trials have been conducted, the performance of biometrics technology has been disappointing. For example, when 10,000 of us took part in a UK government-run trial in 2004, about 20% of participants couldn't have their identity verified by their fingerprints.
Tuesday 7 May 2013
The war of independence
Here in the UK, the Disability Living Allowance (DLA) state benefit is in the process of being replaced by the Personal Independence Payment (PIP), worth between £21 and £134.40 per week. "PIP helps with some of the extra costs caused by long-term ill-health or a disability", which is good, but the whole point is that many claimants will be dependent on this benefit which makes it very odd to call it a "Personal Independence Payment". Something odd is happening to the concept of independence here.
The same odd thing is happening to the concept of control. Mr Andrew Dilnot, chairman of the UK Statistics Authority, argues that there should be a limit of £35,000 on the amount old people pay for their social care. That limit should apply even if an old person owns a house, say, worth £2 million, say. They shouldn't have to sell the house to pay for their care.
Why not?
So that they can leave the house to their children? No, says Mr Dilnot, that's not the point. The point is, he says, that with the state paying for all their care after the first £35,000, old people will have "control over their lives at a time when they're vulnerable and need that control".
No.
Far from granting control, Mr Dilnot's proposal will take it away. If someone else is paying for your care home, they have control and you don't. Yield that control, and your independence goes with it.
That's not the case just for care homes, of course. You don't pay to use Google or Facebook, do you. You don't have any control over what they do with all the personal information about you that they collect. You're in no position to complain. You take what you get. Because you're dependent on them.
Google, by contrast, dispose of considerable power. Just ask Interflora UK. They displeased Google by trying to "game" the search engine's PageRanking system. So Google just omitted them from any search results. With its on-line sales threatened, a contrite Interflora UK swore obedience and was subsequently readmitted to the fold.
That's real power. Power that some web evangelists ignore. Douglas Carswell, for example, and his curate's egg of a book, The End of Politics and the Birth of iDemocracy. Our politicians and our civil servants here in the UK might be ever so useless, yes, but to believe that we could be returned to some prelapsarian iDemocratic idyll if only the government were replaced by the web is to ignore the power of the Googles of this world.
Taking a holiday some years ago from their normal diet of heavy-handed legislation and irritating regulation, so much of it ineffectual, look at the banking world, our politicians got the idea that perhaps they could exert power using the wily tricks of the marketing world. Thus was born Whitehall's Behavioural Insights Team, who were meant to "nudge" us into doing certain things, no legislation required, just clever psychology.
Fat chance.
The Behavioural Insights Team worked with the Department for Business Innovation and Skills (BIS) on the midata project. The idea was to nudge retailers into releasing data back to consumers so that we would all be "empowered". Result? No cigar, so BIS asked parliament for statutory powers to underpin midata. And with that resounding failure to nudge, the Behavioural Insights team have been kicked out of Whitehall and adopted by Capita. Good luck, Capita.
BIS got their statutory powers the other day, with the passing of the Enterprise and Regulatory Reform Act 2013. "Regulatory reform" in the UK is supposed to imply the reduction of government regulation. midata is dealt with at clauses 85-87 of the Act and what do we find at 86(1)?
We should have known.
In the midata consultation document last year BIS and the Behavioural Insights Team said, para.4, p.11:
Responses to that consultation had to be returned to a Mr Craig Belsham at BIS by 10 September 2012. He popped up again the other day:
The Government Digital Service heaped praise on themselves last month for completing the project to bring all central government departmental websites into one single government domain, the award-winning GOV.UK. http://blogs.bis.gov.uk/midata/ shouldn't exist. But it does, and there's Mr Belsham to prove it.
And does Mr Belsham really "head up" the midata programme?
It's not in Mydex's power to grant that control. It's an odd view of control anyway. You get control of your data back, so goes the Mydex argument, by storing it all with them, complete strangers, in a personal data store, out of your control, on the web, in the cloud, the Wild West where – so Symantec tell us – 250,000 cyber-attacks take place every year.
Normally that wouldn't make sense.
But if you're surrounded
The same odd thing is happening to the concept of control. Mr Andrew Dilnot, chairman of the UK Statistics Authority, argues that there should be a limit of £35,000 on the amount old people pay for their social care. That limit should apply even if an old person owns a house, say, worth £2 million, say. They shouldn't have to sell the house to pay for their care.
Why not?
So that they can leave the house to their children? No, says Mr Dilnot, that's not the point. The point is, he says, that with the state paying for all their care after the first £35,000, old people will have "control over their lives at a time when they're vulnerable and need that control".
No.
Far from granting control, Mr Dilnot's proposal will take it away. If someone else is paying for your care home, they have control and you don't. Yield that control, and your independence goes with it.
That's not the case just for care homes, of course. You don't pay to use Google or Facebook, do you. You don't have any control over what they do with all the personal information about you that they collect. You're in no position to complain. You take what you get. Because you're dependent on them.
Google, by contrast, dispose of considerable power. Just ask Interflora UK. They displeased Google by trying to "game" the search engine's PageRanking system. So Google just omitted them from any search results. With its on-line sales threatened, a contrite Interflora UK swore obedience and was subsequently readmitted to the fold.
That's real power. Power that some web evangelists ignore. Douglas Carswell, for example, and his curate's egg of a book, The End of Politics and the Birth of iDemocracy. Our politicians and our civil servants here in the UK might be ever so useless, yes, but to believe that we could be returned to some prelapsarian iDemocratic idyll if only the government were replaced by the web is to ignore the power of the Googles of this world.
Taking a holiday some years ago from their normal diet of heavy-handed legislation and irritating regulation, so much of it ineffectual, look at the banking world, our politicians got the idea that perhaps they could exert power using the wily tricks of the marketing world. Thus was born Whitehall's Behavioural Insights Team, who were meant to "nudge" us into doing certain things, no legislation required, just clever psychology.
Fat chance.
The Behavioural Insights Team worked with the Department for Business Innovation and Skills (BIS) on the midata project. The idea was to nudge retailers into releasing data back to consumers so that we would all be "empowered". Result? No cigar, so BIS asked parliament for statutory powers to underpin midata. And with that resounding failure to nudge, the Behavioural Insights team have been kicked out of Whitehall and adopted by Capita. Good luck, Capita.
BIS got their statutory powers the other day, with the passing of the Enterprise and Regulatory Reform Act 2013. "Regulatory reform" in the UK is supposed to imply the reduction of government regulation. midata is dealt with at clauses 85-87 of the Act and what do we find at 86(1)?
Far from reducing regulation, the Act will increase it.
Regulations may make provision for the enforcement of regulations under section 85 (“customer data regulations”) by the Information Commissioner or any other person specified in the regulations (and, in this section, “enforcer” means a person on whom functions of enforcement are conferred by the regulations).
We should have known.
In the midata consultation document last year BIS and the Behavioural Insights Team said, para.4, p.11:
In what sense can this new regulation have a deregulatory effect? Grant us this power to regulate data access by consumers, BIS answered, otherwise we'll make your life hell with a lot of other regulations about product specification – the logic of the protection racket.
Increased data transparency and greater consumer choice will help promote innovation and competition and could also have a deregulatory effect. By giving people access to their data in a format which is machine readable it may be possible to avoid the need for some types of regulation, for example, specifying product characteristics.
Responses to that consultation had to be returned to a Mr Craig Belsham at BIS by 10 September 2012. He popped up again the other day:
It's not really a blog. You can't submit comments. And there's no feed – you can't add http://blogs.bis.gov.uk/midata/ to a blogroll.
Welcome to my new blog about midata
2 May 2013 Craig Belsham
I’m Craig Belsham from BIS, where I head up the midata programme. This blog is designed to give some insight into that programme, help people and business understand it and hopefully encourage both to start to get involved.
The Government Digital Service heaped praise on themselves last month for completing the project to bring all central government departmental websites into one single government domain, the award-winning GOV.UK. http://blogs.bis.gov.uk/midata/ shouldn't exist. But it does, and there's Mr Belsham to prove it.
And does Mr Belsham really "head up" the midata programme?
- What's happened to Kirstin Green, the deputy director at BIS who led the open forums on midata?
- What's happened to Professor Nigel Shadbolt, chair of the midata programme?
- And what's happened to William Heath, member of the midata strategy board and chairman of Mydex, the only system ever mentioned in connection with the personal data stores BIS want us all to maintain?
It's not in Mydex's power to grant that control. It's an odd view of control anyway. You get control of your data back, so goes the Mydex argument, by storing it all with them, complete strangers, in a personal data store, out of your control, on the web, in the cloud, the Wild West where – so Symantec tell us – 250,000 cyber-attacks take place every year.
Normally that wouldn't make sense.
But if you're surrounded
- by people who call a personal dependence payment a "personal independence payment"
- and who argue that you stay in control of your personal care by giving up control of it
- and who write that politics has ended when clearly it hasn't
- and who conclude that the way to nudge people is to legislate
- and who claim that regulation can have a deregulatory effect
- and who operate a blog that isn't a blog on a website that doesn't exist
- and who represent themselves as the head of a programme when perhaps they're not
- and who congratulate themselves on the completion of a project which is manifestly incomplete
- to take back control of your personal data by giving it away.
The war of independence
Here in the UK, the Disability Living Allowance (DLA) state benefit is in the process of being replaced by the Personal Independence Payment (PIP), worth between £21 and £134.40 per week. "PIP helps with some of the extra costs caused by long-term ill-health or a disability", which is good, but the whole point is that many claimants will be dependent on this benefit which makes it very odd to call it a "Personal Independence Payment". Something odd is happening to the concept of independence here.
The same odd thing is happening to the concept of control. Mr Andrew Dilnot, chairman of the UK Statistics Authority, argues that there should be a limit of £35,000 on the amount old people pay for their social care. That limit should apply even if an old person owns a house, say, worth £2 million, say. They shouldn't have to sell the house to pay for their care.
Why not?
The same odd thing is happening to the concept of control. Mr Andrew Dilnot, chairman of the UK Statistics Authority, argues that there should be a limit of £35,000 on the amount old people pay for their social care. That limit should apply even if an old person owns a house, say, worth £2 million, say. They shouldn't have to sell the house to pay for their care.
Why not?