Sunday, 26 May 2013

Biometrics – the tiger the Center for Global Development has caught by the tail (updated)

Conclusion
The case for investing in the nationwide deployment of biometrics has not been made.


Background
In their 7 May 2013 report Performance Lessons from India’s Universal Identification Program one of the lessons that Alan Gelb and Julia Clark (G&C) draw from UID (also known as "Aadhaar") is that ...
UID’s performance suggests that accurate, biometric-based, identification is quite feasible for large countries, including the US. (p.8)
... restated a page later as ...
UID shows that countries with large populations can implement inclusive, precise, high-quality identity systems by using existing technology. (p.9)
In his 12 May 2013 blog post Biometrics: will the Center for Global Development reconsider? DMossEsq suggested that this conclusion of G&C's needs to be qualified in at least six ways and should read "the US could safely deploy an identity management scheme based on biometrics":
  1. "subject to an annual audit"
  2. "apart from the possibility of cyberattack"
  3. "and as long as we've got our maths right"
  4. "and as long as you realise that it's not identity that's being managed"
  5. "and as long as you're relaxed about the fact that anyone could have any number of entries on the population register"
  6. "and the fact that the discipline of biometrics is out of statistical control"
On 21 May 2013, Alan Gelb posted a comment, which includes this:
... we hold to our conclusion that the data released provides a very significant benchmark on the capabilities of biometric systems in developing country conditions and one that should be studied carefully by other countries.

Some evidence of reconsideration
But that wasn't their conclusion.

Their conclusion was that the usefulness of biometrics to the US and other countries has already been "shown" or demonstrated or established by Aadhaar.

They're not holding to that.

Now, it transpires, the evidence of Aadhaar is insufficient. Something more is needed – careful study – before the usefulness of today's biometrics to the US is established. We cannot yet say, pace G&C's earlier report, that its usefulness has been demonstrated.

What was G&C's original conclusion based on if not careful study?


Audit
In his comment, Mr Gelb ignores the point about the need for an audit of the biometrics performance figures published by UIDAI, the Unique Identification Authority of India.

A striking omission, G&C are endorsing India's investment in biometrics and recommending the same for the US without first getting an independent expert audit of the performance figures. That would be imprudent behaviour for a responsible investment manager.

G&C are convinced that Aadhaar will be beneficial to the millions of Indians whose prospects of escaping poverty are limited for lack of an official identity. Why are they convinced? Is it any more than a hunch or a hope?

They're not convinced because of any government programmes which depend on Aadhaar – as Mr Gelb says:
It is far too early to assess the UID program record in delivering more effective and inclusive services.
Their conviction relies exclusively on the enrolment of people into UIDAI's population register, where they are identified by their biometrics:
... we see the data that it [UIDAI] has released on inclusion and accuracy as a very significant benchmark for biometric systems in developing countries, and a major advance on the use of laboratory data. These appear to be the most extensive field data released so far.
Without an audit, how do G&C know that India's excluded millions really are being granted an identity? Has a benchmark been established? The US doesn't have the same social exclusion problem as India according to G&C so why the interest in using biometrics to identify all Americans?

The Indians and the Americans and everyone else would be well-advised to insist on an audit before any more of their money is invested in biometrics.


Statistical control
G&C cite a paper by three world-class experts, Messrs Wayman, Possolo and Mansfield (WP&M), which argues that the study of biometrics is out of statistical control – biometrics isn't a scientific discipline.

Their case rests on audits of biometrics systems that the three of them have conducted.

You can examine all the test results you like, WP&M say, but those results will tell you nothing about how biometrics systems will perform in the field, in operational use.

They discuss the implications for US homeland security. The National Institute of Standards and Technology (NIST) has a duty under the USA PATRIOT Act to audit biometrics systems and to certify them. The best NIST can manage is to say that the results of the tests they performed are the results of the tests they performed. They can't predict how the systems will perform in the field. No benefits to homeland security can be assured.

The same audit report on Aadhaar's performance figures would dissipate the will to invest in biometrics, whether in India, the US or anywhere else.

G&C rest their pro-investment case on the Aadhaar figures for False Positive Identification Rate (FPIR) and False Negative Identification Rate (FNIR). It is on the basis of two statistics that they recommend investment in biometrics, a technology which WP&M say is out of statistical control.

Look again at the back end of the quotation above:
... we see the data that [UIDAI] has released [as] ... a major advance on the use of laboratory data. These appear to be the most extensive field data released so far.
That is simply false.

You can't measure FNIR in the field. For the reason noted in the DMossEsq blog post – impostors don't come back and tell you that they fooled the system.

So where does UIDAI's figure of 0.0352% for FNIR come from?

They tell us. In their report, Role of Biometric Technology in Aadhaar Enrollment. On pp.18-19. It's the result of a laboratory test:
False accept (FNIR): To compute FNIR, 31,399 known duplicates were used as probe against gallery of 8.4 crore (84M). The biometric system correctly caught 31,388 duplicates (in other words, it did not catch 11 duplicates). The computed FNIR rate is 0.0352%. Assuming current 0.5% rate of duplicate submissions continues, there would only be a very small number of duplicate Aadhaars issued when the entire country of 120 crores is enrolled.
UIDAI's figure of 0.057% for FPIR is also the result of a laboratory test (p.18).

What Mr Gelb calls "field data" three times in his comment is, in each case, laboratory data – data which WP&M say tells us nothing about how Aadhaar will work in the field.

It's not just WP&M who cast doubt on these statistics. So do G&C themselves, when they note that UIDAI have to "relax" the FNIR to keep the FPIR down to manageable proportions, to avoid "drowning in a sea of false positives". With their butcher's thumb on the scales, UIDAI can make the meat weigh whatever they want. Or, dropping the butcher analogy, by varying the matching threshold, UIDAI can choose whatever FPIR they like.

Whatever these FPIR and FNIR statistics are, one thing is clear – they're not a benchmark. UIDAI have chosen 0.057% for the FPIR and they're sticking to it. It doesn't matter how well Aadhaar performs or how badly, the FPIR will always be 0.057%.


Maths
Mr Gelb says in his comment:
To correct the record, we do not assert that the number of bilateral comparisons is the square of the population, n. It is 0.5*n*(n-1) which rises (as we note) with the square of n.
He is saying that the number of matches rises with 0.5*n*(n-1) and that it rises with n². Since 0.5*n*(n-1) is not equal to n² that must be false.

He also says:
...since no identification system will cover 100% of population, we rounded n off to 1 billion for India.
Why 1 billion? Why not 0.8 billion? Or π/5 billion?

Mr Gelb's aim is to prove that the number of false positives generated by Aadhaar is and will remain manageable. There's no need to do any maths to prove that – not when you know that UIDAI have already decided that the FPIR is and always will be 0.057% and therefore is and always will be manageable. It's a management decision and not a scientific observation.


Multiple identities
G&C acknowledge that there is a trade-off between FPIR and FNIR.

In his comment, Mr Gelb says that:
If we accept the field estimate of 0.057% false positive rate against a data base of 84 million, the rate for a 1:1 comparison would have to be very small, in the range of 7 in one trillion.
Hard to understand, it looks as though he is saying that there will be only 7 false positives for every trillion matches. That can't be what he means but, roll with it for the moment, if he is saying that false positives will be at any sort of rock bottom level like 7 per trillion, then he must accept that false negatives will be sky high. That's what the trade-off means.

It means that Aadhaar's population register will be crammed full of people with multiple identities.

If any government programmes do start to rely on Aadhaar, then some individuals will be entitled to multiple votes, multiple food rations, multiple fuel allowances, multiple temporary jobs and multiple bank accounts. And if the banks start to rely on biometrics alone to authorise payments, then some individuals will be entitled to multiple benefit payments.


Cyberattack
That means fraud. Large-scale multiple identities in Aadhaar means large-scale fraud. If Mr Gelb is right about the statistics, then Aadhaar is a machine to automate corruption.

The Indian media openly acknowledge the high incidence of corruption in India's current food security and other welfare programmes. Not just the Indian press. The Economist, too. In a staggeringly awful article they wrote:
Armed with the system [Aadhaar], India will be able to rethink the nature of its welfare state, cutting back on benefits in kind and market-distorting subsidies, and turning to cash transfers paid directly into the bank accounts of the neediest. Hundreds of millions of the poor must open bank accounts, which is all to the good, because it will bind them into the modern economy. Care must be taken so mothers rather than feckless fathers control funds for their children ...

Mr Nilekani [UIDAI chairman] harnessed the genius of Indians abroad, including a man who helped the New York Stock Exchange crunch its numbers and one of the brains behind WebMD, an American health IT firm ...

India plainly needs better data-protection laws, but even if the existing rules remained unchanged, the threat to liberty would be dwarfed by the gains to welfare: to people who live ten to a room, concerns about privacy sound outlandish.

Some of the resistance is principled, but much comes from the people who do well out of today’s filthy system. Indian politics hinge on patronage—the doling out of opportunities to rob one’s countrymen. [Aadhaar] would make this harder. That is why it faces such fierce opposition, and why it could transform India.
Indian fathers are feckless? Emigré Indians are clever and the stay-at-home ones are dim? Poor people don't need privacy the way Economist journalists, for example, do? "Today's filthy system"? This is the case for Aadhaar put by someone who despises India.

Along with the Economist's contempt for the Indians goes a crippling naïvety. Why would Aadhaar make corruption harder? Aadhaar could simply automate corruption. It could increase the incidence of corruption, not reduce it.

At the limit, with their butcher's thumb on the scales, UIDAI – or whoever controls Aadhaar, perhaps a cyberattacker – could choose whatever party they like to be the winner of a general election. Please see for example this cautionary tale in the Washington PostHacker infiltration ends D.C. online voting trial.


Investment
It is wrong to insist on 100% accuracy, Mr Gelb says:
On multiple identities, no system will be able to guarantee 100 percent accuracy. Certainly not the systems in place in the rich countries where identity theft is hardly unknown! The question is not “whether it works or not” but the precision of one system versus another and relative cost-effectiveness. For some applications, such as access to a health insurance program, one might accept a modest level of duplicate or false identities. For others ...
The question is not whether it works or not ...

This looks like a call to be pragmatic.

This is the case you make for investment when you have had to abandon all the unconvincing statistics and unfulfilled promises that bedevil the biometrics industry.


Risk
There is no need whatever for G&C to take the risk of endorsing biometrics. So why take it?

Their report is published by the Center for Global Development (CGD). What are G&C committing CGD to?

Publishing the bald assertoric statement "UID shows that countries with large populations can implement inclusive, precise, high-quality identity systems by using existing technology" opens CGD to the risk that biometrics salesmen will plant stories in the press with lurid headlines like:
"The time has come for the US to do its duty and deploy biometrics for all", biometrics experts Gelb and Clark, of the internationally respected Capitol Hill Center for Global Development
To be clear, that headline is invented to make a point.

This one isn't – Paper highlights positive biometrics role in developing countries:
The research underpinning the paper was performed by Alan Gelb and Julia Clark at the Center for Global Development. According to Gelb and Clark, civil registration systems are often absent or cover only a fraction of the population. In contrast, people in rich countries are almost all well identified from birth. This “identity gap” is increasingly recognized as not only a symptom of underdevelopment but as a factor that makes development more difficult and less inclusive.
That article appeared on the Planet Biometrics website on 15 February 2013 and, to be clear again, it concerns an earlier report by G&C, not the one being discussed here.

Planet Biometrics is a marketing organisation for the biometrics industry. CGD is already being co-opted, thanks to G&C's product endorsements, into the worldwide (planetary?) promotion of the biometrics industry.

"Caught in a dragnet", said the headline, 17 July 2011:
John H. Gass hadn’t had a traffic ticket in years, so the Natick resident was surprised this spring when he received a letter from the Massachusetts Registry of Motor Vehicles informing him to cease driving because his license had been revoked ...

It turned out Gass was flagged because he looks like another driver, not because his image was being used to create a fake identity. His driving privileges were returned but, he alleges in a lawsuit, only after 10 days of bureaucratic wrangling to prove he is who he says he is ...

At least 34 states are using such systems. They help authorities verify a person’s claimed identity and track down people who have multiple licenses under different aliases, such as underage people wanting to buy alcohol, people with previous license suspensions, and people with criminal records trying to evade the law. Lisa Cradit, a spokeswoman for L-1 Identity Solutions, the largest developer of the software, said it can reduce fraud by 80 percent.
With CGD's name associated with biometrics, next time the headline could read:
Caught in Center for Global Development biometrics dragnet
You may say that that won't happen. G&C/CGD endorse composite fingerprint/iris scan biometrics, not face recognition. They're quite different propositions.

Two problems with that.

Firstly, to the mainstream media and the general public, not to mention legislators and public administrators, a biometric is a biometric is a biometric – the distinction won't come across.

Second, US-VISIT uses face recognition and fingerprints, not iris scans. How long before you see the headline:
"India has better security systems than Uncle Sam", Center for Global Development. Napolitano erupts
No doubt CGD has enough staff to defend its reputation if and when the tulipmania passes and the world falls out of love with biometrics. But why get involved in the first place?

----------

Updated:

5 June 2013, 19:02
Remember what Mr Gelb said, quite rightly:
It is far too early to assess the UID program record in delivering more effective and inclusive services.
That hasn't stopped the IT magazine ComputerWorld going for broke in the product endorsement stakes:
Computerworld Honors 2013: ID program empowers citizens in India
Government program, the 21st Century Achievement Award winner for economic development, uses biometrics to assign unique identity numbers, allowing residents of India to participate more fully in society.
ComputerWorld have jumped the gun. UIDAI are getting an award for doing something they haven't done yet. Aadhaar hasn't empowered the citizens of India. UIDAI promise that it will, one day, in the future. Even they don't claim that it already has. What possessed ComputerWorld?


18 June 2013

Premature: Computerworld Honors 2013: ID program empowers citizens in India

Not for India either: The Indian experiment is not for us

2 comments:

Anonymous said...

David Moss, a person I have come to know, is correct is his position that "errors" and "error rates" are not real life applicable but only laboratory suggested "error rates".

Using multiple biometrics, multi-modal assumes people are compatible to the use of the both biometrics. Many people, 10% roughly cannot be cannot have their fingerprints captured because of worn down finger tips and other reasons. This would then allow for facial recognition to be the "determining" biometric. In real life situation where lighting, shading and other "factors" are "in play" facial recognition does not have close to a 90% accuracy rate in what is called a "one to many" application.

I will always continue to defer to David on the "math" issues. That being said, as a person who know from personal experience that there are other "issues" that must be considered. These issues go to the heart of the question of the ethical use of biometrics.

While David focuses on the question on "can we trust the technology" we must also ask can we trust the vendor who provides the technology and the end user of the technology.

The biometrics industry has a long record of intentionally hyping the technology past the point of what the technology can actually do. In addition, since those in the biometrics industry are often relied on to determine testing parameters, there has been occasion for those parameters have been set to ensure a desired result. In addition, the biometrics industry has too often been the subject of criminal and civil investigations. One should

The end-user of the technology for the sake of my comment is "government". Very often, too often in fact, we hear that biometrics is effective as a deterrent. While on the surface one may believe this is a good reason to use biometrics, it is not. Saying biometrics is or can be a deterrent is admitting that accuracy is not what those in government and the biometrics industry would like us to believe. The real life consequences of this is that many people will be impacted negatively by the results of the use of biometrics because the "greater good" will be served. The greater good is "bad people" will not do whatever the may do if biometrics are used. The fear of biometrics is enough to stop bad people from doing bad things. This "thinking" has not ever been proven to be true. Bad people, more often than not, do bad things of ten without thinking of the consequences of then and or if they get caught.

For me personally the question not being asked is why would any people ever support giving government the ability to obtain total control over the people. Biometrics are the lynchpin of a surveillance society. In a surveillance society there is a 24/7 digital footprint that is linked to every person. Not every leader one could argue would abuse such power. I would agree that not every leader would but it would only take one to alter history and thus freedom forever.

Mark Lerner

David Moss said...

Dear Mark

Good to hear from you, as ever, and thank you for your comment.

You raise the question of whether the administration is bluffing with biometrics. Whether it’s in the US, India or anywhere else, are they holding 7-high with no straight and no flush and just hoping that biometrics will be a deterrent?

Don’t let’s forget, Alan Gelb and Julia Clark are very sound on this issue in their report. They talk about the “fiction of infallibility” (p.9) and the "pretense of uniqueness in the ID system" (p.10) and the possibility that "in the longer run, as its mystique evaporates, the identity system will no longer be trusted by anyone, eliminating any value" (p.10).

Quite rightly, you also raise the big issues of trust, power, control, freedom and the "real" intentions behind the worldwide push for biometrics.

In the absence of any evidence to the contrary it is wrong to impugn their motivation, we have to assume that people mean what they say. Politicians want to deploy biometrics because they really do believe it will help. So do public administrators and aid agencies and biometrics suppliers.

In good company, over dinner and with the wine flowing, the discussion of the dynamics of political power can go anywhere. Absolutely anywhere. Without limit. It’s an endlessly fascinating and difficult subject.

While we’re waiting for a definitive answer, let’s do something easy and uncontroversial and sort out a simple question. Do mass consumer biometrics work? No.

Suppose the case for deploying biometrics in India is a little shack built on the edge of a cliff.

Most of the shack is already hanging over the edge, in mid air. The ground promised by the architect of the shack doesn’t exist – there are no government programmes using Aadhaar to provide the foundations.

The shack is desperately holding on to the cliff edge by its fingernails – two numbers, 0.00035 and 0.00057. Those numbers are old and rusted and cracked and weak and giving way. One puff of wind and the shack will be over the cliff and into the drink.

Again, Alan Gelb and Julia Clark are very sound on this issue. It’s performance in the field that matters and they say that the results, like these, of laboratory tests tell you nothing about performance in the field. What’s more, they say that 0.00035 has already sheared off – it’s been "relaxed", to use their word – leaving only 0.00057.

Funnily enough, "aadhaar" means support or foundation. Aadhaar has no aadhaar?

Sensible people of good will need little further stimulation to abandon the shack and look for dependable accommodation for 1.2 billion Indians built on firm foundations.

Best wishes
dm

Post a comment