Biometric system performance evaluation

Dr. M. C. King

Date: August 29th, 2017

Design and Performance Evaluation

Four basic building blocks

A biometrics system is usually built with the

following components:

• sensor

• feature extractor

• database

• matcher

Biometrics: A Pattern Recognition System

• False accept rate (FAR): Proportion of imposters accepted

• False reject rate (FRR): Proportion of genuine users rejected

• Failure to enroll rate (FTE)

• Failure to acquire rate (FTA)

Feature Extractor

Template Database

Authentication Enrollment

Matcher

(Threshold)

Yes/No

Preprocessor Preprocessor

Identity

Operation of a Biometric System • The sensor acquires the raw biometric data

• The feature extractor processes the biometric data to extract a compact feature set that represents the identity of the individual

▪ When this feature set is placed in the database it is known as a template

• The matcher compares two such feature sets and generates a match score

• The decision module uses the match scores to determine the identity or verify the claimed identity

Enrollment

• The process of creating and including an identity in a database

• An individual’s biometric traits are extracted and stored in a database (possibly) along with some biographic information

• Database can be centralized or decentralized

http://www.cl.cam.ac.uk/~jgd1000/SchipholEnrollment.jpg

Smart card

• To ensure total personal privacy of employees' fingerprint database, fingerprint templates can now be stored on a Smart Card

• With Smart Cards, the biometric fingerprint template is stored on the card's internal memory only. The scanned fingerprint is compared with the template stored within the card

Taken from http://www.smesolutions.com.au/index.php?action=ProductsAndServices/IDCards

Example of Templates

Hand features:

Length & width of

fingers, width of palm

Face features:

(a) PCA-coefficients

(b) LDA-coefficients

Fingerprint features:

Minutiae coordinates and

local ridge orientation

Match Scores

• The output of a biometric matcher is the match score, (typically a single number), that quantifies the similarity between the input and the database template representations

• The higher the score (similarity score), the more certain is the system that the two biometric measurements come from the same person

Genuine and Impostor Match Scores

• The identity claim is usually made through the use of a Personal Identification Number (PIN), a user name, or a token (e.g., smart card)

• Genuine score: Match score obtained when two biometric samples from the same individual are compared

• Impostor score: Match score obtained when two biometric samples originating from different individuals are compared

• Therefore, a genuine score should be greater than an impostor score

• A threshold is used by the biometric system to determine if a score is genuine or impostor

Data

• Consider a scenario where the biometric data corresponding to

N users is acquired. Further, assume that each user is asked to

provide t samples of their biometric data.

• A pair of samples from the same user have to be compared

using the matcher to generate a genuine match score;

• A pair of samples from two different users have to be

compared to generate an impostor match score.

• Using this biometric data, a total of Nt(t − 1)/2 genuine scores

and (N(N − 1)t2)/2 impostor scores can be generated by the

matcher.

• It is assumed that the matcher is symmetric in the sense that

comparison of sample A against B gives the same score as the

comparison of B against A.

Two-class system

• Verification can be considered as a two‐category classification problem

• Given a claimed identity I and a query feature set xA, we need to decide if (I, xA) belongs to “genuine” or “impostor” class

• Let y be the stored template corresponding to identity I. Typically, xA is compared with y and a match score s, which measures the similarity between xA and y, is computed. The decision rule is given by

Cont’d

• (I, xA) ∈ genuine, if s ≥η,

impostor, if s <η,

• where η is a pre‐defined threshold

• If a distance score is used in place of the

similarity or match score, the inequalities in

the decision rule should be reversed

• When the identity claim is deemed to be

“genuine”, the user is allowed to access the

services provided by the system

Match Score Distributions

Match Score Distributions

Error Rates

FNMR and FMR are generally referred to as False Reject Rate (FRR) and False Accept Rate (FAR), respectively

False Accept Rate (FAR):

▪ The fraction (or percentage) of impostor scores greater than the threshold

▪ It represents the probability that an impostor will be incorrectly matched with a certain identity at a given threshold

▪ A low FAR is required in high secure systems such as nuclear facilities

False Reject Rate (FRR):

▪ The fraction (or percentage) of genuine scores lower than the threshold

▪ It represents the probability that a genuine user will be incorrectly rejected when claiming his/her true identity

▪ A low FRR is required in systems which focus on user convenience

Probability Density Functions

• Let us use the labels ω0 and ω1 to denote the impostor and genuine classes, respectively. Let p(s|ω1) and p(s|ω0) be the probability density functions of the genuine and impostor scores, respectively

• Both FRR and FAR are functions of the system threshold η

Threshold placement

• If the threshold is increased, FAR will decrease but the FRR will increase and vice versa

• Hence, for a given biometric system, it is not possible to decrease both these errors simultaneously by varying the threshold

Genuine Accept Rate

• Genuine Accept Rate: The Genuine Accept Rate (GAR) or True Accept Rate (TAR) can be used as an alternative to FRR while reporting the performance of a biometric verification system

• GAR is defined as the fraction of genuine scores that exceed the threshold η. Therefore,

𝐺𝐴𝑅 𝜂 = 𝑝 𝑠 ≥ 𝜂 𝜔1 = 1 − 𝐹𝑅𝑅(𝜂) • The same biometric system can be operated at different

thresholds (η) depending on the changing security level or different requirements of different applications

• The FAR and FRR at different values of threshold η are measured and summarized in the form of a Detection Error Tradeoff (DET) curve. – The DET curve plots the FRR against the FAR at various

thresholds

Drawing the Receiver Operating Characteristic (ROC) Curve

Performance Curves

• The FAR and FRR values vary with the threshold

• A Receiver Operating Characteristic (ROC) curve plots the FRR against FAR

• It can also plot the Genuine Accept Rate (GAR) against FAR

ROC Curves

Fa ls

e R

e je

ct R

a te

( F

R R

)

False Accept Rate (FAR)

Equal Error Rate (FRR = FAR)

Forensic Applications

High-security Access Applications

Civilian Applications

ROC Curve

• Receiver Operating Characteristic (ROC) curve

• The ROC curve plots the GAR (rather than the FRR)

against the FAR.

© King 2016

How to draw a ROC curve

• Given a set of match scores {si} L

i =1, where the first

L1match scores correspond to the genuine class, the

subsequent L0 match scores correspond to the

impostor class, and the total number of scores is L =

(L1+L0), the ROC curve can be computed by the

following steps:

© King 2016

Steps to draw ROC curve

© King 2016

{FAR, GAR}

• Change the pair to {FAR, GAR} for drawing

ROC curve. While it’s a matter of preference,

most plots in current literature do not use

{FAR, FRR} pairs to draw the ROC curve

• Recall,

𝐺𝐴𝑅 𝜂 = 𝑝 𝑠 ≥ 𝜂 𝜔1 = 1 − 𝐹𝑅𝑅(𝜂)

© King 2016

How to compare the performance

of two biometric systems? • The best way to compare the performance of two biometric

systems is to examine their ROC curves

• Case 1: If the GAR of one biometric system (say A) is consistently higher than the GAR of the other system (say B) for corresponding values of FAR, one can conclude that the matching performance of biometric system A is better than that of B.

• Case 2: If the two ROC curves intersect, it indicates that system A is better than system B at some operating points (FAR values), while system B is better at other operating points. In this scenario, it is also possible to compare the performance of the two systems by (AUC)

© King 2016

AUC

• For a given set of match scores, the AUC is

computed as

• The occurrence of false accepts and false rejects is

not evenly distributed across the users of a

biometric system

• There are inherent differences in the “recognizability”

of different users

Performance measure using other

single-valued measures • Equal Error Rate(EER) and the d-prime value

• The EER refers to that point in a DET (or ROC)

curve where the FAR (or GAR) equals the FRR;

• A lower EER value indicates better performance

Equal Error Rate

– EER is the point where FRR = FAR

– Systems having lower EER are preferred

d-prime value

• The d-prime value (d’) measures the separation between the means of the genuine and impostor probability distributions in standard deviation units and is defined as,

• where μ1 (μ0) and σ1 (σ0) are the mean and standard deviation, respectively, of the genuine (impostor) score distributions

• A higher d-prime value indicates better performance

Doddington’s Zoo

• Sheep represent users whose biometric feature sets are very distinctive and exhibit low intra-class variations. Therefore, these users are expected to have low false accept and false reject errors

• Goats refer to users who are prone to false rejects. The biometric feature sets of such users typically exhibit large intra-class variations

• Lambs are users whose biometric feature set overlaps extensively with those of other individuals. The biometric feature sets of these users have high inter-user similarity. The false accept rate associated with these users is typically high

• Wolves indicate individuals who are successful in deliberately manipulating their biometric trait in order to impersonate legitimately enrolled users of a system; An adversary attack; increase FAR (e.g., A person forges the signature of another user or mimicking someone else’s voice)

Two other types of failures

• (In addition to false non-match and false match errors)

• The Failure to Enroll(FTE) rate denotes the proportion of users that cannot be successfully enrolled in a biometric system – User training or habituation may be necessary to ensure that an

individual interacts with a biometric system appropriately in order to facilitate the acquisition of good quality biometric data

– This necessitates the design of robust and efficient user interfaces that can assist an individual both during enrollment and recognition

• In authentication, the fraction of authentication attempts in which the biometric sensor cannot capture the sample presented to it is known as Failure to Capture (FTC) or Failure to Acquire(FTA) rate. – The FTA rate is also impacted by sensor wear and tear.

– Periodic sensor maintenance is instrumental for the efficient functioning of a biometric system.

Identification Error Rates and

Biometrics System Design

Considerations

Identification system error rates

• False positive identification rate and False negative identification rate

• False positive identification rate: The expected proportion of identification transactions by users not enrolled in the system, where an identity is returned, is known as the false positive identification rate (FPIR)

– This is analogous to the false match case in biometric verification

– The FPIR depends both on the size of the enrollment database (N) and the threshold (η).

False negative identification rate

• False negative identification rate: The expected

proportion of identification transactions by users

enrolled in the system in which the user’s correct

identity is not returned is called the false negative

identification rate (FNIR)

• •FNIR depends on the size of the enrollment

database (N), the threshold (η) used for the match

scores, and the number of identities t returned by

the identification system

True positive identification rate

• True positive identification rate (TPIR): The expected proportion of identification transactions by users enrolled in the system, where the user’s correct identity is among the t identities returned by the system. – FNIR = 1− TPIR

• If the biometric system outputs the identities of the top t matches, the corresponding TPIR is also known as the rank-t identification rate, which we refer to as Rt

• In particular, the value of TPIR for t = 1 is called the rank-one accuracy

CMC curve

• The rank-t identification rate for different values of t

can be summarized using the Cumulative Match

Characteristic (CMC) curve, which plots Rt against t

for t = 1,2, ··· ,N, where N is the number of enrolled

users.

Some relations

• FPIR = 1 − (1−FMR)N

– A false positive identification occurs when the input falsely matches one or more templates in the database. FPIR is then computed as one minus the probability that no false match is made with any of the database templates

– If the FMR is very small (<< (1/N)), FPIR can be approximated as FPIR ≈ N×FMR

Design Cycle of Biometric Systems

Nature of the application

• In some applications, biometrics may be used to supplement ID cards and passwords, thereby imparting an additional level of security. Such an arrangement is often called a multi-factor authentication scheme

• Depending on the application, we may need to choose between the verification and identification functionalities – This choice need not be always mutually exclusive

– Example: In the large-scale national ID systems, one may need to perform negative identification during enrollment to prevent the possibility of the same user acquiring multiple identities

Summary of the nature of applications

• All the previously discussed system

characteristics profoundly influence the design

of a biometric system.

• Most of the commercial applications of

biometrics, such as access to secure facilities,

have the following attributes:

– verification, cooperative, overt, habituated,

attended enrollment and non-attended

authentication, and closed

Summary of Biometric Traits

• No single biometric is expected to effectively meet all the requirements (e.g., accuracy, practicality, cost) imposed by all applications

– e.g., forensics, access control, government benefits programs, etc.

• No biometric is ideal for every conceivable use case

• The relevance of a specific biometric to an application is established depending upon the nature and requirements of the application, and the properties of the biometric characteristic

Data collection

• Institutional Review Board (IRB) Approval

• Data is required both for designing the feature extraction and matcher modules as well as for the evaluation of the designed biometric system

• Care must also be taken to ensure that the database is neither too challenging (collected under the most adverse conditions) nor too easy (collected under the most favorable conditions)

• Ideally, a database should include samples that are representative of the population and must preferably exhibit realistic intra-class variations – e.g., collecting data over multiple sessions, spread over a

period of time, and in different environmental conditions

Choice of features and matching

algorithm

• Most of the research and development in the field of biometrics has been focused on this

• Needs some prior knowledge about the biometric trait under consideration – e.g., prior knowledge about the “uniqueness” of minutia

points facilitated the development of minutiae-based fingerprint recognition systems

• Another important factor is the interoperability between biometric systems – e.g., the performance of face recognition algorithms is

severely affected when the images used for comparison are captured using different camera types

Evaluation

Evaluation of a complete biometric system is a complex and challenging task. Questions to address for evaluation:

• What are the error rates of the biometric system in a given application? – (matching or technical performance)

• What is the reliability, availability, and maintainability of the system? – (engineering performance)

• What are the vulnerabilities of the biometric system? – What level of security does the biometric system provide to the application in

which it is embedded?

– (security of the biometric system)

• What is the user acceptability of the system? – How does the system address human factor issues like habituation and privacy

concerns?

– (user concerns)

• What is the cost and throughput of the biometric system and what tangible benefits can be derived from its deployment?

– (return on investment)

Evaluation Protocol

• Define a protocol to evaluate the system, select the biometric data and measure the performance

• Evaluations should be conducted by independent organization (third party)

• Testing should be done on biometric data previously unseen by the system

• Size of the data-set and representative sample of the population should be provided