PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Quality-Dependent and Cost-Sensitive Score-Level Multimodal Biometric Fusion Algorithms

Norman Poh, Thirimachos Bourlai, Josef Kittler, Lorene Allano, Fernando Alonso-Fernandez, Onkar Ambekar, John Baker, Bernadette Dorizzi, Omolara Fatukasi, Julian Fierrez, Harald Ganster, Javier Ortega-Garcia, Donald Maurer, Albert Ali Salah, Tobias Scheidat, Claus Vielhauer

arXiv:2111.08703v1cs.CVcs.CR

TL;DR

The paper addresses the lack of a common benchmark for multimodal biometric fusion and evaluates systems using shared scores and quality measures. Across quality-dependent and cost-sensitive evaluations, results include strong device-specific systems in cross-device testing and modest gains for most exhaustive systems; the authors also report sequential fusion as a promising cost-saving strategy.

Abstract

from arXiv · show

Automatically verifying the identity of a person by means of biometrics is an important application in day-to-day activities such as accessing banking services and security control in airports. To increase the system reliability, several biometric devices are often used. Such a combined system is known as a multimodal biometric system. This paper reports a benchmarking study carried out within the framework of the BioSecure DS2 (Access Control) evaluation campaign organized by the University of Surrey, involving face, fingerprint, and iris biometrics for person authentication, targeting the application of physical access control in a medium-size establishment with some 500 persons. While multimodal biometrics is a well-investigated subject, there exists no benchmark for a fusion algorithm comparison. Working towards this goal, we designed two sets of experiments: quality-dependent and cost-sensitive evaluation. The quality-dependent evaluation aims at assessing how well fusion algorithms can perform under changing quality of raw images principally due to change of devices. The cost-sensitive evaluation, on the other hand, investigates how well a fusion algorithm can perform given restricted computation and in the presence of software and hardware failures, resulting in errors such as failure-to-acquire and failure-to-match. Since multiple capturing devices are available, a fusion algorithm should be able to handle this nonideal but nevertheless realistic scenario. In both evaluations, each fusion algorithm is provided with scores from each biometric comparison subsystem as well as the quality measures of both template and query data. The response to the call of the campaign proved very encouraging, with the submission of 22 fusion systems. To the best of our knowledge, this is the first attempt to benchmark quality-based multimodal fusion algorithms.

I. INTRODUCTION

The introduction frames multimodal biometric fusion as a way to improve identification while noting that quality measures and operating costs complicate system design. It motivates a shared benchmark and evaluation campaign for quality-dependent and cost-sensitive fusion.

A. Multimodal Biometric Systems I

  • Multimodal systems combine biometric evidence to improve security and identification, while quality-based fusion additionally weights evidence using objective measures of input-signal quality.The introduction notes that exploiting these nonclass-discriminatory quality measures to improve classification is not trivial.Source: A. Multimodal Biometric Systems I · pp. 1–2

B. State-of-the-Art in Quality-Based Fusion

  • Prior quality-based fusion research spans Bayesian, confidence-based, clustering, polynomial, logistic-regression, and other approaches, with early experiments limited by chimeric data and manually generated quality measures.Later work reported more realistic experiments using discriminative and generative learning.Source: B. State-of-the-Art in Quality-Based… · p. 2

C. Motivation

  • The authors identify a lack of a benchmark for comparing quality-based fusion algorithms on common biometric subsystem scores, alongside cost and robustness questions involving acquisition, processing, and missing scores.More devices or samples may improve performance but add processing time and hardware and maintenance costs; failures can leave scores invalid or missing.Source: C. Motivation · p. 2

D. Cost-Sensitive and Quality-Based Fusion Evaluation Campaign

  • The campaign evaluates quality-dependent fusion and conventional fusion, including user-specific classifiers and handling missing subsystem information from failures-to-acquire or failures-to-match.The cost-sensitive protocol treats fusion as maximizing recognition performance at a desired minimum cost.Source: D. Cost-Sensitive and Quality-Based… · pp. 2–3

E. Contributions

  • The paper presents a first comparison of 22 submitted fusion systems and introduces cost-sensitive and robustness evaluation, sequential fusion, and a benchmark database of score and quality measures.Sequential fusion processes match scores until sufficient confidence is reached or the available scores are exhausted.Source: 1) Benchmark for multimodal biometric… · p. 3

1) Benchmark for multimodal biometric quality-based fusion:

  • The benchmark database supports comparison of quality-based fusion systems; the cited XM2VTS score-level fusion benchmark does not include quality measures for each biometric modality.The paper states that the benchmark datasets were made publicly available.Source: 1) Benchmark for multimodal biometric… · p. 3

F. Paper Organization

The paper outlines its organization and introduces notation for comparing submitted fusion classifiers. It distinguishes classifier families and describes transformation-based fusion as a two-stage approach.

F. Paper Organization

  • The paper first summarizes submitted classifiers, then describes the BioSecure DS2 data, evaluation protocols, results, and conclusions.The evaluation includes 22 fusion-system submissions from seven sites.Source: F. Paper Organization · p. 3, §II · p. 3

A. Classifier Categorization and Notation

  • Fusion classifiers are categorized as generative or discriminative, with both types accounting for dependencies among observations such as scores or quality measures.Generative classifiers estimate class-dependent densities before making decisions; discriminative classifiers estimate the decision boundary directly.Source: A. Classifier Categorization and… · p. 3
  • Transformation-based fusion independently normalizes subsystem match scores and then combines them with a fixed rule, such as sum or product.Product fusion requires normalized scores to satisfy range and non-negativity conditions; the paper notes that the sum rule is commonly effective.Source: A. Classifier Categorization and… · pp. 3–4
  • The paper uses common notation to distinguish conventional fusion, based on subsystem outputs, from quality-dependent fusion, which also takes quality measures as input.The submitted systems are listed in Table I.Source: A. Classifier Categorization and… · pp. 3, 5, Table I · p. 4

B. Submitted Systems

The submitted generative systems estimate class probabilities or likelihoods from biometric subsystem outputs. They handle missing observations through marginalization or alternative score-processing strategies.

1) Generative Classifier Estimating the Posterior Probability:

  • Generative fusion estimates client probabilities using Bayes’ rule and class-conditional likelihoods, with GET-1 and CWI-IMOFA identified as examples.GET-1 uses a Gaussian mixture model, while CWI-IMOFA uses a mixture of factor analyzers with fewer parameters through low-dimensional latent factors.Source: 1) Generative Classifier Estimating… · p. 5
  • UniS’s GMM-Bayes classifier handles missing observations by marginalizing the observed features, using Gaussian means and covariances rather than explicit integration.The passage says this marginal calculation can be performed efficiently.Source: 1) Generative Classifier Estimating… · p. 5
  • GET’s GMM-Bayes systems use Min–Max normalization and average fusion when observations are missing, while CWI-IMOFA substitutes median values without significantly affecting development-set generalization.The normalized subsystem outputs and GMM-Bayes outputs share a range, though only the latter are interpreted as probabilities.Source: 1) Generative Classifier Estimating… · p. 5

2) Generative Classifier Using the Log-Likelihood Ratio Test:

This section describes generative and quality-dependent fusion strategies, including a log-likelihood ratio test and quality-state clustering. It notes that modeling quality directly can raise dimensionality relative to the available training data.

2) Generative Classifier Using the Log-Likelihood Ratio Test:

  • Modeling quality measures directly alongside subsystem outputs can increase observation dimensionality and create an estimation problem because training samples are fixed.The section describes a UniS alternative that incorporates quality measures without directly increasing the modeled dimensionality.Source: 2) Generative Classifier Using the… · p. 6
  • The UniS quality-based model assumes subsystem outputs and quality measures are conditionally independent given a quality state, defined as a cluster of similar quality measures.Its complexity depends on the number of clusters rather than the number of quality measures; the implementation constrains the GMM clusters to one through three.Source: 2) Generative Classifier Using the… · p. 6
  • Quality measures may be clustered by device, refined within each device, or clustered independently of device; UniS uses device-dependent clusters and estimates quality state probabilistically at test time.The device is known during training but not testing, and the device-specific quality densities are modeled with a GMM.Source: 2) Generative Classifier Using the… · p. 6
  • The described procedures produce outputs combined by a sum rule, so a subsystem with no score observation contributes no term to the final sum.JHUAPL is described as a realization of the same general approach, with a histogram-based density estimate that requires scalar quality measures.Source: 2) Generative Classifier Using the… · p. 6

3) Generative Classifier Using the Dempster–Shafer Theory:

The classifier represents biometric evidence as mass distributions over identity hypotheses and combines selected subsystem outputs using Dempster’s rule.

3) Generative Classifier Using the Dempster–Shafer Theory:

  • Dempster’s rule combines each new mass distribution with the distribution derived from prior information sources, producing a joint distribution.Its conflict term normalizes the combination and displays conflict between new evidence and current knowledge.Source: 3) Generative Classifier Using the… · pp. 6–7
  • Each biometric score is interpreted as the percentage certainty that the claimed identity is true and assigned mass to the genuine-user set.Additional mass is assigned to the whole frame of discernment.Source: 3) Generative Classifier Using the… · p. 7
  • Subsystem outputs with missing observations do not participate in Dempster’s rule; the operation is described as analogous to the probability sum rule.The framework treats information sources as not necessarily disjoint and distributes one unit of mass among them.Source: 3) Generative Classifier Using the… · pp. 6–7

4) Discriminative Classifier Using Linear Logistic Regression:

The section describes logistic-regression fusion variants that incorporate quality information through modality-dependent observations, device-specific normalization, or score-quality interactions.

4) Discriminative Classifier Using Linear Logistic Regression:

  • The Naive Bayes logistic-regression variant combines modality-dependent observations that may contain a score alone or a score augmented with same-modality quality measures.The variant applies the Naive Bayes assumption to the modality-dependent observations.Source: 4) Discriminative Classifier Using… · p. 7
  • UPM uses logistic regression to infer the acquisition device, normalize scores with the corresponding device-dependent function, and combine the normalized scores.Quality measures guide device inference; they are not used in the final combination term.Source: 4) Discriminative Classifier Using… · p. 7
  • UniS qfuse uses logistic regression with a reduced second-order polynomial expansion between score and quality vectors to model their interaction.The reduced expansion limits the number of parameters and avoids modeling unnecessary interactions among score and quality elements.Source: 4) Discriminative Classifier Using… · p. 7

5) Error-Based Fusion Classifier:

The error-based classifier weights subsystem scores using development-set performance and sample quality, and provides rules for missing scores and quality-dependent grouping.

5) Error-Based Fusion Classifier:

  • Subsystem scores are combined linearly using weights associated with each subsystem, with a normalization constraint on the weights.The weight design uses each subsystem’s development-set equal error rate; subsystems below a weight threshold do not contribute.Source: 5) Error-Based Fusion Classifier · p. 7
  • Missing matching scores are replaced with the average of the genuine and impostor score-distribution means.Source: 5) Error-Based Fusion Classifier · p. 8
  • The UniS fixed-rule classifier combines similar-quality score groups with the sum rule and different-quality groups with the product rule.When one group is unobserved, the corresponding cases are handled separately; high quality is determined by a threshold on its quality measure.Source: 5) Error-Based Fusion Classifier · p. 8

7) Other Classifiers:

The section covers additional submitted classifiers, sequential fusion strategies, and the BioSecure database setup used for benchmarking.

7) Other Classifiers:

  • The classifier list is not exhaustive and includes CWI-SVM’s support vector machine, with further discussion cited in.The paper also points to prior biometric fusion use and a general survey.Source: 7) Other Classifiers · p. 8

C. Sequential Fusion

  • Sequential fusion processes scores until confidence crosses an accept or reject threshold, or available scores are exhausted, aiming to reduce average computation.CWI-IMOFA orders modalities face, iris, then fingerprint; GET-2 starts with face and may add iris or fingerprint, using up to eight scores and quality measures.Source: C. Sequential Fusion · p. 8

A. BioSecure Database as a Test Bed

  • The BioSecure test bed provides scores and quality measures for each access so fusion systems receive the same inputs and can be compared on equal grounds.The evaluation uses the desktop scenario’s still face, iris, and fingerprint modalities; the proposed schemes can also be applied to the Internet and mobile datasets.Source: A. BioSecure Database as a Test Bed · p. 8

IV. EVALUATION PROTOCOL

The protocol separates development from evaluation to support unbiased comparison, then tests cost-sensitive fusion under constrained device use and missing subsystem outputs.

IV. EVALUATION PROTOCOL

  • Disjoint development and evaluation user populations support unbiased performance assessment, with 51 genuine users for development and 156 for evaluation.The remaining 126 subjects serve as zero-effort impostors.Source: §IV · p. 10
  • Session-specific impostor sets let Session 1 train or tune classifiers while Session 2 validates them without reusing the same impostors.Client-specific classifiers may train on Session 1 evaluation data but are assessed on Session 2 data.Source: §IV · p. 10
  • Missing observations from segmentation or other failures are retained as dummy scores of 999, and failed quality extraction yields vectors of 999s.The protocol also notes site-dependent performance bias associated with differences in acquisition supervision.Source: §IV · p. 10

A. Cost-Sensitive Evaluation

  • Cost-sensitive evaluation tests robustness to missing scores or quality measures and performance under minimal computation and hardware cost.Acquisition and processing time are also treated as costs.Source: A. Cost-Sensitive Evaluation · p. 10
  • The study charges one unit for a device’s first use and 0.3 for each subsequent use, counting acquisition even when its score is unused.Fusion can use a fixed set of channels for every access or dynamically choose different channels for different requests.Source: A. Cost-Sensitive Evaluation · pp. 10–11

B. Cross-Device Quality-Dependent Evaluation

The evaluation tests whether fusion can handle device changes that alter query quality and unpredictable acquisition or matching failures. It also specifies how missing outputs are simulated and how verification errors are measured.

B. Cross-Device Quality-Dependent Evaluation

  • Cross-device evaluation asks fusion systems to select reliable channels when high-quality-device templates are matched against queries from high- or low-quality devices.The evaluation set contains four channels drawn from combinations that may include face or fingerprint device mismatch, with device identity unknown to the fusion algorithm.Source: B. Cross-Device Quality-Dependent… · p. 11
  • The four channel combinations require different fusion strategies, making device changes a challenge when query data differ from stored templates.Failure scenarios are also introduced to assess robustness to failure-to-acquire and failure-to-match.Source: B. Cross-Device Quality-Dependent… · p. 11, C. Simulation of Failure-to-Acquire… · p. 11

C. Simulation of Failure-to-Acquire and Failure-to-Match Scenarios

  • Failures are simulated as random and spontaneous, independent of device and subject, and missing values are introduced only in evaluation scores.Observed values are replaced by 999 at deletion ratios of 10%, 20%, 30%, and 40%, with each higher-ratio set retaining the prior set’s missing values.Source: C. Simulation of Failure-to-Acquire… · p. 11

V. EXPERIMENTAL RESULTS

  • HTER is the average of FAR and FRR, and reported HTER uses participant-supplied thresholds selected to satisfy the EER constraint on development data.Thus a poorly estimated threshold can yield high HTER even when EER is low.Source: §V · p. 11

A. Cost-Sensitive Evaluation

The cost-sensitive comparison includes fixed, exhaustive, and sequential fusion systems. After accounting for threshold estimates, GET-1 is the best exhaustive classifier on this dataset, while most exhaustive systems offer only marginal gains over fixed-static fusion.

1) Baseline Systems:

  • Baseline systems have higher error rates than recently reported results because they were not fine-tuned and failures were retained in the evaluation.The iris baseline also had suboptimal segmentation and relatively uncontrolled acquisition, including blurred images.Source: 1) Baseline Systems · p. 12

3) Exhaustive Fusion Systems:

  • GET-1 is the best exhaustive fusion classifier on this dataset when decision thresholds are correctly estimated.The three leading solutions—GET-1, UPM, and UniS qfuse—are trainable classifiers; Fig. 2 presents the exhaustive-system comparison.Source: Figure 2 · p. 12, 3) Exhaustive Fusion Systems · p. 13

2) Fixed Static Fusion Systems:

  • Adding an iris subsystem raises the fixed-static group’s cost to 3.3, without a significant HTER improvement over the two-sensor group.These fixed-static systems use Naive Bayes classifiers assuming independence.Source: 2) Fixed Static Fusion Systems · p. 12

3) Exhaustive Fusion Systems:

  • The best exhaustive system achieves the lowest generalization error, but most exhaustive systems are only marginally better than fixed-static systems.Exhaustive fusion has a maximum cost of 4.5 units.Source: 3) Exhaustive Fusion Systems · p. 12
  • Choosing an exhaustive fusion candidate for a given cost is challenging because development and evaluation populations and class priors differ.There are 255 nonempty combinations but only 18 unique costs; participants submitted one exhaustive system at the maximum cost of 4.5 units.Source: 3) Exhaustive Fusion Systems · p. 12

4) Post-Experimental Analysis:

The post-experimental analysis compares fusion performance against access cost across candidate combinations, then examines dynamic fusion under missing observations. Sequential systems remain stable as failures increase, while exhaustive fusion generally achieves lower HTER at higher cost.

4) Post-Experimental Analysis:

  • Exhaustive fusion generally has the lowest HTER, but its higher average access cost makes the sequential CWI and GET-2 solutions preferable when cost is considered.The sequential systems usually rely on a single modality, giving a cost slightly above one unit.Source: 4) Post-Experimental Analysis · p. 13, 5) Further Analysis of Dynamic Fusion · p. 14
  • The GMM-Bayes classifier is preferred for these 255 tasks because genuine scores from different fingers of one subject correlate up to 0.6, while different modalities correlate close to zero.No correlation is observed for impostor match scores.Source: 4) Post-Experimental Analysis · p. 13
  • Two-fold cross-validation evaluates all 255 fusion combinations, with the lowest performance curve identifying the candidate with the least error at a given cost.The reported performance is averaged EER across the two folds; only one combination yields a cost of 2.5.Source: 4) Post-Experimental Analysis · p. 13
  • The a priori curve represents achievable generalization using development-set selection, whereas the a posteriori curves use evaluation-set information and may also assume the optimal threshold is known.The second and third rank-one curves are oracle cases and are not achievable; their error decreases as evaluation-set and threshold information are added.Source: 4) Post-Experimental Analysis · p. 14

5) Further Analysis of Dynamic Fusion:

B. Cross-Device Quality-Dependent Evaluation

The cross-device evaluation examines fusion as image quality changes and observations go missing, and the paper concludes that quality measures help identify the likely capture device. The benchmarking campaign also identifies scope boundaries and future research needs.

B. Cross-Device Quality-Dependent Evaluation

  • As missing observations increase, automatic rejection of requests with missing scores or quality measures increases genuine-access false rejections and EER.The baseline assessment deletes an increasing proportion of entries, from 0% up to 40%.Source: B. Cross-Device Quality-Dependent… · p. 15
  • The top cross-device systems, UniS BNq and UPM, use quality measures to estimate the likely data channel and apply its corresponding device-dependent fusion function.GET-1 is next best but does not use the provided quality measures; DET curves show JHUAPL performs well at low FRRs and UniS-BNq dominates at low FARs.Source: B. Cross-Device Quality-Dependent… · p. 15

C. Future Research Issues

  • The quality-dependent benchmark is limited to variation from cross-device matching, leaving within-device quality variation outside its evaluated scope.The paper identifies user interaction and acquisition environment as other factors affecting sample quality.Source: C. Future Research Issues · p. 15
  • The dataset has a limited number of quality measures, and the sequential-fusion strategy leaves candidate selection and confidence-threshold selection as open research issues.The next-candidate selection is described as heuristic.Source: C. Future Research Issues · p. 16

VI. CONCLUSION

  • The campaign benchmarked 22 diverse fusion algorithms on a common score-and-quality dataset designed for physical access control in establishments of roughly 300–500 users.The methods include generative, discriminative, and transformation-based classifier combiners.Source: §VI · p. 16
  • The conclusion reports that top-performing systems under changing image quality exploit automatically extracted trait-quality measures to identify the likely source device.The quality changes may result from acquisition-device or device-configuration changes.Source: §VI · p. 16
Loading 2111.08703v1…