PaperlayerView exact revision on arXiv

Source-linked AI summary

Face Image Quality Assessment: A Literature Survey

Torsten Schlett, Christian Rathgeb, Olaf Henniger, Javier Galbally, Julian Fierrez, Christoph Busch

arXiv:2009.01103v3cs.CV

TL;DR

Face-image quality assessment must estimate biometric utility even though recognition performance depends on comparisons involving other samples, creating the quality paradox. This survey synthesizes FIQA methods, applications, and evaluation practices, finding a progression toward monolithic deep-learning approaches while identifying limited comparability and visible-spectrum coverage as important boundaries.

  • Problem

    Single-image FIQA must predict the accuracy or certainty of comparison scores for sample pairs whose other samples are unknown to the assessor, creating the quality paradox.

  • Method

    The survey categorizes FIQA approaches, reviews their evaluations and applications, and examines Error-versus-Reject-Characteristic methodology across the literature.

  • Results

    The literature progressed from predominantly factor-specific methods toward monolithic deep-learning approaches, including methods that use or integrate face-recognition systems.

  • Takeaways & Limitations

    Comparable evaluations require greater standardization and public implementations, while future FIQA work should expand beyond predominantly visible-spectrum inputs.

Abstract

from arXiv · show

The performance of face analysis and recognition systems depends on the quality of the acquired face data, which is influenced by numerous factors. Automatically assessing the quality of face data in terms of biometric utility can thus be useful to detect low-quality data and make decisions accordingly. This survey provides an overview of the face image quality assessment literature, which predominantly focuses on visible wavelength face image input. A trend towards deep learning based methods is observed, including notable conceptual differences among the recent approaches, such as the integration of quality assessment into face recognition models. Besides image selection, face image quality assessment can also be used in a variety of other application scenarios, which are discussed herein. Open issues and challenges are pointed out, i.a. highlighting the importance of comparability for algorithm evaluations, and the challenge for future work to create deep learning approaches that are interpretable in addition to providing accurate utility predictions.

I. INTRODUCTION

FIQA estimates the biometric utility of face images, typically using a scalar score intended to predict face-recognition performance. This survey defines FIQA, organizes its literature and applications, and highlights the quality paradox, evaluation comparability, and future challenges.

  • FIQA takes a face image as input and produces a quality estimate, typically a scalar score intended to predict face-recognition performance.The score can support decisions without necessarily running a face-recognition algorithm.
  • Biometric utility measures a sample’s fitness for a biometric function, whereas fidelity measures similarity to the source characteristic and character describes inherent source attributes.The survey uses utility as its primary definition of face-image quality.
  • The survey organizes FIQA approaches with a taxonomy distinguishing factor-specific and monolithic methods, alongside other methodological aspects.The taxonomy is presented as part of the survey’s contributions.
  • It surveys more than 60 FIQAA publications from 2004 to 2021 and provides overview tables covering publications and datasets.The authors describe the survey as the most comprehensive FIQA survey to date.
  • The survey introduces the Error-versus-Reject-Characteristic methodology and evaluates varied FIQA approaches while discussing its weaknesses and standardization potential.It also identifies diverse application areas beyond image selection and discusses issues and future challenges.

II. QUALITY ASSESSMENT IN FACE RECOGNITION

FIQA assesses face images for biometric utility, distinguishing this goal from general perceptual image quality and organizing methods by their reference-data requirements. The section also contrasts controlled and unconstrained acquisition scenarios and explains why reference-based approaches are less common for FIQA.

  • Face recognition acquires reference templates during enrolment and compares processed probe images against them during verification or identification.
  • Acquisition scenarios: Controlled acquisition permits cooperation and environmental control, whereas unconstrained acquisition lacks subject cooperation and control over conditions.FIQA can provide immediate feedback in controlled acquisition and filter low-quality images in unconstrained acquisition.
  • FIQA versus IQA: FIQA targets biometric utility, while general IQA usually targets subjective perceptual quality; applying either type directly to the other domain performs poorly.
  • Reference requirements: Full-reference IQA compares an input with a known higher- or equal-quality version, reduced-reference IQA uses incomplete reference information, and no-reference IQA needs no input reference.
  • Reference requirements: Full- and reduced-reference approaches are more common for general IQA than FIQA because suitable original face references are less often available for biometric assessment.

D. The Quality Paradox

Single-sample FIQA faces a quality paradox: a score for one image must predict comparison behavior involving that image and another sample. The survey therefore examines how different ground-truth and FR-integrated designs address this constraint.

  • A single-sample FIQA score must indicate the accuracy or certainty of comparison scores for sample pairs containing the assessed image.
  • General IQA repurposing is not conceptually linked to face-recognition utility, independently of the quality paradox.
  • Ground-truth generation: Human-defined quality labels depend generally on subjective assessments, although protocols such as pairwise comparisons can guide them.
  • Ground-truth generation: FR-derived ground truth requires selecting comparison pairs and transforming pairwise FR scores into per-sample quality scores, making label generation part of FIQA design.
  • Ground-truth generation: Examples include FaceQnet's normalized comparison score, FaceQnet v1's multi-system score fusion, and PCNet's loss-based training.
  • FR integration: FR-integrated approaches use recognition models during FIQA training or inference and can estimate the quality or certainty of generated feature embeddings more directly.

E. Application Areas of FIQA

FIQA supports decisions throughout the biometric pipeline, from acquisition feedback and database maintenance to recognition adaptation and presentation-attack risk reduction. Its uses include selecting, weighting, enhancing, compressing, and summarizing face data according to assessed quality.

  • Acquisition: During acquisition, FIQA can reject images below a threshold or provide feedback on requirements for enrolment, passports, and government-issued identity documents.
  • Selection and monitoring: Quality scores can rank video frames, select database images, and identify devices, locations, times, or users associated with persistently low-quality samples.
  • Image processing: FIQA can support conditional enhancement and compression control by applying processing selectively according to quality ranges or measured quality changes.
  • Recognition: Quality can activate recognition-context switching, weight score or decision fusion, or enter comparisons directly through feature-uncertainty modeling.
  • Security: Quality filtering or weighting may reduce some presentation-attack opportunities, but pure FIQA is not intended for comprehensive presentation-attack detection.
  • Recognition: Progressive identification can search higher-quality templates first and stop once acceptable-certainty matches are found, although fast full-database search can make this irrelevant.

III. CATEGORIZATION

The survey taxonomy separates factor-specific FIQA, which can expose interpretable and actionable deficiencies, from monolithic FIQA, which produces comparatively opaque scores. Additional categories organize methods by data integration, training targets, and acquisition-related factors.

  • Taxonomy: Factor-specific approaches measure interpretable factors such as blur, whereas monolithic approaches produce comparatively opaque assessments without a separately interpretable factor.
  • Factor aspects: Capture-related factors depend on external acquisition conditions, while subject-related factors depend on properties such as pose, expression, or movement.
  • Factor aspects: Pose is predominantly subject-related, while blur is both capture-related and subject-related because motion or camera configuration can cause it.
  • Data aspect: The taxonomy organizes methods from hand-crafted designs through utility-agnostic training and ground-truth quality-score training to FR-based inference.
  • Survey organization: The surveyed literature timeline and taxonomy provide counts and categories for comparing FIQA approaches across the survey period.
  • Data aspect: Ground-truth quality-score training includes human-ground-truth and FR-based-ground-truth categories, while FR-based inference uses FR models without FIQA training on ground-truth scores.

C. Aspect: Fusion

FIQA fusion combines multiple separable quality-assessment outputs, either explicitly, through trained models, or in cascaded stages. The survey also distinguishes broader methodological trends, datasets, and scope boundaries in the literature.

  • Fusion: FIQA fusion methods combine multiple separable FIQA outputs, whose resulting scores may remain opaque despite interpretable component algorithms.The survey marks only pure FIQA fusion methods and distinguishes them from fusion incorporating FR scores or EXIF data.
  • Fusion: Explicit fusion computes quality scores using manually determined weighted sums or other hand-crafted fusion functions.
  • Fusion: Trained fusion learns weights or uses machine-learning models including neural networks, Gaussian mixture models, AdaBoost, and random forests.
  • Fusion: Cascaded fusion assesses quality in stages and can exit early when quality is too low, reducing workload in applications such as video-frame selection.
  • Broader literature: Most surveyed FIQA works are non-DL, while most recent works use deep learning; non-DL methods are commonly factor-specific and DL methods commonly monolithic.

A. Factor-specific - Commonalities

Factor-specific FIQA decomposes quality into interpretable image properties such as size, illumination, blur, symmetry, pose, and other attributes. The surveyed literature includes hand-crafted measures, learned combinations, and evaluations revealing task- and metric-dependent behavior.

  • Factor taxonomy: Factor-specific FIQA organizes methods around size, illumination, blur, symmetry, pose, and less common attributes such as glasses, noise, skin tone, alignment, and occlusion.
  • Blur: Blur measures use edge, frequency, or low-pass-filter analysis to estimate focus or sharpness.
  • Symmetry: Symmetry methods compare fixed or fitted face halves, while localized methods compare paired key points or facial landmarks.
  • Evaluations: Illumination subsumed edge density and SEMC focus for FR-performance prediction in experiments using FRVT 2006 data and similar FR algorithms.
  • Evaluations: BioLab-ICAO introduced 30 FIQA measures for ISO/ICAO requirements, and most tested methods either outperformed both COTS systems or lacked a testable COTS counterpart.
  • Evaluations: The 13 quality measures showed a substantial gap from the approximately optimal GPO oracle, while some measures worsened FMR and PCA-LDA generalized poorly.
  • Evaluations: FIIQD contained 224,733 images with 200 illumination patterns, and predicted illumination scores strongly correlated with subjective labels without an evaluated FR-performance impact.
  • Evaluations: In smartphone FR, the COTS FIQA and a random forest incorporating vertical edge density achieved the best partial (20%) ERC AUC among evaluated methods.

C. Monolithic - Commonalities

Monolithic FIQA produces comparatively opaque quality assessments and is categorized by how training data or FR models provide utility information. A newer category integrates FIQA and FR training into one model.

  • Taxonomy: Monolithic FIQA has no factor-specific subcategories, so its taxonomy emphasizes the data used to train or infer quality.
  • Data categories: Utility-agnostic monolithic methods use general IQA models or fixed training-image statistics without directly training on quality labels.
  • Data categories: Human-ground-truth methods learn from human quality assessments, sometimes transferring labels to additional unlabeled images.
  • Data categories: FR-based ground-truth methods derive training data from FR models, and most monolithic approaches belong to this category.
  • FR relationships: FR-based inference methods use FR models during quality assessment without altering FR training, whereas FR-integrated methods intertwine FIQA and FR training.

D. Monolithic - Literature introductions

The monolithic literature spans reference-image comparisons, hand-crafted and learned ranking, FR-derived supervision, CNNs, uncertainty representations, and FR-integrated approaches. Studies differ in whether quality labels reflect subjective judgments, FR comparisons, or other image statistics.

  • Reference-based methods: Reference-based methods compare face images with fixed average, Gaussian-blur, or training-set-derived references to produce quality scores.
  • FR-based methods: The impostor-based uniqueness measure compares a face with 1,000 feature vectors from different subjects through the FR system itself.
  • Learned scoring: Learning-to-rank FIQA learns weighted sums over feature outputs so scores distinguish datasets assigned an assumed quality ordering, producing a scalar RQS normalized to [0, 100].
  • Learned scoring: CNN-based methods used diverse supervision, including FR-performance scores, human assessments, gallery comparison distances, and quality labels transferred through semi-supervised training.
  • FR-based methods: FR-derived quality models included KPLSR using luminance and Laplacian-gradient features, with ground truth from LBP-based FR comparison scores.
  • FR-based methods: FaceQnet training used BioLab-ICAO-selected high-quality images to compute FR-embedding-distance quality labels for remaining training images.
  • FR-based methods: PFE represents uncertainty as a vector corresponding directly to the FR feature vector, although producing a scalar quality score was not its primary goal.

V. EVALUATION

The evaluation section introduces a common methodology for assessing FIQAAs by their ability to predict biometric utility for a fixed face-recognition system and dataset, then applies it to 14 approaches.

  • The methodology evaluates FIQAAs against biometric utility for a given face-recognition system and dataset.

A. Error-versus-Reject-Characteristic

ERC evaluates how quality-based rejection changes face-recognition error, using progressively stricter quality thresholds and standardized rejection fractions. AUC and pAUC summarize performance, but comparisons remain constrained by inconsistent literature evaluations and configuration choices.

  • ERC plots face-recognition error against the fraction of images rejected using quality scores.A face-recognition system and subject-labeled face dataset are required alongside the FIQAA.
  • Quality scores are combined for image pairs, then progressively higher thresholds remove lower-quality samples and recompute the error measure.FNMR is suggested as the primary error measure, with FMR-based thresholds optionally derived.
  • ERCs should reject samples rather than individual comparisons, so all comparisons involving samples below the threshold are discarded together.This makes the horizontal axis the maximum rejected-image fraction and favors stepwise interpolation.
  • AUC covers the full 0–100% rejection range, whereas pAUC focuses on the 0–20% range.Subtracting the theoretical-best area is unnecessary when only rankings within a fixed range are analyzed.
  • An oracle could provide a more realistic approximation of an optimal FIQAA, but recent FIQA literature has not continued exploring this approach.
  • Precise cross-paper performance comparison is impossible when studies use different evaluation outputs, face-recognition systems, and datasets.

B. Selective Evaluation

The selective evaluation compares 14 FIQA approaches on LFW using FNMR ERC analysis across full and cropped images. MagFace performs best in this configuration, while rankings vary by rejection range and the evaluation has several limitations.

  • The study evaluates 14 FIQA approaches, including hand-crafted, utility-agnostic, and FR-based methods.
  • LFW is challenging for FIQA ERC evaluation because state-of-the-art face recognition is nearly saturated and its quality distribution is relatively narrow.
  • The evaluation computes ERCs on both full LFW images and RetinaFace-cropped, landmark-aligned variants.
  • MagFace achieves the best results throughout this evaluation, while other rankings depend on the pAUC range.
  • The study tests relatively few FIQA, FR, dataset, preprocessing, and hyperparameter configurations compared with the available options.The authors note that broader evaluation would require substantial reimplementation and configuration exploration.
  • The evaluation excludes non-mated pairs because it uses FNMR, leaving FMR and other error metrics for possible future analysis.
  • Comparisons among FR-trained approaches are affected by differing training data, preprocessing, and training-time choices.

VI. OPEN ISSUES AND CHALLENGES

The survey identifies improving FIQA predictive and computational performance as an ongoing challenge, particularly as deep-learning FIQA advances remain intertwined with broader deep-learning and face-recognition research.

  • Improving FIQA predictive and computational performance remains an open challenge.
  • For deep-learning FIQA, better architectures and training methods are connected to progress in general deep-learning research.
  • FIQA methods intended to predict face-recognition utility also depend on advances in face-recognition research.

A. Comparability and Reproducibility

Comparing FIQA approaches is difficult because studies vary in datasets, evaluation methods, FR algorithms, and configurations. Public implementations, standardized evaluation campaigns, and computational reporting could improve comparability and reproducibility.

  • Comparability: FIQA evaluations differ in datasets, evaluation methods, FR algorithms, and other configurations, hindering fair direct comparison.Dataset subsets, synthetic degradation, image-pair selection, ground-truth construction, and FR model choices can each alter results.
  • Comparability: Standardized evaluation campaigns could assess FIQA approaches under aligned conditions and limit the scope of comparison challenges.The survey identifies campaign-based assessment as one possible future solution.
  • Reproducibility: Public source code and trained models can enable integration into different evaluation environments and reduce divergence caused by reimplementation.Source code is especially important for machine-learning FIQA because results depend on training data and implementation details.
  • Reproducibility: Public datasets, precise evaluation configurations, and raw evaluation results could support reproducible analyses and cross-work visualizations.Raw comparison and quality scores can be reused to create visualizations combining results from multiple studies.
  • Evaluation scope: Computational performance is relevant but rarely evaluated, with tests typically measuring processing duration for inputs of specified format and resolution.The survey distinguishes computational workload from utility-prediction performance evaluation.

B. Explainability and Interpretability

Recent monolithic deep-learning FIQA methods target FR utility but provide less human-interpretable feedback than factor-specific approaches. Future work should seek models that combine utility prediction with explanations of quality-degrading causes, supported by varied realistic data.

  • Explainability: Monolithic deep-learning FIQA methods predict FR utility but are less interpretable than approaches estimating human-understandable factors such as blur.The survey frames the tension as accurate utility prediction versus feedback about quality-degrading causes.
  • Explainability: Future FIQA models should ideally predict FR utility while identifying quality-degrading causes that can guide acquisition adjustments.The proposed direction combines predictive performance with actionable feedback for human operators.
  • Data: Large realistic datasets spanning quality levels and degradation causes could support more certain robustness verification during evaluation and training.Synthetic degradation and fully synthetic face images are presented as possible ways to increase scale and control.
  • Interoperability: Interoperability can conflict with maximizing FR-system-specific utility prediction, but may reduce vendor lock-in and coincide with increased robustness.The survey identifies this as a future goal rather than a settled trade-off.
  • Robustness: Adversarial attacks remain an open area because surveyed machine-learning FIQA studies did not examine input modifications designed to confuse FIQA models.The scope includes both physical attacks and digital modifications after capture and processing.

F. Standardization

The survey points toward broader FIQA standardization and application beyond visible-spectrum face recognition. It highlights a long-term goal of standardizing a specific FIQA approach while expanding evaluation across modalities, tasks, and operational settings.

  • Standardization: A long-term FIQA objective is standardizing a specific approach, analogous to the normative standardization of NFIQ 2.The survey connects this objective to advances in comparability, interpretability, robustness, and evaluation methodology.
  • Further applications: FIQA applications remain underexplored for lossy compression, soft biometrics, attention estimation, and emotion analysis.Compression control was not considered in the surveyed literature despite compression artifacts being recognized as quality-degrading factors.
  • Modalities: Almost all surveyed FIQA literature focused on visible-spectrum data, while near-infrared work was limited to comparisons against human rankings.The cited near-infrared study combined sharpness, brightness, resolution, pose, and expression measures.
  • Modalities: Depth FIQA has received little biometric attention, although one study used a deformable shape model to improve 3D FR through sample rejection.The reported depth-FIQA use focused on identifying excessive expression variations.
  • Further applications: Face sketch recognition and synthesis literature has generally emphasized perceptual measures rather than biometric utility prediction.The survey identifies this as another possible direction for FIQA research.
  • Applications: FIQA supports acquisition feedback, filtering, database maintenance, and monitoring, including large-scale systems with many data sources.The survey cites systems such as SIS, VIS, EES, and ESTA as examples of large-scale operational contexts.
  • Methodological direction: Recent FIQA methods increasingly use monolithic deep learning, including approaches that rely on FR systems during inference or integrate FIQA into FR models.This progression contrasts with older factor-specific approaches that were more independent of concrete FR systems.
  • Evaluation: Comparability remains a key challenge because the literature uses many evaluation configurations, motivating public implementations and broader computational reporting.The NIST FRVT Quality Assessment evaluation is identified as an available submission pathway.
Loading 2009.01103v3…