PaperlayerView exact revision on arXiv

Source-linked AI summary

EX-FIQA: Leveraging Intermediate Early eXit Representations from Vision Transformers for Face Image Quality Assessment

Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira, Jan Niklas Kolf, Andrea Atzori, Fadi Boutros, Naser Damer

arXiv:2604.22842v1cs.CVeess.IV

TL;DR

ViT-based FIQA has largely relied on final-layer representations, leaving the value of intermediate features insufficiently investigated despite the importance of efficient face recognition. The paper analyzes all twelve transformer blocks, introduces early exits and depth-weighted score fusion without additional training, and evaluates them across eight datasets and four face-recognition models. Middle exits provide up to 50% computational savings with competitive performance, while EX-FIQA-FW achieves strong results on large-scale benchmarks and supports deployment-specific operating points.

  • Problem

    ViT-based FIQA approaches primarily rely on final-layer representations, despite the importance of efficient face-quality assessment for face recognition systems.

  • Method

    The paper analyzes all twelve transformer blocks and adds early exits plus score fusion that combines intermediate quality predictions using pre-trained parameters.

  • Results

    Up to 50% computational savings are achieved with middle-layer exits while maintaining comparable performance, and EX-FIQA-FW achieves top-1 performance on IJB-C across four face-recognition models.

  • Takeaways & Limitations

    Intermediate ViT representations support adaptive computation and deployment-specific trade-offs between efficiency, balanced performance, and maximum accuracy.

Abstract

from arXiv · show

Face Image Quality Assessment is crucial for reliable face recognition systems, yet existing Vision Transformer-based approaches rely exclusively on final-layer representations, ignoring quality-relevant information captured at intermediate network depths. This paper presents the first comprehensive investigation of how intermediate representations within ViTs contribute to face quality assessment through early exit mechanisms and score fusion strategies. We systematically analyze all twelve transformer blocks of ViT-FIQA architectures, demonstrating that different depths capture distinct and complementary quality-relevant information, as evidenced by varying attention patterns and performance characteristics across network layers. We propose a score fusion framework that combines quality predictions from multiple transformer blocks without architectural modifications or additional training. Our early exit analysis reveals optimal performance-efficiency trade-offs, enabling significant computational savings while maintaining competitive performance. Through extensive evaluation across eight benchmark datasets using four FR models, we demonstrate that our fusion strategy improves upon single-exit approaches. Our proposed quality fusion approach employs depth-weighted averaging that assigns progressively higher importance to deeper transformer blocks, achieving the best quality assessment performance by effectively leveraging the hierarchical nature of feature learning in ViTs. Our work challenges the conventional wisdom that only deep features matter for face analysis, revealing that intermediate representations contain valuable information for quality assessment. The proposed framework offers practical benefits for real-world biometric systems by enabling adaptive computation based on resource constraints while maintaining competitive quality assessment capabilities.

I. INTRODUCTION

FIQA evaluates facial-image utility for identity verification, while ViT-FIQA approaches primarily use final-layer representations. EX-FIQA investigates intermediate exits and score fusion to improve the performance–efficiency trade-off in ViT-based FIQA.

  • FIQA measures suitability for identity verification rather than aesthetic image quality.It evaluates whether images such as passport scans or live captures contain sufficient biometric information for identity matching.
  • Existing FIQA methods primarily rely on final-layer representations from CNNs or ViTs.
  • Early exits can reduce ViT computation by terminating inference at intermediate depths, supporting resource-constrained and surveillance deployments.Low-quality frames can be filtered before expensive recognition stages.
  • The study analyzes all twelve transformer blocks to quantify performance–computation trade-offs across network depths.
  • EX-FIQA-FW fuses quality predictions from multiple transformer blocks to improve assessment accuracy with minimal computational overhead.
  • Up to 50% computational savings are achieved by middle-layer exits while maintaining comparable performance to final-layer exits across eight benchmark datasets.

II. RELATED WORK

ViT-FIQA builds on transformer representations and early-exit methods, while EX-FIQA applies the pretrained FIQA components at intermediate blocks. The architecture processes patch embeddings through 12 transformer blocks and derives quality scores from either a quality token or concatenated patch representations.

  • Related work: ViTs refine image representations through self-attention blocks, with early layers capturing low-level patterns and deeper layers encoding higher-level attributes.This hierarchical structure motivates examining intermediate representations for face quality assessment.
  • Early exits in ViTs: Early-exit methods terminate inference at intermediate depths to reduce computation, including applications that filter insufficient-quality surveillance frames before expensive recognition.Prior approaches may add specialized heads, self-distillation, or dynamic routing, whereas this work evaluates pretrained FIQA models at inference time.
  • EX-FIQA architecture: EX-FIQA applies the original pretrained regression components to representations extracted from each of the 12 transformer blocks without modifying the base architecture.The ViT-FIQA variants share patch embedding and transformer processing, while their quality heads use either a quality token or concatenated patch features.
  • ViT-FIQA architectures: ViT-FIQA (T) computes quality from the final quality token through a batch-normalized regression head, whereas ViT-FIQA (C) concatenates patch representations before its two-layer feature network and regression head.The two variants were trained with a combined CosFace recognition loss and Smooth L1 FIQA regression loss, weighted by λ = 10.

B. Early Exit Mechanism: EX-FIQA

EX-FIQA enables early quality prediction from intermediate ViT blocks by reusing the original regression components without architectural changes. Constant token dimensionality across layers makes these intermediate exits directly usable, while their per-exit overhead differs substantially between variants.

  • Early Exit Implementation: Intermediate representations from each transformer block are extracted and passed through the original pre-trained quality assessment components without additional parameters.This preserves the capabilities of the pre-trained representations while enabling exits at blocks l ∈ {1, 2, . . . , L}.
  • Architectural Basis: Constant token dimensionality D = 512 across all L = 12 transformer layers allows the original regression weights to operate on intermediate representations.This contrasts with CNNs, whose spatial dimensions and channel depth change progressively.
  • Variant-Specific Exits: ViT-FIQA (T) computes quality from the block’s quality token using the original regression head, whereas ViT-FIQA (C) processes concatenated patch representations through its pre-trained feature network before regression.The two variants therefore reuse different intermediate representations and regression pathways.
  • Computational Complexity: EX-FIQA (T) requires approximately 3000 FLOPs per exit, while EX-FIQA (C) requires approximately 76M FLOPs because its feature network processes concatenated patches.The larger per-exit overhead partially limits computational reduction for EX-FIQA (C).

C. Score Fusion Strategies: EX-FIQA-F & EX-FIQA-FW

The fusion framework combines quality scores from all transformer blocks, either uniformly or with progressively greater weight assigned to deeper blocks. It is motivated by the complementary quality information captured across intermediate and deep representations.

  • Motivation: Intermediate representations capture distinct quality-relevant information, so aggregating them can potentially exceed performance from relying solely on the final layer.The paper connects this complementarity to differing semantic information across layers and attention-map observations.
  • Fusion Formulation: Score fusion combines the quality predictions from all L transformer blocks into a single fused quality score.The fused score is formed as a weighted combination of the block-level scores.
  • EX-FIQA-F: EX-FIQA-F applies uniform weighting across all transformer blocks, assuming quality-relevant information is distributed evenly across depths.Its weights satisfy w_l = 1/L.
  • EX-FIQA-FW: EX-FIQA-FW applies depth-based linear weighting, assigning progressively higher importance to deeper transformer blocks.The weighting follows w_l ∝ l and reflects the assumption that representational quality improves with depth.

IV. EXPERIMENTAL DETAILS

The experiments use pre-trained ViT-FIQA models without modifying their base architectures. Intermediate representations are evaluated with the original pre-trained regression components to measure their native quality-assessment capabilities.

  • Experimental Setup: Pre-trained ViT-FIQA models are used without modification to the base architecture while intermediate representations are extracted from each transformer block.The original pre-trained regression components are then applied to those representations.

B. Datasets and Evaluation Protocols

The evaluation spans eight face datasets, multiple face-recognition models, and pAUC-EDC measurements under fixed FMR thresholds. Results show that middle exits can be competitive with the final layer, while depth-weighted fusion improves assessment across challenging settings with deployment-relevant trade-offs.

  • Datasets and Metrics: Eight benchmark datasets are evaluated using EDC curves, with FNMR and pAUC measured to quantify verification performance as low-quality images are discarded.The pAUC is measured up to a 30% rejection rate under established protocols.
  • Evaluation Protocol: Table I reports pAUC-EDC for Exits 1–12 and EX-FIQA-F/EX-FIQA-FW across eight datasets using ArcFace, where lower values indicate better quality assessment.The table includes fixed FMR settings of 1e−3 and 1e−4.
  • Exit Performance: 36.978/47.487 at Exit 1 falls to 27.255/34.536 at Exit 5 for EX-FIQA (C), while later improvements are more limited, reaching 26.526/37.528 at Exit 11.Middle exits are best on challenging benchmarks such as CALFW and IJB-C, although Exit 12 performs best on some smaller benchmarks.
  • Exit Performance: 26.481/33.294 is the EX-FIQA (T) mean pAUC-EDC at Exit 12, with optimal mean performance at Exit 10 and middle exits achieving the best IJB-C results.Exit 12 remains strongest on some individual datasets, including Adience and CPLFW.
  • Fusion Results: EX-FIQA-FW outperforms Exit 12 and uniform fusion across both variants, surpassing Exit 12 on 5 of 8 datasets for EX-FIQA-FW (C).The weighted approach is especially effective on challenging datasets.
  • Computational Trade-offs: 6.75 vs 11.49 GFLOPs gives EX-FIQA (C) Exit 7 similar performance to Exit 12, while EX-FIQA (T) Exit 10 saves 17% computation and improves mean performance from 26.481 to 25.866.Exit 8 for EX-FIQA (T) achieves comparable results with a 33% computational reduction.
  • Fusion Cost: Fusion overhead is minimal for EX-FIQA-FW (T), while EX-FIQA-FW (C) rises from 11.49 to 12.33 GFLOPs, approximately a 7% increase.The C variant incurs additional processing because its fusion uses the feature network over concatenated patches.
  • Operating Points: 50% computational savings are available with EX-FIQA (T) Exit 6, changing FNMR@FMR=1e−3 from 26.481 to 27.070; EX-FIQA (C) Exit 7 saves 41%.The paper identifies weighted fusion as the maximum-accuracy operating point, with minimal or reasonable overhead depending on the variant.

B. Comparison with State-of-the-Art

EX-FIQA-FW compares favorably with established IQA and FIQA methods across four face-recognition models, especially on large-scale challenging datasets. The framework combines intermediate transformer representations while preserving computational efficiency.

  • Benchmark comparison: EX-FIQA-FW (T) consistently achieves top-1 performance on IJB-C across all four face-recognition models.On Adience, EX-FIQA-FW (C) achieves the best performance across all four models and remains competitive on smaller benchmarks.
  • Evaluation scope: The evaluation spans eight benchmark datasets and four state-of-the-art face-recognition models, comparing EX-FIQA against three IQA and twelve FIQA approaches.This broad comparison evaluates performance across multiple recognition-model and dataset settings.
  • Fusion strategy: Depth-weighted fusion consistently outperforms uniform averaging and final-layer predictions across challenging benchmark datasets.The fusion strategy assigns progressively greater importance to deeper transformer blocks.
  • Ethical considerations: FIQA can improve biometric-system reliability but may introduce demographic bias, discriminatory access, and mass-surveillance risks.The paper recommends diverse training data, regular bias testing, legal compliance, oversight, and fair deployment.

APPENDIX

The appendix documents how EX-FIQA quality predictions, attention patterns, exit performance, computational trade-offs, and verification behavior are evaluated across exits and datasets. It combines visualizations, ablations, state-of-the-art comparisons, and verification analyses to characterize deployment-relevant behavior.

  • Experimental overview: The supplementary material organizes experiments around prediction evolution, optimal exit selection, method comparisons, and computational trade-offs.These questions structure the appendix’s experimental analysis.
  • Quality-score evolution: Table III reports quality-score evolution for representative face samples, with higher scores indicating better predicted quality and parenthesized rankings from 1 to 4.The table is intended to show how predictions progressively refine across exits.
  • Ablation analysis: Table IV compares AUC-EDC for all 12 individual exits and fusion strategies across eight datasets using ArcFace, where lower values indicate better assessment.The table includes EX-FIQA-F and EX-FIQA-FW under fixed FMR thresholds of 1e-3 and 1e-4.
  • Visual evidence: Attention visualizations track EX-FIQA (C) and EX-FIQA (T) across 12 exits, representative face types, and multiple benchmark datasets.Figures include child, frontal, occluded, and challenging-pose samples, plus dataset-level mean attention maps.
  • Efficiency analysis: EDC curves and performance-complexity plots evaluate FNMR across discard fractions and relate pAUC-EDC or AUC-EDC to computational cost.These analyses support balancing quality-assessment accuracy against deployment constraints.
  • Comparative and verification results: State-of-the-art comparisons span three FNMR@FMR thresholds, eight benchmark datasets, and four face-recognition models, while IJB-C verification reports TAR at multiple FAR thresholds.The appendix also compares against three IQA and twelve FIQA methods.
Loading 2604.22842v1…