PaperlayerView exact revision on arXiv

Source-linked AI summary

AGVBench: A Reliability-Oriented Benchmark of Data Augmentation for Vein Recognition

Haiyang Li, Yuming Fu, Qun Song, Hongchao Liao, Jing Chen, Mounim A. EI-Yacoubi, Xin Jin

arXiv:2607.02271v1cs.CV

TL;DR

Vein recognition is limited by scarce annotated data and imaging variation, while natural-image augmentations can disrupt identity-critical vascular structure. AGVBench benchmarks 30 augmentation strategies across five datasets and seven architectures, finding that strong clean recognition often conflicts with calibration and adversarial robustness. The results support standardized, multidimensional evaluation for reliable vein-recognition augmentation.

  • Problem

    Vein recognition has limited annotated data, and natural-image augmentation strategies may disrupt the fine-grained topology and textures needed for identity discrimination.

  • Method

    AGVBench evaluates 30 augmentation strategies across five vein datasets and seven architectures using standardized experiments spanning recognition, reliability, robustness, and efficiency.

  • Results

    Multi-image augmentations generally provide stronger recognition, but high clean accuracy is often accompanied by poor calibration and adversarial fragility.

  • Takeaways & Limitations

    Accuracy-centric evaluation is insufficient; reliable vein augmentation requires jointly considering recognition, calibration, and adversarial security.

  • Takeaways & Limitations

    Geometric augmentations consistently degrade performance, while mixup methods can combine high accuracy with ECE above 30% and severe adversarial weakness.

Abstract

from arXiv · show

Vein recognition is a secure biometric technology often constrained by limited annotated data and imaging variations. While data augmentation mitigates this, strategies designed for natural images may disrupt the fine-grained topology and textures essential for identity discrimination. We present AGVBench, which evaluates 30 representative augmentation strategies on five public palm- and finger-vein datasets with seven backbone architectures, covering classic CNNs, vision transformers, and vein-specific recognition models. Our results show that multi-image mixing methods (e.g., MixUp, PuzzleMix, StarMixup) generally provide the strongest recognition performance. However, they are often poorly calibrated and vulnerable to adversarial perturbations, revealing a clear inconsistency between clean accuracy and adversarial security. We also find that severe geometric transformations frequently degrade recognition, which is potentially due to feature misalignment or spatial cropping, and that augmentation effectiveness varies across palm and finger vein datasets. These findings prove that accuracy-centric evaluation is insufficient for biometric augmentation. AGVBench provides standardized protocols to support reproducible research and guide the design of reliable, secure, and robust vein recognition systems. Our codebase is available at https://github.com/Advance-VeinTech-Innovators/AGVBench.

I. INTRODUCTION

AGVBench addresses the lack of standardized, vein-specific evidence about augmentation by evaluating methods across datasets, architectures, and reliability dimensions. It shows that clean-accuracy gains can conflict with calibration and adversarial robustness, motivating multidimensional assessment.

  • Vein recognition faces a small-sample dilemma, while natural-image augmentations may damage the delicate topology and high-frequency details needed for identity discrimination.
  • Existing augmentation methods are insufficiently understood across vascular biometrics, diverse datasets, and CNN or ViT architectures.
  • AGVBench evaluates 30 augmentation methods on five datasets and seven architectures across recognition, calibration, robustness, and efficiency dimensions.
  • Top-performing mixup methods on clean accuracy can exhibit severe calibration errors and adversarial vulnerability, demonstrating the need for multidimensional evaluation.
  • The benchmark establishes standardized protocols and releases a complete library to support reproducible research and robust, vein-specific augmentation design.

II. BACKGROUND AND RELATED WORKS

The paper formulates vein recognition as empirical-risk minimization over labeled vascular images and introduces augmentation to address sparse training data. Augmentation generates virtual samples conditioned on the original empirical distribution, potentially changing latent vein-feature distributions.

  • Vein recognition learns a discriminative mapping that projects raw vascular patterns into a high-dimensional feature space for identity verification.
  • A vein dataset is represented as labeled image pairs, with image dimensions H, W, and C and identity labels drawn from K classes.
  • The model optimizes learnable parameters θ by minimizing empirical risk over the training data.
  • Cross-entropy loss evaluates each sample’s predicted class probability against its identity label.
  • Because small-sample training data sparsely approximates the true distribution, augmentation generates virtual samples near the original data and conditions training on that empirical distribution.
  • Different augmentation mappings can significantly alter the distribution of vein features in latent space.

B. Data Augmentation

AGVBench organizes augmentation into single-image, multi-image, and label-enhancement methods within a unified, modular framework. This specialization addresses the risk that natural-image transformations damage vein topology and discriminative textures.

  • Motivation: Natural-image augmentation strategies can be counterproductive because aggressive transformations may obliterate delicate topological and high-frequency details needed for identity discrimination.Vein patterns are fine-grained and topologically sensitive, making robust feature extraction challenging.
  • Framework: AGVBench provides a unified PyTorch/MMCV framework with modular data, model, augmentation, and execution components.Its codebase separates preprocessing, architectures, augmentation methods, and execution scripts.
  • Single Image Augmentation Methods: Single-image methods apply geometric, photometric, occlusion, and policy-driven transformations to individual vein samples.Examples include Flip, Rotate, Affine Transformation, Blur, Noise, Cutout, RandomErasing, and GridMask.
  • Multi Image Augmentation Methods: Multi-image methods mix samples or labels through interpolation, spatial cutting and pasting, frequency-guided masks, attention guidance, or dynamic optimization.The benchmark includes MixUp, CutMix, ResizeMix, GridMix, RICAP, FMix, SaliencyMix, GuidedMixup, PuzzleMix, and StarMixup.
  • Label Enhancement Methods: Label-enhancement methods soften or dynamically update target distributions to mitigate overconfidence and address noisy annotations.LabelSmooth and ConfidencePenalty use static softening, while Bootstrapping blends original labels with model-guided targets.

B. Backbones

AGVBench evaluates augmentation across general-purpose CNN and transformer backbones alongside vein-specific recognition models. Its experiments span five public palm- and finger-vein datasets and multiple recognition metrics.

  • Backbone Models: The benchmark includes lightweight and residual CNNs, vision transformers, hierarchical attention models, and domain-specific vein recognition architectures.Representative general backbones are MobileNetv2, ResNet18, ViT-small, and Swin Transformer-Tiny.
  • Experimental Pipeline: The experimental pipeline is configuration-driven, with users specifying datasets, augmentations, backbones, and hyperparameters before standardized execution.AGVBench supports reproducible training and evaluation through its configuration and execution workflow.
  • Datasets: Five public datasets cover three palm-vein datasets and two finger-vein datasets with variation in sensors, lighting, and image quality.The datasets are SCUT1100, TJU600, VERA220, FV-USM, and SDUMLA-HMT.
  • Evaluation Metrics: Recognition evaluation uses Top-1 Accuracy, Equal Error Rate, and TAR@FAR=0.0001 to measure classification and stringent verification performance.Lower EER indicates a better balance between false acceptance and false rejection, while TAR@FAR=0.0001 targets stringent security conditions.

1) Calibration:

AGVBench evaluates calibration alongside recognition and robustness under corruption and adversarial perturbations. Calibration is measured by comparing confidence with accuracy across confidence bins.

  • Calibration: Expected Calibration Error partitions predictions into M equally spaced confidence bins and computes a weighted average gap between accuracy and confidence.The weighting uses each bin’s sample count relative to the total number of samples.
  • Calibration: The benchmark investigates how augmentation strategies affect model calibration because accurate neural networks may still be overconfident.Overconfidence is described as predicted probabilities exceeding actual precision, particularly under intense augmentation.
  • Corruption: Corruption evaluation tests generalization against 19 out-of-distribution image distortions modeled after the ImageNet-C protocol.The suite targets environmental and sensor-induced degradations encountered in practical vein-recognition deployments.
  • Adversarial Robustness: White-box adversarial evaluation uses FGSM and PGD attacks to test security under malicious perturbations.PGD uses α = 0.05/255 for 10 iterations within an ℓ∞ perturbation bound of ϵ = 0.2/255.

4) Occlusion:

AGVBench evaluates occlusion tolerance and computational efficiency, then ranks augmentations using Pareto efficiency rather than an arbitrary weighted score. This exposes performance–resource trade-offs beyond accuracy alone.

  • 4) Occlusion:: Spatial occlusion assessment randomly masks continuous square image regions to test whether models rely on global topology rather than local patches.The setting reflects sensor smudges and finger misalignment that can obscure vein patterns.
  • Efficiency: Efficiency is measured using per-epoch training time, peak memory, GFLOPs, and the ratio of augmentation-added learnable parameters.Paug compares additional augmentation parameters with the vanilla backbone’s parameter count.
  • APEX Ranking: APEX identifies non-dominated augmentations across performance, training time, memory, GFLOPs, and added parameters as a multi-objective optimization.This avoids reducing distinct resource and performance dimensions to an arbitrary weighted sum.
  • APEX Ranking: Rank-1 APEX methods cannot improve one metric without degrading at least one other, while successive non-dominated layers receive higher ranks.The ranking removes methods with high computational costs and only marginal performance gains.
  • Experimental Pipeline of AGVBench Codebase: The configuration-controlled workflow specifies datasets, augmentations, backbones, and hyperparameters before standardized training and evaluation scripts run.The framework instantiates registered components and builds the data pipeline from the selected configuration.

A. Experimental Setup

AGVBench evaluates augmentation across diverse vein datasets and models using recognition, verification, calibration, corruption, adversarial, and occlusion criteria. Results show strong performance trade-offs across augmentation families and evaluation dimensions.

  • Dataset and evaluation scope: Experiments cover five vein datasets, seven backbone architectures, and multiple recognition and robustness dimensions.The evaluation includes recognition, verification, calibration, corruption robustness, adversarial robustness, occlusion robustness, and computational efficiency.
  • Recognition performance: MixUp, PuzzleMix, and StarMixup generally lead Top-1 Accuracy, especially on palm-vein datasets.On VERA220 with ResNet18, MixUp and PuzzleMix reach 95.27% and 95.55%, compared with 71.45% for Vanilla.
  • Recognition performance: Geometric transformations such as Flip, Rotate, and Translate frequently reduce accuracy below the baseline.These operations can disrupt the spatial structure of vascular patterns.
  • Reliability evaluation: MixUp improves verification metrics but performs poorly in calibration and adversarial testing.On ResNet18, it reduces EER from 0.30% to 0.07% and raises TAR@FAR=0.0001 from 97.30% to 99.63%, while mixup methods are worse than Vanilla in calibration and adversarial settings.
  • Robustness evaluation: Cutout, CutMix, and PuzzleMix remain stable as occlusion increases from 0% to 50%.Masking-based methods outperform methods that degrade rapidly under spatial information loss.

A. Orthogonality of Augmentations

Composing augmentation families produces stronger results than combining methods within one family. Cross-category combinations improve recognition and verification, with three-level compositions achieving the best reported results.

  • Cross-category composition: Cross-category compositions consistently improve performance, indicating complementary benefits among augmentation families.The study combines single-image, multi-image, and label-enhancement methods on ResNet18 using VERA220 and TJU600.
  • Cross-category composition: AutoAugment plus LabelSmoothing raises VERA220 accuracy from 80.82% to 89.73% and TAR@FAR from 65.09% to 78.64%.Label regularization also improves high-performing mixing methods.
  • Cross-category composition: The full AutoAugment + PuzzleMix + LabelSmoothing composition reaches 98.00% accuracy, 0.56% EER, and 95.27% TAR@FAR on VERA220.On TJU600, it reaches 96.50%, 0.45%, and 96.12%, respectively.
  • Composition comparison: Intra-category combinations provide only marginal gains compared with cross-category fusion.AutoAugment + MixUp reaches 97.00% versus 95.27% for vanilla MixUp on VERA220.
  • Efficiency trade-offs: MixUp offers a strong efficiency–accuracy balance, improving MobileNetv2 accuracy from 71.64% to 95.55% with nearly unchanged inference cost.Higher accuracy can require additional computation and training time.

C. Discussion

The discussion finds that augmentation benefits depend on dataset and modality, while clean recognition gains often conflict with calibration and adversarial robustness. AGVBench therefore supports multidimensional, modality-aware evaluation and future composition research.

  • Dataset dependence: Augmentation produces the largest accuracy gains on smaller datasets, while stricter EER and TAR@FAR metrics reveal benefits on larger datasets.On SCUT1100, MixUp reduces EER from 0.30% to 0.07% and raises TAR@FAR=0.0001 from 97.30% to 99.63%.
  • Reliability trade-offs: Geometric methods can violate vascular topology, while mixup methods combine high accuracy with poor calibration and adversarial fragility.The limitations include ECE above 30% and MixUp accuracy falling to 4.87% under PGD on TJU600.
  • Reliability trade-offs: No single augmentation method resolves the trade-off between recognition accuracy, calibration, and adversarial robustness.MixUp performs strongly on clean metrics but poorly on reliability measures, whereas LabelSmoothing provides stronger adversarial defense with poor calibration.
  • Modality dependence: Mixup-based methods work especially well on palm-vein datasets but can degrade or reverse their benefits on finger-vein configurations.On SDUMLA-HMT, modality-specific structural characteristics affect which augmentation family is optimal.
  • Broader applicability: AGVBench’s modular codebase can be extended to fingerprint, iris, and periocular recognition by reusing augmentation modules and evaluation protocols.New biometric tasks can be integrated through customized preprocessing pipelines.

APPENDIX

The appendix supplies expanded calibration, adversarial, corruption, ROC, and hyperparameter materials. These results extend the main-paper analyses across datasets, backbones, and corruption severities.

  • Calibration: Appendix Table XI reports complete ECE calibration results across VERA220, TJU600, and SCUT1100.Transformer backbones are evaluated only on TJU600 and SCUT1100.
  • Adversarial robustness: Appendix Table XII provides full SCUT1100 adversarial results under FGSM and PGD attacks.The expanded results reinforce that different augmentation families can be favored by accuracy, calibration, corruption, and adversarial robustness.
  • Corruption robustness: Tables XIII and XIV report corruption robustness at C1, C2, and C3 for each backbone.These results distinguish methods that help under mild corruption from those that remain stable under severe corruption.
  • ROC analysis: Figures 8–10 provide additional ROC curves across palm-vein and finger-vein datasets using different backbones.The appendix expands the ROC analysis beyond the main-paper figures.
  • Reproducibility: Table XV lists representative AGVBench augmentation hyperparameter settings, generally adopting defaults from original implementations.These settings support standardized reproduction of the benchmark experiments.
Loading 2607.02271v1…