Source-linked AI summary
Video-Based Palm-Vein Authentication under Challenging Conditions
Xiaofeng Yan, Kechen Liu, Abhilash Venkatesh, Cathy Zhang, Xia Zhou, Salvatore Stolfo
TL;DR
Palm-vein recognition lacks public evidence under realistic surface degradation and video capture conditions. The paper introduces CUP and a spatial-temporal matching design, finding broad robustness gains across degraded surfaces, backbones, and datasets while identifying two preliminary warm-condition group differences.
Problem
Public palm-vein evidence has limited coverage of surface degradation, while pooled embeddings can let corrupted regions damage comparisons.
Method
The paper introduces CUP and combines temporal consensus with a parameter-free matcher that fuses global similarity and saliency-steered region-level optimal transport.
Results
The full design leads on every CUP surface and metric, improves four frozen backbones without retraining, and transfers to four public single-image datasets.
Takeaways & Limitations
CUP supports more realistic palm-vein robustness evaluation, and matching-time regional comparison transfers across encoders and datasets.
Takeaways & Limitations
CUP does not yet cover cross-session use, other contamination types, sensor noise, degraded enrollment, or the full deployment distribution.
Abstract
from arXiv · showhide
Palm-vein biometrics are increasingly used for secure, contactless authentication. Yet real-world deployment exposes them to surface noise (sweat, dirt), illumination and motion variation, and temperature-driven changes in vascular visibility, which remain underexplored for lack of data captured under such conditions. To study these effects, we introduce the Columbia University Palm-vein (CUP) dataset, to our knowledge the first public video-based palm-vein dataset. CUP records every palm under four surface conditions (a clean baseline, warm, wet, and dirty) and pairs each subject with physiological and demographic metadata. On it we benchmark twenty-one recognizers spanning static, video, and multi-frame aggregation architectures. Models that verify reliably on clean palms lose most of their accuracy on dirty ones, and the mean equal error rate (EER) roughly quadruples. We recover much of that robustness along both axes of the capture. Temporally, a consensus over the few frames the sensor already returns cancels transient corruption; spatially, a test-time matcher that adds no learned parameters fuses the global cosine with a saliency-steered region-level optimal transport that routes the comparison around corrupted regions. The full design leads on every surface of CUP in EER, TAR@FAR=0.01, and Rank-1, at 4.3M parameters and 3.1 GFLOPs, a fraction of the video models' cost. Attached to four frozen state-of-the-art backbones it cuts their mean EER by 29-37% without retraining, and on four public single-image datasets the regional matching alone still helps. A preliminary audit across ten demographic and physiological traits finds two warm-condition gaps, along body water and gender, that survive multiple-comparison correction. CUP will be released for non-commercial research use at https://github.com/MobileX-CU/CUP_v1 upon publication.
I. INTRODUCTION
The paper introduces CUP to measure palm-vein recognition under realistic surface degradation and evaluates twenty-one recognizers. It then combines temporal consensus with parameter-free regional matching to recover robustness, while using metadata for a preliminary group audit.
- Dataset and benchmark: Mean EER roughly quadruples from 4.11% to 17.35% as captures move from clean to dirty palms across the benchmarked recognizers.The degradation is reported as severe and universal, with clean verification accuracy largely lost on dirty palms.
- A matcher built on what the benchmark reveals: Temporal consensus and saliency-steered region-level optimal transport recover information from clips and route matching around corrupted regions without adding learned matcher parameters.The method retains the global embedding while adding regional comparison and frame consensus.
- A matcher built on what the benchmark reveals: The full method leads on every CUP surface and metric, improves four frozen backbones without retraining, and generalizes to four public single-image datasets.A few evenly sampled frames realize the full multi-frame gain, while consecutive frames are largely redundant.
- A dataset-enabled fairness signal: A preliminary audit across ten demographic and physiological traits finds two warm-condition differences, along body water and gender, that survive multiple-comparison correction.The authors present these findings as signals for larger studies rather than a characterization of fairness.
- Dataset and benchmark: CUP is, to the authors’ knowledge, the first public video-based palm-vein dataset with four surface conditions and paired metadata.The benchmark uses a matched protocol across twenty-one recognizers.
II. RELATED WORK
Prior palm-vein datasets broadened capture conditions but generally omitted surface degradation. Existing recognizers also commonly collapse spatial evidence into one pooled embedding, motivating matching that preserves reliable regions.
- A. Contactless Palm-Vein Datasets: Earlier palm-vein datasets expanded scale, wavelengths, contactless capture, or other variation, but did not include surface degradation.CUP adds degraded conditions, near-infrared video, and paired demographic and physiological metadata.
- B. Palm-Vein Recognition: Most palm-vein recognizers reduce each capture to a single holistic descriptor before matching.This pattern spans handcrafted and deep recognition pipelines.
- B. Palm-Vein Recognition: Pooling lets wet or dirty regions drag down similarity even when most of the vein pattern remains intact.Region-aware training does not fully solve the problem because regional structure is collapsed again at comparison time.
- B. Palm-Vein Recognition: Matching reliable regions at comparison time remains less explored than region-aware representation learning.The related work therefore motivates a matcher that preserves regional evidence after feature extraction.
C. Matching under Partial Corruption
The paper frames partial palm-vein corruption as a spatial and temporal matching problem. CUP supplies continuous captures with controlled surface stressors, while the proposed approach uses region-level transport and video information to preserve reliable evidence.
- Spatial and temporal matching: Optimal transport provides a principled way to compare local descriptor sets while shifting emphasis toward reliable regions.Entropic regularization enables fast, differentiable transport through the Sinkhorn algorithm.
- Spatial and temporal matching: Palm-vein matching cannot rely on canonical parts or an external trust oracle because degradation is diffuse attenuation rather than a discrete occluder.The transport must therefore discover which regions to trust.
- Spatial and temporal matching: The method keeps a strong global recognizer and adds saliency-steered region transport, combining frames at matching time so reliable regions decide the comparison.This design preserves global evidence when local patches are misleading.
- CUP acquisition protocol: CUP records continuous NIR video with natural pose and distance variation under clean, warm, wet, and dirty conditions.The protocol uses two-second clips and repeats the movement protocol across all four surface conditions.
- CUP acquisition protocol: CUP contains 5,049 curated two-second clips from 109 subjects with demographic and physiological metadata.The release is intended for non-commercial research use.
B. Data Processing
CUP preprocessing detects hand landmarks, constructs a canonical palm ROI, smooths landmark jitter across frames, and filters clips through automated quality screening followed by manual review.
- 21 hand landmarks provide stable anchor points for constructing the palm ROI under unconstrained video capture.The method selects four anatomically stable points from the detected landmarks before defining the ROI polygon.
- The ROI is the polygon A-B-F-E, warped by perspective transformation into a canonical square image.
- A moving average over 2k+1 frames, with k=2, smooths landmark coordinates before ROI extraction.This low-pass filtering reduces quantization noise and small detection errors, stabilizing alignment across frames.
- Clips undergo automated screening using skewness, aspect ratio, shadow, and entropy, followed by manual review.
C. Data Analysis
CUP exhibits substantially greater real-world variability than public datasets, while standard recognizers degrade sharply under dirty conditions and motivate dual-view video matching.
- C. Data Analysis: 2–3× greater within-identity dispersion separates CUP from every public set under every tested backbone.The increased scatter reflects surface, pose, and distance variation across repeated captures of the same palm.
- C. Data Analysis: CUP benchmarks static and multi-frame recognizers under clean-only and mixed-surface training, evaluating performance separately for each surface.
- C. Data Analysis: A strong static model rises from 3.3% EER on clean palms to 16.5% on dirty palms, while the lowest dirty-condition EER remains about 12%.Mixed-surface training and frame aggregation mitigate degradation only partially.
- A. Dual-View Encoder: The encoder produces a global embedding and a P × P grid of regional descriptors from each input clip.The shared encoder processes probe and gallery clips, preserving both holistic and spatial feature views.
- C. Data Analysis: Mixed-surface training lowers open-set EER toward dirty conditions but leaves a substantial residual above the clean baseline floor.
- A. Dual-View Encoder: Frame averaging and ℓ2 normalization produce aggregated descriptors, with averaging intended to cancel transient corruption while stable vein structure accumulates.
- A. Dual-View Encoder: The dual-consensus objective supervises both pooled clip embeddings and individual frame embeddings with a shared ArcFace classifier.The region descriptors receive no separate loss because region trust is selected during transport-based matching.
B. Region Path: Entropic Optimal-Transport Matching
The region path compares probe and gallery grids with entropy-regularized optimal transport, using saliency-weighted gallery marginals to emphasize discriminative regions and route mass around corruption.
- Optimal transport assigns mass softly between probe and gallery regions rather than selecting a single best-matching region.Mass balance limits any single region pair to at most 1/R of transported mass, requiring agreement across many regions.
- The transport cost is the cosine distance between gallery region i and probe region j, with R=P^2 regions per grid.
- Entropy regularization and Sinkhorn iterations solve the transport problem under gallery and probe marginal constraints.The transport polytope couples the two marginals, while ε controls the entropy regularization weight.
- The region score accumulates cosine similarity exactly where the transport plan places mass.With a unit-mass plan, the transported similarity is equivalent to one minus the transported cost.
- Gallery saliency increases transport budget for discriminative regions identified from clean gallery descriptors.This avoids spreading transport uniformly and concentrates matching on regions more useful for identity discrimination.
C. Complementary Fusion
Complementary fusion combines a global cosine path with the region score after per-probe standardization, allowing each path to compensate for the other’s failure cases.
- The global path uses cosine similarity between aggregated embeddings and remains useful for clean captures or locally ambiguous patches.
- Per-probe standardization makes the heterogeneous global and region scores commensurable across the gallery.The two scores have different scales and gallery-score spreads, so each is normalized using its gallery-set mean and standard deviation.
- A fixed weighted sum fuses the standardized scores, and identity is selected by arg maxG S(G).The fusion weight remains fixed across surfaces and databases and introduces no trained parameters.
- The region and global paths fail on different probes, so their fusion outperforms either path alone.
V. EXPERIMENTS
The experiments benchmark the method on CUP, dissect its components and temporal behavior, test transfer across backbones and databases, and audit fairness.
- The study benchmarks twenty-one recognizers on CUP, analyzes component and frame-budget effects, evaluates transfer, and conducts a preliminary fairness audit.
A. Experimental Setup
The evaluation uses CUP and four public palm-vein databases under open-set protocols, with broad architectural baselines and fixed implementation settings.
- Data: CUP contains 210 palms and 5,049 two-second clips across four surface conditions, while four public datasets provide single-image evaluation.
- Protocol: Training and test palms are disjoint, and recognizers are evaluated separately on each surface in a gallery–probe setup.
- Baselines: The benchmark compares twenty-one vascular-specific, convolutional, transformer, and video recognizers.
- Implementation: Inputs are 224×224 single-channel regions with affine and gamma augmentation, using shared optimization schedules across models.
- Implementation: The proposed method uses dual-consensus training and a fixed test-time region matcher with a 7×7 grid and entropic optimal transport.
B. Results and Analysis on CUP
On CUP, the proposed matcher leads across surfaces and metrics, with gains driven by complementary regional and global views, sparse temporal sampling, and robustness to subject-disjoint evaluation.
- Benchmark: The method achieves the best EER, TAR@FAR=0.01, and Rank-1 on every CUP surface.
- Benchmark: The region-OT matcher surpasses NAN and CAFace on every surface and metric despite sharing their EfficientNet-B0 backbone.
- Ablations: 32% relative mean EER reduction results from fusing global and regional views, while saliency refinement and dual-consensus training further reduce error.
- Frame budget: Mean EER falls from 4.48% at K=1 to 3.26% at K=4 and remains near 3.2%–3.3% through K=60 with evenly spaced sampling.
- Deployment cost: The matcher adds 6.6 ms per probe against 0.06 ms for raw cosine, while encoding costs 4.4 ms.
- Subject-disjoint evaluation: The method’s mean EER is 29% below NAN on the standard split and 50% below it under subject-disjoint evaluation.
C. Generalization
Test-time temporal consensus and regional matching transfer across frozen backbones and public single-image databases without retraining.
- Across backbones: The full test-time design improves every frozen backbone on every surface and metric, reducing mean EER by 29% to 37% without retraining.
- Across databases: On four public single-image databases, regional matching lowers EER and raises TAR@FAR=0.01 in fifteen of sixteen backbone–database combinations.
D. User-Group Fairness
CUP’s fairness audit is preliminary: it tests recognition differences across ten demographic and physiological traits using per-subject EERs, finding two warm-condition signals that survive multiple-comparison correction. The authors caution that these signals rest on a small subject pool and the warm surface alone.
- Audit design: The audit compares per-subject EER across ten demographic and physiological traits, retaining only groups with at least three subjects.The analysis uses recorded labels, standard BMI cut-offs, and tertiles for continuous traits; it is restricted to clean and warm captures.
- Scope and limitations: The fairness conclusions are limited by a small subject pool and coverage of only clean and warm captures.The paper cautions against over-reading effect magnitudes and calls for larger, adequately powered studies.
- Findings: The figure contrasts worst-group and best-group per-subject EER for ten traits on clean and warm surfaces.Dark and gray indicate worst- and best-group EER, while stem color marks the rank-permutation p-value.
- Audit design: Exact rank-based permutation tests use 20,000 label permutations to handle ties and small samples.Mann–Whitney U is used for two groups and Kruskal–Wallis H for three groups.
- Findings: Two warm-condition differences, along body water and gender, survive multiple-comparison correction.The authors report these as preliminary signals rather than a characterization of palm-vein fairness.