PaperlayerView exact revision on arXiv

Source-linked AI summary

Palmprint Recognition in Uncontrolled and Uncooperative Environment

Wojciech Michal Matkowski, Tingting Chai, Adams Wai Kin Kong

arXiv:1911.12514v1cs.CV

TL;DR

Palmprint research has largely focused on cooperative online capture or high-resolution latent prints, leaving uncontrolled, uncooperative images less studied. The paper introduces an Internet-collected database and an end-to-end alignment and feature-extraction approach, which outperforms existing methods on NTU-PI-v1.

Abstract

from arXiv · show

Online palmprint recognition and latent palmprint identification are two branches of palmprint studies. The former uses middle-resolution images collected by a digital camera in a well-controlled or contact-based environment with user cooperation for commercial applications and the latter uses high-resolution latent palmprints collected in crime scenes for forensic investigation. However, these two branches do not cover some palmprint images which have the potential for forensic investigation. Due to the prevalence of smartphone and consumer camera, more evidence is in the form of digital images taken in uncontrolled and uncooperative environment, e.g., child pornographic images and terrorist images, where the criminals commonly hide or cover their face. However, their palms can be observable. To study palmprint identification on images collected in uncontrolled and uncooperative environment, a new palmprint database is established and an end-to-end deep learning algorithm is proposed. The new database named NTU Palmprints from the Internet (NTU-PI-v1) contains 7881 images from 2035 palms collected from the Internet. The proposed algorithm consists of an alignment network and a feature extraction network and is end-to-end trainable. The proposed algorithm is compared with the state-of-the-art online palmprint recognition methods and evaluated on three public contactless palmprint databases, IITD, CASIA, and PolyU and two new databases, NTU-PI-v1 and NTU contactless palmprint database. The experimental results showed that the proposed algorithm outperforms the existing palmprint recognition methods.

I. INTRODUCTION

The paper addresses palmprint recognition in uncontrolled, uncooperative images, a setting underrepresented by commercial online and latent-print research. It introduces a new Internet-collected database and an end-to-end learning approach.

I. INTRODUCTION

  • In forensic images, hands may remain visible even when faces or tattoos are hidden, making palms a potential identification source.The paper notes that evidence images may come from the Internet, social media, hard drives, or police bookings.Source: §I · p. 1
  • Conventional palmprint pipelines use separate modules and often hand-crafted features, with acquisition conditions commonly controlled.The typical pipeline includes preprocessing, segmentation, ROI extraction, feature extraction, and matching.Source: §I · pp. 1–2
  • Deep learning learns hierarchical representations from data, but its success depends strongly on the amount of training data.The paper situates this approach alongside CNN applications in face, fingerprint, iris, and palmprint recognition.Source: §I · p. 2
  • Existing research mainly covers cooperative online recognition and high-resolution latent prints, not low-resolution images captured without cooperation.The target setting lacks the high-resolution minutiae and ridges used in latent-print identification.Source: §I · p. 2
  • The paper introduces NTU-PI-v1 and an end-to-end deep-learning algorithm to address uncontrolled, uncooperative palmprint recognition.The algorithm includes alignment, feature extraction, and in-network data augmentation.Source: §I · p. 2

II. RELATED WORK

Prior palmprint methods largely assume controlled capture or separately extracted ROIs, while the paper targets end-to-end recognition under uncontrolled conditions. Earlier approaches include contour-based alignment, hand-crafted descriptors, and several deep-learning variants.

A. Palmprint Recognition Methods

  • Traditional palmprint methods locate ROIs from hand contours and landmarks, but depend on clear boundaries and controlled backgrounds.Coding methods can perform poorly on contactless images because of misalignment.Source: A. Palmprint Recognition Methods · pp. 2–3
  • Earlier deep-learning studies used Siamese networks, fixed-filter or pretrained CNNs, and learned descriptors, with varied datasets and evaluation setups.Some studies evaluated on contact-based or manually cropped ROIs rather than contactless, uncontrolled images.Source: A. Palmprint Recognition Methods · p. 3
  • Prior CNN pipelines were not end-to-end trainable because ROI extraction occurred before training, and their alignment considered only linear transformations.The proposed approach instead combines nonlinear alignment with end-to-end training.Source: A. Palmprint Recognition Methods · p. 3

B. Contactless Palmprint Databases

Common contactless benchmarks have limited acquisition diversity and cooperation relative to forensic images. The paper presents NTU-PI-v1 and NTU-CP-v1 as additional databases, including Internet-collected images intended to reflect uncontrolled capture.

B. Contactless Palmprint Databases

  • CASIA, IITD, and PolyU are widely used contactless benchmarks, but their limited image diversity and controlled, cooperative capture make them less suited to forensic settings.CASIA contains grayscale images; IITD and PolyU contain color images, with PolyU collected across two sessions.Source: B. Contactless Palmprint Databases · p. 3

III. NTU PALMPRINT DATABASES

  • NTU-PI-v1 contains 7781 Internet-collected hand images from 2035 palms and 1093 subjects, representing varied demographics and uncontrolled acquisition.The database aims to simulate forensic scenarios; hand regions were manually cropped, with hand detection out of scope.Source: §III · p. 4
  • NTU-PI-v1 varies in gesture, viewpoint, illumination, background, image quality, and resolution; its hand images range from 30 × 30 to 1415 × 1415 pixels.The median full-hand image size is 115 × 115 pixels, and the database also provides annotated landmarks and segmented hand images.Source: §III · p. 5
  • NTU-CP-v1 contains 2478 images from 655 palms and 328 subjects, collected in Singapore across two contactless sessions without strict pose requirements.Its hand-image sizes range from 420 × 420 to 1977 × 1977 pixels, with a median of 1373 × 1373 pixels.Source: §III · p. 5

IV. THE ARCHITECTURE DESIGN

The architecture section motivates landmark-based ROI alignment for challenging hand images. Its landmark configuration supports non-rigid transformation, and a spatial transformer uses a sampling grid to extract the palmprint ROI.

A. Hand Landmarks

  • Traditional contour-based ROI extraction is sensitive to pose variation, elastic deformation, unclear contours, and complex backgrounds.Thresholding and boundary methods rely on visible hand-background contrast and clear spaces between fingers.Source: A. Hand Landmarks · p. 5

IV. THE ARCHITECTURE DESIGN

  • A spatial transformer creates a sampling grid from hand landmarks and samples the hand image into a regular palmprint ROI grid.The figure describes thin plate spline transformation for this ROI extraction.Source: Figure 7 · p. 5

A. Hand Landmarks

  • The proposed nine-landmark configuration spans the palm and supports non-rigid alignment; three points determine only affine transformations.Additional landmarks help avoid cutting across palm information and allow higher degrees of freedom.Source: A. Hand Landmarks · p. 5

B. Spatial Transformer

The Spatial Transformer localizes and aligns palm regions by deforming a sampling grid using hand landmarks. The approach uses a modified pre-trained VGG-16 for alignment and feature extraction.

B. Spatial Transformer

  • ROI extraction aligns palmprints into a common coordinate system.The Spatial Transformer is used for this task.Source: B. Spatial Transformer · pp. 5–6
  • The Spatial Transformer uses a trainable Localization Network, grid generator, and sampler to transform images differentiably within a CNN.Here, hand-landmark coordinates parameterize a thin plate spline deformation of the sampling grid.Source: B. Spatial Transformer · p. 6

C. A Building Block for Palmprint Alignment and Feature Extraction

  • The alignment and feature-extraction building blocks use ImageNet-pretrained VGG-16 pruned at the top layers, with separate parameters for hand images and palm ROIs.The paper motivates this choice by small palmprint databases, overfitting risk, and the low-level line and texture features shared with natural images.Source: C. A Building Block for Palmprint… · p. 6

D. The Proposed Architecture

EE-PRnet combines ROI localization and alignment with palmprint feature extraction and recognition. Experiments use separate database-specific protocols, including a gallery/probe split for NTU-PI-v1.

D. The Proposed Architecture

  • ROI-LAnet aligns hand images and extracts palmprint ROIs, which FERnet processes for recognition using an independent CNN without shared weights.FERnet outputs a 512-dimensional palmprint descriptor and palmprint labels; the Localization Network is trained with L2 loss and FERnet with Softmax loss.Source: D. The Proposed Architecture · pp. 6–7

A. Evaluation Protocols

  • CASIA and IITD use the survey protocol, while PolyU and NTU-CP-v1 follow the protocol in.Across experiments, databases are divided into training and testing sets, and left hands are flipped and treated as different subjects.Source: A. Evaluation Protocols · p. 7
  • For NTU-PI-v1, even per-palm image counts are split randomly 50/50 between training/gallery and testing/probe, with an odd extra image assigned to training.The resulting split contains 4501 training images from 2035 palms and 3380 test images from 1805 palms; no outside palms are used for training.Source: A. Evaluation Protocols · p. 7

B. Implementation Details

The implementation uses pre-trained VGG-16 blocks and specified optimization and image settings. Training proceeds from separate localization and feature-extraction stages to joint end-to-end training, where ROI-LAnet can provide on-the-fly ROI augmentation.

B. Implementation Details

  • The networks use ADAM with momentum 0.9, learning rate 0.001, no weight decay, and batch size 128.ROI-LAnet inputs are resized to 56 × 56 pixels and ROI images to 112 × 112 pixels; pre-trained VGG-16 convolutional blocks are used.Source: B. Implementation Details · p. 7
  • VGG-16, ResNet-50, and GoogLeNet are fine-tuned from ImageNet for comparison, with network-specific lower-layer learning rates and training durations.VGG-16 and GoogLeNet are trained for 20 epochs, while ResNet-50 is trained for 60 epochs.Source: B. Implementation Details · pp. 7–8

C. Training Strategy

  • Training progresses from localization-network training, to FERnet training on palmprint ROIs, then to joint end-to-end training on full hand images.In the end-to-end stage, the Stage II-trained FERnet is not reused.Source: C. Training Strategy · p. 8
  • During end-to-end training, ROI-LAnet dropout and localization-network updates can produce different landmarks and palmprint ROIs from the same hand image.The paper terms this on-the-fly, in-network ROI augmentation and illustrates it with corresponding hand landmarks and ROI outputs.Source: C. Training Strategy · p. 8, Figure 9 · p. 8, Figure 10 · p. 8

D. Localization Network Training

On NTU-PI-v1, the connected, end-to-end strategy with PLS outperforms separated training, while omitting ROI extraction sharply reduces performance. Landmark localization is initialized before recognition training and further refined within EE-PRnet.

D. Localization Network Training

  • The Localization Network is initialized on annotated NTU-PI-v1 hand images using L2 loss, augmented rotations, and staged training of regression layers followed by convolutional layers.The regression layers are trained for 10 epochs before the convolutional layers are unfrozen for 15 more.Source: D. Localization Network Training · p. 8
  • 1.99±0.36% NME: the trained Localization Network’s landmark accuracy is used for initialization, while EE-PRnet later fine-tunes it with Softmax loss.The paper notes that some landmarks are not well defined and their ground-truth annotations may have higher errors.Source: D. Localization Network Training · p. 8

E. Experiments on NTU Palmprints from the Internet

  • The training-strategy comparisons vary end-to-end connection, dropout use, ROI-LAnet block fine-tuning, and dropout decay, alongside a no-color-augmentation condition.The strategy set also includes direct hand-image input to FERnet, contrasted with ROI-based input.Source: E. Experiments on NTU Palmprints from… · p. 9
  • 3.11% at rank-1 and 3.67% at rank-30: connected end-to-end training with PLS outperforms separated networks on NTU-PI-v1; omitting ROI extraction drops performance by 23.13%.The comparisons are S0 versus S5 for separated versus connected training, and S0h versus S0 for direct hand-image input versus ROI extraction.Source: E. Experiments on NTU Palmprints from… · p. 9, Table IV · p. 9

F. Experiments on CASIA, IITD, and PolyU Contactless Palmprint Benchmark Databases

The experiments test whether EE-PRnet transfers to CASIA, IITD, and PolyU, including fine-tuning a Localization Network pretrained on NTU-PI-v1. The reported CASIA results also favor proposed strategies combined with spatial augmentation.

F. Experiments on CASIA, IITD, and PolyU Contactless Palmprint Benchmark Databases

  • CASIA images were zero-padded, IITD images centrally cropped, and PolyU hands segmented using Otsu thresholding to prepare square inputs.Source: F. Experiments on CASIA, IITD, and… · p. 9
  • A Localization Network pretrained on NTU-PI-v1 generalized to CASIA, IITD, and PolyU and could be fine-tuned for each database (S2 vs. S4/S5).The network was trained for 40 epochs; the learning rate in block D was increased for S4 and S5 because that block was pretrained on NTU-PI-v1.Source: F. Experiments on CASIA, IITD, and… · p. 9
  • CASIA’s grayscale images used aggressive spatial augmentation at; supplied IITD and PolyU ROIs were also used to train FERnet under S0.Source: F. Experiments on CASIA, IITD, and… · pp. 9–10
  • On CASIA, the proposed strategies combined with augmentation at benefited performance despite the database’s visual distance from NTU-PI-v1.Source: Table IV · p. 10

G. Comparison with the State-of-the-Art Palmprint Recognition Methods

EE-PRnet is compared with existing palmprint and deep-learning methods across five databases. It leads on the reported NTU-PI-v1 identification results and outperforms considered methods on verification EER across four databases.

G. Comparison with the State-of-the-Art Palmprint Recognition Methods

  • The comparison used S5-trained EE-PRnet and database-specific augmentation, while other methods received the proposed ROI-LAnet crops on NTU-PI-v1 and NTU-CP-v1.Without ROI-LAnet, the other methods’ preprocessing could not handle those databases’ complex backgrounds or finger gaps.Source: G. Comparison with the… · p. 10
  • 41.92% rank-1 and 64.73% rank-30 accuracy on NTU-PI-v1; EE-PRnet outperformed existing methods by 12.58% at rank-1 and 14.79% at rank-30.Source: G. Comparison with the… · p. 11
  • EE-PRnet achieved the highest identification performance on CASIA and IITD and 99.77% rank-1 accuracy on PolyU, where DoN was reported at 100%.The paper notes that DoN’s parameters were carefully tuned for PolyU, whereas EE-PRnet used fixed training/testing policies and no post-processing.Source: G. Comparison with the… · p. 11
  • 0.76%, 0.26%, 0.15%, and 0.73% EER on NTU-CP-v1, IITD, PolyU, and CASIA, respectively, were best among considered methods on all four databases.The respective margins over the second best were 7.00%, 0.25%, 0.07%, and 0.06%.Source: G. Comparison with the… · p. 11
  • On Tongji and REST, EE-PRnet achieved 98.63% and 93.34% rank-1 accuracy, respectively, comparable to their corresponding studies; DDBPD comparisons were also comparable on IITD, CASIA, and Tongji.DDBPD comparison was unavailable for other databases, including NTU-PI-v1, because its code was not publicly available.Source: G. Comparison with the… · p. 11

H. Performance Analysis

On NTU-PI-v1, performance analysis relates rank-1 accuracy to training-sample availability and image size. More training samples matter more than image size, and some evaluated ROIs are very small.

H. Performance Analysis

  • 95.24% rank-1 accuracy was reached with 8 training samples; performance rose almost linearly with sample count, although each palm averages only 2.88 samples.Source: H. Performance Analysis · p. 11
  • Image size affected performance less than sample count, even though the first two bins correspond to approximate ROIs of only 20×20 or 40×40 pixels.Source: H. Performance Analysis · pp. 11–12
  • The 40–80-pixel full-hand cases illustrate the database’s challenge; the authors state that a forensic database should reflect varied scenarios.Source: H. Performance Analysis · p. 12

VI. CONCLUSIONS AND DISCUSSION

The paper addresses palmprint identification in uncontrolled forensic images, which differ from cooperative commercial captures and high-resolution latent prints. It introduces NTU-PI-v1 and EE-PRnet, reporting gains over existing methods across five databases.

VI. CONCLUSIONS AND DISCUSSION

  • Uncontrolled forensic evidence may lack faces, tattoos, or high-resolution palm features, while hands can remain visible in images where faces are hidden.Source: §VI · p. 13
  • NTU-PI-v1 contains 7781 Internet images from 2035 palms, aims to simulate forensic scenarios, and is described as the largest publicly available database by palms and subjects.Source: §VI · p. 13
  • EE-PRnet outperformed existing methods across five databases and performed significantly better on NTU-PI-v1 even when competitors used ROI-LAnet for ROI extraction.Source: §VI · p. 13
  • EE-PRnet combines nonlinear alignment, feature extraction, and in-network ROI augmentation in an end-to-end trainable network.The paper contrasts this with prior pipelines using traditional ROI extraction and linear alignment.Source: §VI · p. 13
  • On NTU-PI-v1, rank-1 and rank-30 accuracy were 41.92% and 64.73%; higher-rank results can reduce suspect lists and investigators’ manual searches.Source: §VI · p. 13

Related papers

Loading 1911.12514v1…