PaperlayerView exact revision on arXiv

Source-linked AI summary

Graphical Representation for Heterogeneous Face Recognition

Chunlei Peng, Xinbo Gao, Nannan Wang, Jie Li

arXiv:1503.00488v3cs.CV

TL;DR

Heterogeneous face recognition must match images from different modalities, while existing approaches can complicate recognition through transformations or neglect spatial structure. G-HFR models heterogeneous patches with Markov networks and compares their graphical representations using a coupled similarity metric. It reports superior performance across four HFR scenarios.

  • Problem

    Heterogeneous face recognition remains challenging because images from different modalities are difficult to represent homogeneously, and existing methods may rely on transformations or ignore spatial information.

    Source: §1 · p. 1

  • Method

    G-HFR uses Markov networks to represent heterogeneous image patches separately and a coupled representation similarity metric to compare their graphical representations.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    G-HFR achieves superior performance over state-of-the-art methods across viewed-sketch, forensic-sketch, near-infrared, and thermal-infrared scenarios.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    The conclusion identifies spatial information as crucial for face recognition and notes that the graphical representation could also apply to standard face and facial-expression recognition.

    Source: §5 · p. 12

    USTC-NVIE evaluation is limited to 129 subjects because some thermal and visible videos were lost.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Heterogeneous face recognition (HFR) refers to matching face images acquired from different sources (i.e., different sensors or different wavelengths) for identification. HFR plays an important role in both biometrics research and industry. In spite of promising progresses achieved in recent years, HFR is still a challenging problem due to the difficulty to represent two heterogeneous images in a homogeneous manner. Existing HFR methods either represent an image ignoring the spatial information, or rely on a transformation procedure which complicates the recognition task. Considering these problems, we propose a novel graphical representation based HFR method (G-HFR) in this paper. Markov networks are employed to represent heterogeneous image patches separately, which takes the spatial compatibility between neighboring image patches into consideration. A coupled representation similarity metric (CRSM) is designed to measure the similarity between obtained graphical representations. Extensive experiments conducted on multiple HFR scenarios (viewed sketch, forensic sketch, near infrared image, and thermal infrared image) show that the proposed method outperforms state-of-the-art methods.

1 INTRODUCTION

The introduction identifies a spatial-representation gap in heterogeneous face recognition and proposes G-HFR, which models image patches graphically and compares them with a coupled similarity metric. It reports superior performance across four heterogeneous face-recognition scenarios.

1 INTRODUCTION

  • Existing approaches synthesize images, project them into a common space, or use feature descriptors, but may depend on difficult synthesis, lose information, or ignore facial spatial structure.The introduction identifies these limitations as challenges for heterogeneous face matching.Source: §1 · p. 1
  • G-HFR represents overlapping face patches with Markov networks and compares the resulting heterogeneity-independent weight matrices using a coupled representation similarity metric.The metric accounts for spatial structure between heterogeneous image patches.Source: §1 · pp. 1–2
  • G-HFR achieves superior performance over state-of-the-art methods across four HFR scenarios: viewed sketch, forensic sketch, near-infrared, and thermal-infrared recognition.The evaluations include CUFSF, IIIT-D Sketch, PRIP-VSGC, a collected forensic-sketch database, CASIA NIR-VIS 2.0, and USTC-NVIE.Source: §1 · p. 2
  • The paper highlights three contributions: spatially informed graphical representations, a spatially coupled matching metric, and leading accuracies across multiple HFR scenarios.The introduction also states that the paper reviews prior work, presents the approach, reports experiments, and concludes.Source: §1 · p. 2

2 RELATED WORK

The review organizes prior heterogeneous face-recognition methods into synthesis, common-space projection, and feature-descriptor approaches. It also discusses work specifically addressing forensic-sketch matching.

Synthesis-based methods

  • Synthesis methods transform images between modalities, including sketch-photo synthesis and NIR/VIS or thermal-image-to-photo synthesis, with approaches ranging from patch mappings to Markov models.Some methods explicitly model spatial constraints between neighboring patches, while others incorporate the test image during learning.Source: §2 · p. 2

Common space projection based methods

  • Common-space methods learn mappings or metrics for matching heterogeneous images, including CDFE, CCA, CSR, PLS, CMML, and MvDA.The reviewed approaches variously use correlation, constraints, or inter-view and intra-view relationships.Source: §2 · p. 2

Feature descriptor based methods

  • Feature-descriptor methods encode local image information using approaches such as SIFT, MLBP, coupled information-theoretic encoding, LRBP, HAOG, LBP-like filters, and LDoGBP.The review also covers forensic-sketch matching with local descriptors, prototype similarities, and component-based methods.Source: §2 · pp. 2–3

Forensic sketch methods

  • Forensic-sketch research targets matching sketches to mug shots, including composite sketches produced with facial-composite software.Reviewed approaches include multi-scale circular Weber’s local descriptors with evolutionary memetic optimization and component-based matching.Source: §2 · pp. 2–3

3 GRAPHICAL REPRESENTATION FOR HETEROGENEOUS FACE RECOGNITION

The method represents heterogeneous face images through patchwise Markov-network weights, then compares the representations with a coupled similarity metric. It uses spatial compatibility between neighboring patches and illustrates how similarity maps distinguish same-person from different-person pairs.

3 GRAPHICAL REPRESENTATION FOR HETEROGENEOUS FACE RECOGNITION

  • G-HFR derives patchwise graphical representations and compares them with CRSM, without relying on synthesis or projection procedures.The approach is described for sketch-photo recognition using a representation dataset of paired images.Source: §3 · p. 3

3.1 Graphical Representation

  • Each patch is reconstructed from K nearby same-modality patches in the representation dataset, while joint Markov modeling accounts for compatibility between adjacent patches.The neighboring compatibility function favors compatible overlaps between neighboring patches.Source: §3 · p. 3, §3.1 · p. 3

3.2 Coupled Representation Similarity Metric

  • CRSM scores coupled patch pairs by summing weights assigned to shared nearest neighbors, then averages patch scores for image matching.The weights associated with shared neighbors are used to describe semantic similarity between the coupled representations.Source: §3.2 · p. 5
  • On CUFSF examples, same-person sketch-photo pairs tend to show more similarity-map areas above 0.5 than different-person pairs.The maps have the same size as the face images, and bright areas indicate scores above 0.5.Source: Figure 2 · p. 5, §3.2 · p. 5

Algorithm 1. Graphical Representation for HFR (G-HFR)

G-HFR divides face images into overlapping patches, finds nearest neighbors, and computes graphical representations for the probe and gallery images. It then scores their similarity and returns the gallery photo with the largest score.

Algorithm 1. Graphical Representation for HFR (G-HFR)

4 EXPERIMENTS

The experiments evaluate G-HFR across viewed-sketch, forensic-sketch, near-infrared, and thermal-infrared recognition tasks. They examine its graphical representation and similarity metric, then compare its performance with state-of-the-art methods on multiple heterogeneous-face databases.

4 EXPERIMENTS

  • The evaluation covers viewed-sketch, forensic-sketch, near-infrared, and thermal-infrared face recognition tasks.Source: §4 · p. 5
  • Experiments separately assess the graphical representation and CRSM, then examine parameter and feature choices.Source: §4 · p. 5
  • The paper reports superior performance over state-of-the-art methods on multiple heterogeneous-face databases.Source: §4 · p. 6

4.1 Databases

The evaluation uses randomly partitioned datasets to test G-HFR on viewed and forensic sketches, near- and thermal-infrared images, and an enlarged photo gallery. The database descriptions specify the image sources, available pairs, and train/test protocols.

4.1 Databases

  • Experiments use random partitions into representation, training, and test sets, and report accuracies averaged over 10 partitions.Source: §4.1 · p. 6

4.1.1 Viewed Sketch Database

  • CUFSF contains 1,194 viewed-sketch/photo pairs, with 694 people reserved for testing; the easier CUFS database is omitted.Source: §4.1.1 · p. 6

4.1.2 Forensic Sketch Databases

  • Forensic-sketch evaluation includes semi-forensic, software-generated composite, and real-world forensic sketches from three databases.IIIT-D has 140 semi-forensic pairs; PRIP-VSGC has 123 composite sketches, with Identi-Kit examples used for testing; the collected database has 168 real-world sketches, with 112 people for training and 56 for testing.Source: §4.1.2 · p. 6

4.1.3 Near Infrared Database

  • CASIA NIR-VIS 2.0 contains 725 subjects, with one randomly selected NIR and VIS image per subject and 208 pairs reserved for testing.Source: §4.1.3 · p. 6

4.1.4 Thermal Infrared Database

  • USTC-NVIE supplies 129 thermal-infrared/photo pairs, with 60 for representation, 30 for training, and 39 for testing.Source: §4.1.4 · pp. 6–7

4.1.5 Enlarged Gallery

  • An enlarged gallery adds 10,000 photos from 5,329 people across FERET, XM2VTS, CAS-PEAL, and LFW-a to approximate real-world retrieval scenarios.Source: §4.1.5 · p. 7

4.2 Experimental Settings

The experiments use local descriptors and vary spatial modeling, similarity metrics, neighbor counts, and feature fusion. On CUFSF, combining three feature models improves accuracy beyond the individual descriptors, while multiple K-specific similarity metrics also improve recognition.

4.2 Experimental Settings

  • The setup aligns faces using five facial points, crops them to 100 × 125 pixels, and extracts SURF, SIFT, or HOG descriptors from 10 × 10 patches.The reported settings use 50% patch overlap, yielding 456 patches per image; thermal-image facial points are manually located.Source: §4.2 · p. 7
  • Spatial information is important: modeling neighboring-patch relationships with the compatibility function yields superior performance in the reported experiment.Source: §4.2 · p. 7
  • CRSM is more effective than L1 norm and cosine distance because it is designed for sparse representation vectors whose corresponding positions share similar semantic meanings.Source: §4.2 · p. 8
  • Recognition accuracy varies irregularly with K, so the experiments use K values from 15 through 40, which the authors say suffice despite increasing memory needs at larger K.Source: §4.2 · p. 8
  • Fusing six similarity metrics raises accuracy from 92.22% to 94.24%.Source: §4.2 · p. 8, Figure 4 · p. 8
  • 96.04% accuracy is reached by fusing SURF, SIFT, and HOG models, compared with 94.24% (SURF), 89.48% (SIFT), and 89.05% (HOG).The feature and fusion experiments use CUFSF.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 Experiments on the Viewed Sketch Database

On CUFSF, G-HFR outperforms the compared methods on reported recognition measures. The paper links competing methods’ weaker results to synthesis artifacts, poor common-space performance, or descriptors that ignore face structure.

4.3 Experiments on the Viewed Sketch Database

  • G-HFR achieves a (96.04±0.0076)% first match rate and a (99.86±0.0088)% tenth match rate on CUFSF, both with 95% confidence intervals.These results are reported alongside comparison methods in Table 1.Source: §4.3 · p. 9
  • G-HFR records 99.14% verification rate at 0.1% FAR, compared with Zhang et al.’s 98.70% on CUFSF.Source: §4.3 · p. 9
  • On CUFSF, TFSPS synthesis artifacts degrade recognition, PLS and MvDA perform poorly, and LRBP and LDoGBP ignore facial spatial structure.LRBP and LDoGBP achieve 91.12% and 91.04% accuracy, respectively.Source: §4.3 · p. 9

4.4 Experiments on the Forensic Sketch Databases

Across three forensic-sketch databases, G-HFR reports higher rank-50 accuracy than the cited comparison methods. The section also notes that real-world forensic-sketch recognition is difficult and that the available database limits training.

4.4 Experiments on the Forensic Sketch Databases

  • G-HFR achieves rank-50 accuracies of (30.36±0.07)% on IIIT-D, (51.22±0)% on PRIP-VSGC, and (31.96±0.41)% on real-world forensic sketches, outperforming the reported comparisons.The IIIT-D and real-world results include 95% confidence intervals; PRIP-VSGC’s fixed split yields zero standard deviation and confidence interval.Source: §4.4 · pp. 9–10, Figure 6 · p. 9, Figure 7 · p. 10
  • On IIIT-D, G-HFR’s rank-50 accuracy exceeds MCWLD’s 28.52%, while PCA and Fisherface each achieve (10.71±0.07)% and (10.71±0.09)%, respectively.G-HFR uses 124 semi-forensic sketches for training, 168 forensic sketches as probes, and a gallery enlarged with 10,000 photos.Source: §4.4 · p. 9
  • The real-world forensic-sketch database is small, and the authors say recognition performance may improve with more forensic sketches available.The reported split trains on 112 people and tests on 56, with the gallery enlarged by 10,000 photos.Source: §4.4 · p. 10

4.5 Experiments on the Near Infrared Database

On CASIA NIR-VIS 2.0, G-HFR is evaluated under both a one-pair-per-subject training setup and the database’s standard protocols. Its reported accuracy varies across these evaluations and cited methods.

4.5 Experiments on the Near Infrared Database

  • With one NIR-VIS pair per subject for training, G-HFR reaches (54.90±0.30)% rank-1 and (83.32±0.23)% rank-50 accuracy on the paper’s enlarged-gallery test.The setup uses 100 pairs as the representation dataset, 417 for training, and 208 for testing; the gallery adds 10,000 photos.Source: §4.5 · p. 11
  • Under standard View 2 protocols, G-HFR achieves (85.30±0.03)% rank-1 accuracy, below Yi et al.’s reported 86.16% after removing the first 11 PCA components.Dense SIFT with subspace LDA reports 73.28%, and RBM reports 84.22%.Source: §4.5 · p. 11
  • The standard-protocol experiment follows CASIA NIR-VIS 2.0 View 2, uses 150 people per sub-experiment for the representation dataset, and retains the experimental-setting parameters.Source: §4.5 · p. 11

4.6 Experiments on the Thermal Infrared Database

On USTC-NVIE, G-HFR reports high rank-1 and rank-50 accuracy for thermal-infrared-to-visible matching. The evaluation uses a small set of 129 TIR-VIS pairs and a gallery enlarged by 10,000 photos.

4.6 Experiments on the Thermal Infrared Database

  • G-HFR achieves (77.44±2.17)% rank-1 and (95.38±0.91)% rank-50 accuracy on USTC-NVIE, exceeding the reported PCA and Fisherface results.PCA reports (0±0)% at both ranks; Fisherface reports (8.72±1.09)% rank-1 and (36.15±2.50)% rank-50.Source: §4.6 · p. 11, Figure 10 · p. 11
  • The study uses 60 TIR-VIS pairs for representation, 30 for training, and 39 for testing, from 129 total pairs.The gallery is enlarged by 10,000 photos.Source: §4.6 · p. 11
  • The cited synthesis-based method reports 50.06% rank-1 accuracy on a 47-subject gallery, while P-RS reports 46.7% using 667 training and 333 test subjects.Source: §4.6 · p. 11

5 CONCLUSIONS

G-HFR represents heterogeneous faces with Markov networks that retain spatial information and compares those representations using CRSM. Across four HFR scenarios, it achieves superior face-recognition accuracy to state-of-the-art methods.

5 CONCLUSIONS

  • G-HFR combines Markov-network representations of heterogeneous faces with a coupled representation similarity metric (CRSM).The experiments compare the proposed representation and metric with commonly used representations and similarity metrics.Source: §5 · p. 11
  • Across viewed-sketch, forensic-sketch, near-infrared, and thermal-infrared scenarios, G-HFR achieves superior face-recognition accuracy to state-of-the-art methods.Source: §5 · p. 12
  • The authors identify spatial information as crucial for face recognition and state that the graphical representation could also apply to standard face recognition and facial-expression recognition.Future work would investigate more feature types and additional HFR scenarios.Source: §5 · p. 12
Loading 1503.00488v3…