PaperlayerView exact revision on arXiv

Source-linked AI summary

One-to-many face recognition with bilinear CNNs

Aruni RoyChowdhury, Tsung-Yu Lin, Subhransu Maji, Erik Learned-Miller

arXiv:1506.01342v5cs.CV

TL;DR

Face identification requires evaluating a probe against a gallery, while verification results on LFW were approaching 100% and IJB-A introduced a challenging identification benchmark. The paper applies B-CNNs with fine-tuning and template pooling, and also converts pretrained VGG-Face into a B-CNN without further network training. Fine-tuned B-CNNs outperform standard CNNs across both pooling strategies, while conversion of VGG-Face also improves accuracy; the authors identify limited gains without retraining as a boundary.

  • Problem

    As LFW verification rates approached 100%, a new protocol was needed to advance face recognition through the distinct one-to-many identification task.

    Source: §1 · p. 1, §1.2 · p. 2

  • Method

    The paper applies bilinear CNNs to IJB-A, evaluates feature-level max-pooling across template images, and converts pretrained VGG-Face into a B-CNN without further network training.

    Source: §1 · pp. 1–2, §1.3 · p. 2, Network architectures · p. 5

  • Results

    Fine-tuned B-CNNs outperform standard CNNs for both pooling strategies, and a B-CNN converted from pretrained VGG-Face improves the network’s final accuracy.

    Source: §1 · p. 1, §5 · p. 6

  • Takeaways & Limitations

    The authors consider B-CNNs a sensible starting point for continued face-identification experiments.

    Source: §6 · p. 8

    Without training the bilinear model, conversion of VGG-Face yields only a small improvement; the authors believe retraining on a suitably large dataset is needed for a significant boost.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

The recent explosive growth in convolutional neural network (CNN) research has produced a variety of new architectures for deep learning. One intriguing new architecture is the bilinear CNN (B-CNN), which has shown dramatic performance gains on certain fine-grained recognition problems [15]. We apply this new CNN to the challenging new face recognition benchmark, the IARPA Janus Benchmark A (IJB-A) [12]. It features faces from a large number of identities in challenging real-world conditions. Because the face images were not identified automatically using a computerized face detection system, it does not have the bias inherent in such a database. We demonstrate the performance of the B-CNN model beginning from an AlexNet-style network pre-trained on ImageNet. We then show results for fine-tuning using a moderate-sized and public external database, FaceScrub [17]. We also present results with additional fine-tuning on the limited training data provided by the protocol. In each case, the fine-tuned bilinear model shows substantial improvements over the standard CNN. Finally, we demonstrate how a standard CNN pre-trained on a large face database, the recently released VGG-Face model [20], can be converted into a B-CNN without any additional feature training. This B-CNN improves upon the CNN performance on the IJB-A benchmark, achieving 89.5% rank-1 recall.

1. Introduction

The paper applies bilinear CNNs to one-to-many face identification on IJB-A, where varied poses and multi-image templates make identification a distinct challenge from verification. It reports that B-CNNs outperform alternatives after fine-tuning and can also improve a pretrained face CNN without further network training.

1.1. The IARPA Janus benchmark A

  • IJB-A was designed to renew face-recognition evaluation as LFW verification rates approached 100%, and its hand-gathered images include wider pose variation, stills, videos, and multi-image templates.A test-time probe is typically a template containing multiple images.Source: §1 · p. 1, §1.1 · p. 2

1. Introduction

  • B-CNNs consistently outperform standard CNNs after fine-tuning on generic face data and IJB-A, while a converted pretrained VGG-Face B-CNN improves accuracy without further network training.The comparison covers both pooling strategies for fine-tuned networks.Source: §1 · pp. 1–2
  • The paper applies bilinear CNNs to IJB-A and reports results across baseline, FaceScrub-fine-tuned, and IJB-A-further-fine-tuned networks.It also investigates feature-level pooling alongside score pooling for multiple probe images.Source: §1 · p. 1

1.2. Verification versus identification

  • Unlike verification, identification uses a gallery of known subjects to learn individual models; IJB-A typically provides multiple images and clips per gallery subject.The paper identifies gallery adaptation as critical to its performance.Source: §1.2 · p. 2

1.3. Pose variability and multiple images

  • To handle pose variation across template images, the paper adopts max-pooling over feature descriptors rather than simply averaging descriptors, which can overcount frequent viewpoints.The approach draws on the multi-view CNN strategy of Su et al. [23].Source: §1.3 · p. 2

1.4. Face identification as fine-grained classification

  • Face identification is fine-grained because individuals have large within-class variation and often subtle between-class differences; the paper applies B-CNNs, developed for fine-grained classification, to IJB-A.The paper reports substantially exceeding the benchmark performance reported for face identification.Source: §1.4 · p. 3

2. Introduction to CNNs

CNNs provide generic learned features, but pose variation can obscure subtle class differences in fine-grained recognition. The cited approaches either localize parts or use texture-style descriptors, while B-CNN filters can respond to facial features and other image cues.

2. Introduction to CNNs

  • Deep CNNs built from convolutional, pooling, and nonlinear layers learn generic features on large labeled datasets such as ImageNet.The paper notes applications spanning image classification, detection, segmentation, and texture recognition.Source: §2 · p. 3
  • CNNs paired with part detectors have achieved state-of-the-art fine-grained recognition, but pose variation can overwhelm subtle category differences and part detection requires annotation and computation.The paper illustrates B-CNN filters responding to facial features, accessories, hair, and potentially informative background cues.Source: §2 · p. 3, Figure 1 · p. 3
  • Texture-style representations such as BoVW, Fisher vectors [21], and VLAD avoid explicit part modeling and annotations, while pose-normalized CNNs can outperform them.Deep Fisher vectors use convolutional features, and examples of pose-normalized CNN approaches include DeepFace and bird-species models [1].Source: §2 · p. 3

3. Bilinear CNNs

Bilinear CNNs combine two networks’ convolutional features into an image-level descriptor. The model can be trained from image labels and adapted from pretrained networks without part annotations.

  • At each image location, two CNNs’ convolutional outputs are outer-producted and pooled into an orderless descriptor for the image.Source: §3 · pp. 3–4
  • Signed square-root normalization followed by ℓ2 normalization is applied to the bilinear feature.Source: §3 · p. 4
  • A bilinear vector can resemble a part-based representation or, approximately, a Fisher vector when its feature extractors serve those respective roles.Source: §3 · p. 4
  • B-CNNs can be trained from image labels without ground-truth part annotations, and pretrained generic networks can be fine-tuned for face recognition.Source: §3 · p. 4
  • With identical networks, the bilinear layer resembles a quadratic polynomial SVM kernel, while pooling across image locations and end-to-end training distinguish its feature computation.Source: §3 · p. 4

4. Experiments

The experiments describe the IJB-A identification protocol and the datasets, preprocessing, network variants, fine-tuning settings, pooling strategies, and pretrained-model conversion used for evaluation.

4.1. Datasets and protocols

  • Across 10 IJB-A splits, training has 333 identities, the gallery 112, and probes 167, including 55 distractors; CMC reports matched-probe recall, while DET reports FNIR at FPIR 0.1 and 0.01.Source: §4.1 · p. 4

Pre-processing

  • IJB-A bounding boxes define face crops, which are resized to each network’s prescribed dimensions without preserving aspect ratio.Source: Pre-processing · p. 5

Network architectures

  • CNN uses rectified fc7 as a 4,096-dimensional descriptor; B-CNN combines layer-14 maps from two M-nets into a normalized 512 × 512 bilinear descriptor.Source: Network architectures · p. 5

Network fine-tuning

  • Fine-tuning compares ImageNet-only, FaceScrub-adapted, and FaceScrub-plus-IJB-A-train models; FaceScrub training excludes overlapping identities and uses 513 identities and 89,045 images.Source: Network fine-tuning · p. 5, §4.1 · p. 5

Classifiers and pooling

  • One-versus-rest linear SVMs train on gallery images individually; at test time, max pooling combines either per-image scores or features within each probe template.Source: Classifiers and pooling · p. 5

Conversion of a pre-trained CNN to a B-CNN

  • The VGG-Face CNN, trained on 2.6 million images without identity overlap with IJB-A, is replicated as a symmetric B-CNN; gallery SVM training leaves its core network parameters unchanged.Source: Conversion of a pre-trained CNN to a… · p. 6

5. Results

On IJB-A identification, fine-tuned B-CNNs outperform standard CNNs and the reported benchmark baselines. A B-CNN built from pre-trained VGG-Face also improves on its CNN baseline without additional training, though the gain is small.

Comparison of fine-tuning methods

  • 52.5% versus 44.5%: FaceScrub-fine-tuned B-CNN outperforms the standard CNN by almost 10% at rank-1.Further IJB-A fine-tuning raises rank-1 performance from 44.5% to 53.6% for CNN and from 52.5% to 58.8% for B-CNN; ImageNet-pretrained models score 28.9% and 31.2%, respectively.Source: Comparison of fine-tuning methods · p. 6, Table 1 · p. 6

Comparison of template-pooling methods

  • Feature pooling either outperforms score pooling or has negligible difference for fine-tuned B-CNNs, while the two schemes achieve almost the same accuracy for CNNs.The authors attribute the B-CNN preference to learned semantic part filters whose detections can be combined across views; the impact is less evident at higher ranks.Source: Comparison of template-pooling methods · p. 6
  • The B-CNN consistently improves relative performance over the standard CNN across ranks and outperforms the GOTS baseline by a large margin.Figure 4 compares the CNN and B-CNN using feature-pooled templates, with FaceScrub-only fine-tuning and additional IJB-A fine-tuning.Source: Comparison of fine-tuning methods · p. 6, Figure 4 · p. 7

Comparison with IJB-A benchmark

Run-time and feature size information

  • Feature pooling is twice as fast as score pooling on average, with negligible difference in accuracy.B-CNN encoding takes about 0.0764 seconds per image, and one-versus-rest SVM training takes about 17 seconds per gallery person using 262,144-dimensional features.Source: Run-time and feature size information · pp. 7–8

5.2. Pre-trained networks

  • 89.5% versus 89.2%: a B-CNN made from pre-trained VGG-Face outperforms the unmodified CNN baseline without fine-tuning.The B-CNN also exceeds Chen et al.'s method at 86%; without training the bilinear model, the improvement over VGG-Face CNN is small.Source: Table 3 · p. 8, §5.2 · p. 8

6. Discussion

The discussion presents bilinear CNNs as a promising direction for face identification, both after fine-tuning and as modifications to pretrained models. It also proposes larger-scale training and template-aware objectives as directions for further exploration.

6. Discussion

  • A Multi-view CNN-like objective could train the network parameters with max pooling across multiple images in a template.This is proposed as a way to retrain the entire model.Source: §6 · p. 8
  • Training with a dataset larger than FaceScrub, such as CASIA WebFaces, is expected to further improve performance.The passage describes CASIA WebFaces as containing 10,000 identities and half a million images.Source: §6 · p. 8
  • Training a very deep architecture from scratch on a sufficiently large face dataset is another proposed direction.Source: §6 · p. 8
  • B-CNNs’ success relative to nonbilinear architectures makes them a sensible starting point for further experiments.The discussion notes that CNN architectures successful in other fine-grained recognition problems also perform well at face identification after fine-tuning or as modifications to pretrained models.Source: §6 · p. 8

Related papers

Loading 1506.01342v5…