PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Comparative Attention Networks for Person Re-identification

Hao Liu, Jiashi Feng, Meibin Qi, Jianguo Jiang, Shuicheng Yan

arXiv:1606.04404v2cs.CV

TL;DR

Person re-identification must handle missing spatial and temporal cues alongside large cross-camera appearance variation, while many methods inspect whole images or fixed local parts only once. CAN recurrently compares raw person-image pairs across multiple glimpses, adaptively focusing on discriminative regions and integrating their features. Experiments report that CAN outperforms state-of-the-art methods in most cases, including 35.9% mAP on Market-1501 single-query evaluation and new state-of-the-art performance under multiple-query evaluation.

  • Problem

    Image-based person re-identification lacks spatial and temporal cues and faces variations in lighting, viewpoint, pose, color, occlusion, and resolution.

  • Method

    CAN uses recurrent comparative attention on raw person-image triplets to localize multiple discriminative regions without manual annotations and integrate their features.

  • Results

    CAN outperforms other state-of-the-art methods in most cases, achieving 35.9% mAP on Market-1501 under single-query evaluation and new state-of-the-art performance under multiple-query evaluation.

  • Takeaways & Limitations

    Comparative attention features are beneficial for recognition accuracy in person matching, with attention maps shifting across different person parts over time.

Abstract

from arXiv · show

Person re-identification across disjoint camera views has been widely applied in video surveillance yet it is still a challenging problem. One of the major challenges lies in the lack of spatial and temporal cues, which makes it difficult to deal with large variations of lighting conditions, viewing angles, body poses and occlusions. Recently, several deep learning based person re-identification approaches have been proposed and achieved remarkable performance. However, most of those approaches extract discriminative features from the whole frame at one glimpse without differentiating various parts of the persons to identify. It is essentially important to examine multiple highly discriminative local regions of the person images in details through multiple glimpses for dealing with the large appearance variance. In this paper, we propose a new soft attention based model, i.e., the end to-end Comparative Attention Network (CAN), specifically tailored for the task of person re-identification. The end-to-end CAN learns to selectively focus on parts of pairs of person images after taking a few glimpses of them and adaptively comparing their appearance. The CAN model is able to learn which parts of images are relevant for discerning persons and automatically integrates information from different parts to determine whether a pair of images belongs to the same person. In other words, our proposed CAN model simulates the human perception process to verify whether two images are from the same person. Extensive experiments on three benchmark person re-identification datasets, including CUHK01, CHUHK03 and Market-1501, clearly demonstrate that our proposed end-to-end CAN for person re-identification outperforms well established baselines significantly and offer new state-of-the-art performance.

I. INTRODUCTION

Person re-identification is difficult because disjoint-camera images lack spatial and temporal cues and exhibit substantial appearance variation. CAN addresses this by recurrently comparing image pairs, locating discriminative local regions, and integrating information across glimpses.

  • Image-based re-identification is challenged by pose, color, illumination, occlusion, low resolution, and missing spatial and temporal cues across cameras.
  • Existing methods often use pre-defined parts or extract local representations only once, making performance sensitive to region splits, illumination, and occlusion.
  • CAN recurrently localizes discriminative regions from raw person images without manual annotations and integrates features from multiple glimpses.
  • The comparative network uses online triplet selection and directly generates attention parts from raw person-image pairs in an end-to-end framework.
  • Experiments compare CAN with state-of-the-art methods on CUHK01, CUHK03, Market-1501, and VIPeR.

II. RELATED WORK

Prior person re-identification work combines feature extraction with cross-image comparison, using handcrafted features, metric learning, CNNs, saliency, and attention. CAN differs by generating and comparing multiple attention regions directly from raw image pairs.

  • Person re-identification commonly separates image-feature extraction from learning a metric to compare features across camera views.
  • Saliency-related methods build dense patch correspondences or combine salience and patch matching to improve person representation.
  • Metric-learning methods learn transformations that reduce distances for same-person features and increase them for different-person features.
  • CNN approaches include filter-pairing, Siamese, deeper convolutional, and parts-based architectures for learning discriminative person representations.
  • Unlike single-branch attention models, CAN compares image triplets and learns attention regions from raw images through end-to-end training.

III. MODEL ARCHITECTURE

CAN processes person-image triplets with shared CNNs and comparative attention components. Joint triplet and identification objectives train the architecture end-to-end to separate matched and mismatched persons.

  • CAN applies shared CNNs to triplets of raw images, then passes global feature maps to LSTM-based comparative attention components.
  • Training uses an anchor, same-identity positive, and different-identity negative image so matched attention features become closer than mismatched features.
  • The triplet objective introduces a margin between anchor-positive and anchor-negative feature distances to strengthen discrimination.
  • Each attention feature is also classified with a softmax identity objective whose symbols include the identity count and class-weight matrix.
  • Triplet and identification losses are jointly optimized, allowing CNN and attention components to train end-to-end from raw person images.

2) Test Phase:

In testing, CAN receives person-image pairs, computes their Euclidean distance, and ranks query-candidate pairs using the resulting distances.

  • The trained CAN computes Euclidean distances for testing image pairs and sends them to a ranking unit for final retrieval results.
  • Performance is evaluated with average CMC and accuracy at top ranks.

B. Long Short-Term Memory Networks

The CAN uses an LSTM to predict softmax attention maps recurrently, mask CNN feature maps, and retain useful information across multiple glimpses. Its attention components are trained end-to-end with the CNN.

  • Attention prediction: At each time step, an LSTM takes a masked CNN feature map and the previous hidden state to generate the next attention step.The previous hidden state predicts the location map, which masks the CNN feature map before recurrent processing.
  • LSTM operation: The LSTM gates weigh input information, remember useful content in the cell state, erase unnecessary content, and control emitted memory.The input, forget, cell, output, and hidden states coordinate the recurrent update.
  • Attention prediction: The attention map is a K × K softmax location map predicted from the previous hidden state through learnable parameters W_i,h.The predicted map is used to compute the masked feature through weighted average pooling.
  • Initialization: The CNN feature map remains the same across time steps, while learned initialization produces the initial attention location and input.The initial memory and hidden states are generated using two-layer perceptrons and CNN feature slices.
  • Comparative attention: The comparative attention component uses several LSTM time steps and concatenated hidden states as features for the triplet loss layer.Weight-sharing attention components simultaneously process positive and negative pairs during training.

C. The Comparative Attention Component

The comparative attention component combines CNN global features with recurrently generated local attention regions. It concatenates selected hidden states from repeated comparisons and normalizes them before triplet learning, while learning multiple attention maps end-to-end.

  • Component design: CNNs provide global person features, while LSTM-based comparative attention components generate local regions by comparing person images.The design is described as analogous to aspects of the human visual system.
  • Feature integration: Hidden states from selected time steps are concatenated to integrate information from repeatedly compared local regions.The concatenation lets CAN use features from different glimpses and attend to different local regions over time.
  • Normalization: The concatenated features R are passed through an ℓ2-normalization layer so triplet distances remain within the margin more often during training.This addresses wild loss fluctuations in the three-branch recurrent framework and allows more triplet constraints to contribute.
  • Novelty: Unlike prior saliency methods using pre-defined regions or separate low-level saliency processing, CAN learns multiple attention maps from raw images end-to-end.The maps highlight different local regions recurrently rather than producing only one map per image.

D. Triplet Selection

CAN selects triplets that support effective metric learning while avoiding exhaustive computation over the full training set. The paper contrasts offline checkpoint-based selection with online mini-batch selection.

  • Selection challenge: Triplet selection seeks positive and negative samples that violate the constraint in Eqn. (1), but whole-dataset argmin and argmax computation is impractical.The need to compare image pairs and generate attention locations for every image further motivates an efficient selection method.
  • Offline triplets selection: Offline selection regenerates triplets every few steps using the most recent network checkpoint to compute argmin and argmax.The checkpoint provides the representations used for periodic triplet mining.
  • Online triplets selection: Online selection performs triplet mining within a mini-batch instead of generating all feasible triplets.Generating all possible mini-batch triplets can waste resources and slow convergence when many triplets do not contribute to training.

IV. EXPERIMENTS

The experiments evaluate CAN on public person re-identification benchmarks using ranked probe-gallery matching and CMC curves. The datasets cover varied camera viewpoints, illumination, detector errors, occlusions, and training-test split protocols.

  • Evaluation setup: Experiments use CUHK01, CUHK03, Market-1501, and VIPeR as public person re-identification benchmarks.For each pedestrian, a probe image is matched against gallery images from another camera.
  • Evaluation setup: CMC curves summarize the ranks of true matches after Euclidean distances between probe and gallery samples are computed.The metric reflects how frequently the correct identity appears within each rank.
  • CUHK01: CUHK01 contains 971 persons across two camera views, with frontal or back views in one camera and side views in the other.The paper evaluates two training-test settings and averages results over 10 repeated splits.
  • CUHK03: CUHK03 contains 13,164 images of 1,360 identities captured by six cameras, including manually cropped and DPM-detected person images.Its setting includes misalignment, occlusions, missing body parts, and detector errors; results use 20 random splits.
  • VIPeR: VIPeR contains 632 persons photographed by two non-overlapping cameras with different viewpoints and illumination conditions.The dataset is split into 316-person training and test subsets, repeated 10 times.

4) Market-1501:

The experiments examine CAN design choices, including CNN features, end-to-end training, pooling alternatives, recurrent step counts, and hidden-state selection.

  • End-to-end versus non-end-to-end CAN: End-to-end training lets comparison loss update both attention components and CNN features, unlike offline pre-extracted features.Offline classification features may lack sufficient comparative information.
  • Time-step analysis: Rank1 performance improves from 5 to 8 time steps, while steps 9–14 add no significant improvement and increase computational cost.The model therefore uses 8 time steps as the best performance–cost trade-off.
  • Hidden-state selection: Concatenating hidden states from steps 2, 4, and 8 performs best because adjacent states can be redundant and all-step concatenation may over-smooth features.

2) Effect of Different CNNs and Losses:

CAN performs better with VGG-16 than AlexNet, and combining identification and triplet losses is more effective than using either loss alone.

  • CNN backbone: CAN achieves better performance with VGG-16 than with AlexNet, using the VGG-16 fifth max-pooling-layer feature cube.The passed VGG-16 feature cube has dimension 7 × 7 × 512.
  • Loss functions: Multi-task loss outperforms either triplet loss or identification loss individually for both AlexNet- and VGG-16-based CANs.The comparison indicates that identification and ranking information are both useful for learning discriminative comparative-attention features.

1) Results on CUHK01 and CUHK03:

Across CUHK01 and CUHK03, CAN generally outperforms compared methods, including saliency and deep-learning approaches, while detector-generated CUHK03 images remain more difficult.

  • CUHK01: CAN beats all compared methods at low ranks on CUHK01 with 100 test identities and also outperforms other methods with 486 test identities.It also exceeds other saliency-based methods by a large margin.
  • CUHK03 settings: CUHK03 detector-generated images perform worse than manually cropped images, reflecting misalignment, occlusions, missing body parts, and detector errors.
  • CUHK03 detected setting: With detector boxes, CAN is less affected and outperforms other approaches, including deep-learning methods, by a large margin, especially at Rank1.The authors suggest that attending to and integrating different discriminative parts may explain this robustness.
  • Market-1501: On Market-1501 single-query evaluation, CAN has 35.9% mAP and slightly trails DNS in Rank1; under multiple-query evaluation, CAN achieves new state-of-the-art performance.
  • VIPeR: On challenging VIPeR, CAN beats most compared state-of-the-art methods except SCSP, while limited training data makes the deep model prone to overfitting.The study also evaluates combining CAN with LOMO features after pretraining and fine-tuning.

F. Visualization of Attention Maps and Discussions

The visualizations show CAN shifting attention across person-image regions over time to support comparative matching, including under detector errors. Failure cases arise when severe occlusion and multiple people prevent consistent localization of corresponding parts.

  • CAN shifts attention from heads to upper bodies and then lower regions across successive time steps.The model uses different image parts for comparisons between positive and negative pairs.
  • In CUHK01 training examples, attention focuses on clothing regions that distinguish matching and nonmatching persons.For example, the model compares red clothing in a positive pair with purple clothing in negative pairs.
  • Market-1501 attention maps change similarly despite detector-induced misalignment and missing body parts.Unlike CUHK01, Market-1501 images are automatically detected and contain errors from the DPM detector.
  • CAN fails when it does not focus on corresponding parts of positive pairs, particularly in heavily occluded multi-person query images.Such cases make it difficult to compare image pairs and select the appropriate local region.
  • Across three public datasets, experiments found that the model outperformed other state-of-the-art methods in most cases.The conclusion also reports that comparative attention improved recognition accuracy in person matching.
Loading 1606.04404v2…