PaperlayerView exact revision on arXiv

Source-linked AI summary

STA: Spatial-Temporal Attention for Large-Scale Video-based Person Re-Identification

Yang Fu, Xiaoyang Wang, Yunchao Wei, Thomas Huang

arXiv:1811.04129v1cs.CV

TL;DR

Large-scale video-based person Re-ID is challenged by occlusion, pose variation, and other appearance changes that weaken frame-level pooling. STA uses parameter-free spatial-temporal attention, inter-frame regularization, and feature fusion to build clip representations from discriminative regions. Experiments on MARS and DukeMTMC-VideoReID demonstrate its effectiveness, while the approach currently assumes provided person tracklets.

  • Problem

    Large-scale video-based person Re-ID needs representations robust to occlusion, pose variation, and spatial misalignment, which frame-level pooling often cannot provide.

    Source: Introduction · p. 1

  • Method

    STA generates a 2-D spatial-temporal attention map, uses inter-frame regularization, and fuses global with discriminative features without additional parameters.

    Source: Introduction · p. 1, Proposed Method · p. 3, Inter-Frame Regularization · p. 4

  • Results

    STA demonstrates effectiveness on the MARS and DukeMTMC-VideoReID large-scale benchmarks.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The proposed module provides a more robust clip-level representation than direct frame-level aggregation for large-scale video-based person Re-ID.

    Source: Conclusion · p. 7

    The approach currently uses provided person tracklets and leaves integration with person detection and tracking algorithms for future study.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

In this work, we propose a novel Spatial-Temporal Attention (STA) approach to tackle the large-scale person re-identification task in videos. Different from the most existing methods, which simply compute representations of video clips using frame-level aggregation (e.g. average pooling), the proposed STA adopts a more effective way for producing robust clip-level feature representation. Concretely, our STA fully exploits those discriminative parts of one target person in both spatial and temporal dimensions, which results in a 2-D attention score matrix via inter-frame regularization to measure the importances of spatial parts across different frames. Thus, a more robust clip-level feature representation can be generated according to a weighted sum operation guided by the mined 2-D attention score matrix. In this way, the challenging cases for video-based person re-identification such as pose variation and partial occlusion can be well tackled by the STA. We conduct extensive experiments on two large-scale benchmarks, i.e. MARS and DukeMTMC-VideoReID. In particular, the mAP reaches 87.7% on MARS, which significantly outperforms the state-of-the-arts with a large margin of more than 11.6%.

Introduction

Video-based person Re-ID remains difficult under viewpoint, pose, illumination, occlusion, and background variation, while frame-level pooling struggles with frequent occlusions and spatial misalignment. STA addresses these issues by mining discriminative spatial parts across frames and aggregating them with a parameter-free spatial-temporal attention mechanism.

  • Large-scale video-based Re-ID performance is limited by viewpoint, pose, illumination, occlusion, and background variations.Source: Introduction · p. 1
  • Average or maximum pooling across frames often fails under frequent occlusion and pose-induced spatial misalignment.Source: Introduction · p. 1
  • STA jointly performs frame selection, discriminative-part mining, and feature aggregation without additional parameters.Source: Introduction · p. 1
  • The framework assigns spatial-region attention across frames, producing a 2-D attention score matrix for weighted feature aggregation.Source: Figure 1 · p. 2
  • Inter-frame regularization restricts dissimilarity among frames, while feature fusion combines global and discriminative information for feature aggregation.Source: Introduction · p. 2

Related Work

Related work spans image-based and video-based person Re-ID and applies attention to select informative frames or body regions. STA differs by mining spatial regions across frames without additional parameters and by combining global and discriminative information.

Image-based person re-identification

  • Image-based person Re-ID commonly uses discriminative learning or metric learning, including triplet-loss-based deep metric learning.Source: Related Work · p. 2

Video-based person re-identification

  • Video-based person Re-ID extends image-based Re-ID and has used recurrent networks or long-range motion context from frame sequences.Source: Related Work · p. 2

Attention models in person re-identification

  • Attention-based Re-ID methods estimate frame quality, select discriminative frames, or learn latent representations of different body parts.Source: Related Work · pp. 2–3
  • Unlike prior attention models, STA uses no additional parameters and combines spatial-region discovery with temporal attention for video representation.Source: Related Work · p. 3

Proposed Method

STA generates spatial-temporal attention scores for regions across frames, regularizes attention maps between frames, and fuses global and discriminative information into a Re-ID representation.

Spatial-Temporal Attention

Inter-Frame Regularization

  • Inter-frame regularization restricts differences among attention maps from frames in the same tracklet.The regularization minimizes a squared Frobenius-norm difference between randomly selected frame attention maps and is weighted by λ in the total objective.Source: Inter-Frame Regularization · p. 4

Feature Fusion Strategy

  • The feature-fusion strategy selects the highest-scoring region for each spatial block and combines it with attention-weighted feature maps.The two resulting feature maps are concatenated, followed by global average pooling and a fully connected layer to produce the representation vector.Source: Feature Fusion Strategy · p. 4, Loss Function · p. 5

Loss Function

  • STA is trained jointly with batch-hard triplet loss and softmax cross-entropy loss.This combines metric learning with discriminative learning for person identification.Source: Loss Function · pp. 4–5

Experiments

Experiments on MARS and DukeMTMC-VideoReID evaluate STA through component ablations, sequence-length and spatial-region studies, and state-of-the-art comparisons. STA improves retrieval performance, remains robust across sequence lengths, and achieves strong results on both benchmarks.

Datasets and Evaluation Protocol

  • The experiments use MARS and DukeMTMC-VideoReID, with CMC Rank-1, Rank-5, and Rank-20 scores and mAP as evaluation measures.mAP is described as suitable for both single-match and multiple-match retrieval objectives.Source: Datasets and Evaluation Protocol · p. 5

Ablation Study

  • STA improves Rank-1 and mAP over Baseline + TL by 4.0% each on MARS and by 2.7% and 2.4% on DukeMTMC-VideoReID.The model uses a 4 × 4 attention score matrix to weight spatial regions across frames.Source: Ablation Study · p. 6
  • Inter-frame regularization improves Rank-1/mAP by 1.0%/1.7% on MARS and 0.9%/1.0% on DukeMTMC-VideoReID.The reported improvement is attributed to balancing frame diversity.Source: Ablation Study · p. 6
  • Rank-1 accuracies remain above 85% and mAP above 80% on MARS for tested sequence lengths of 4, 6, and 8.The model trained with sequence length 4 is evaluated at sequence lengths 2, 4, 6, and 8.Source: Ablation Study · p. 6, Table 2 · p. 6
  • STA achieves its best result with 4 spatial regions across experiments using 2, 4, and 8 regions.The paper attributes weaker results for 2 or 8 regions to overly coarse or overly small regions.Source: Ablation Study · p. 6, Table 3 · p. 7

Comparison with the State-of-the-arts

  • 86.3/80.8 on MARS before re-ranking and 87.2/87.7 after re-ranking are reported for R1/mAP; DukeMTMC-VideoReID reaches 96.2/94.9.On MARS, the reported post-re-ranking mAP surpasses CSACSE + Optical Flow by 11.6%.Source: Comparison with the State-of-the-arts · p. 7

Conclusion

The paper presents STA for large-scale video-based person re-identification, using a 2-D spatial-temporal map and inter-frame regularization to produce robust clip-level representations. Experiments on MARS and DukeMTMC-VideoReID demonstrate its effectiveness against existing methods, while the current approach assumes provided person tracklets.

Conclusion

  • STA replaces direct frame-level aggregation with a 2-D spatial-temporal map for robust clip-level feature representation.Inter-frame regularization and feature fusion further improve the representation without additional parameters.Source: Conclusion · p. 7
  • Extensive experiments on MARS and DukeMTMC-VideoReID demonstrate STA’s effectiveness compared with existing methods.Source: Conclusion · p. 7
  • The current approach uses provided person tracklets and does not yet combine STA with person detection and tracking algorithms.The paper identifies integration with detection and tracking as a future study for real-world multi-camera systems.Source: Conclusion · p. 7

Related papers

Loading 1811.04129v1…