PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversity Regularized Spatiotemporal Attention for Video-based Person Re-identification

Shuang Li, Slawomir Bak, Peter Carr, Xiaogang Wang

arXiv:1803.09882v1cs.CV

TL;DR

Video-based person re-identification needs robust sequence representations because occlusions and pose changes can corrupt or misalign whole-frame features. The paper uses diverse spatial attention with Hellinger-based regularization and temporal aggregation to combine useful local regions across frames. Evaluations on three datasets show that the framework outperforms state-of-the-art methods under multiple metrics.

  • Problem

    Video re-identification requires a robust latent representation of each variable-duration sequence, while whole-frame temporal pooling is vulnerable to occlusions and spatial misalignment.

  • Method

    Multiple spatial attention models discover discriminative body parts or accessories, Hellinger-distance regularization encourages diversity, and temporal attention aggregates each model’s local features across frames.

  • Results

    The framework outperforms state-of-the-art methods under multiple evaluation metrics on three challenging video re-identification datasets.

  • Takeaways & Limitations

    The approach represents a person using diverse local regions and useful partial information gathered across the video rather than relying on complete, consistently aligned frames.

  • Takeaways & Limitations

    The empirical evidence indicates that training can be unstable because softmax attention values near zero drop sharply under the logarithm in KL divergence.

Abstract

from arXiv · show

Video-based person re-identification matches video clips of people across non-overlapping cameras. Most existing methods tackle this problem by encoding each video frame in its entirety and computing an aggregate representation across all frames. In practice, people are often partially occluded, which can corrupt the extracted features. Instead, we propose a new spatiotemporal attention model that automatically discovers a diverse set of distinctive body parts. This allows useful information to be extracted from all frames without succumbing to occlusions and misalignments. The network learns multiple spatial attention models and employs a diversity regularization term to ensure multiple models do not discover the same body part. Features extracted from local image regions are organized by spatial attention model and are combined using temporal attention. As a result, the network learns latent representations of the face, torso and other body parts using the best available image patches from the entire video sequence. Extensive evaluations on three datasets show that our framework outperforms the state-of-the-art approaches by large margins on multiple metrics.

1. Introduction

Video-based person re-identification must build representations from variable-length sequences despite occlusions, pose changes, and spatial misalignment. The proposed spatiotemporal attention scheme discovers diverse salient regions, aggregates them across time, and outperforms state-of-the-art methods on three datasets.

  • Video-based person re-identification matches pairs of video sequences, requiring a strong latent feature representation for each sequence.
  • Existing methods pool whole-frame features across time, but occluders can corrupt frame representations while visible body portions remain useful.Pose changes also make temporal aggregation spatially misaligned.
  • The proposed scheme uses multiple spatial attention models to localize discriminative image regions instead of encoding whole images or fixed grids.The models can discover body parts and accessories without predefined categories.
  • Spatial attention addresses alignment and occlusion, while temporal attention aggregates features extracted by each spatial model into a final video representation.
  • Hellinger-distance diversity regularization prevents multiple spatial attention models from discovering the same body part.
  • The approach outperforms state-of-the-art methods under multiple evaluation metrics on three challenging video re-identification datasets.

2. Related Work

Related work spans image- and video-based re-identification, including discriminative feature learning, metric learning, temporal models, and attention mechanisms. Prior attention approaches face redundancy or mode-collapse concerns when multiple attentions focus on similar regions.

  • Image-based person re-identification primarily studies discriminative feature extraction and robust metric learning, increasingly using deep networks to address both.
  • Video-based methods extend image-based re-identification to sequence pairs and use ranking objectives, recurrent models, temporal pooling, reliable feature selection, or spatiotemporal dynamics.
  • Attention-based re-identification methods assign importance across frames or combine global and local image content.
  • Jointly training multiple attentions can cause mode collapse, requiring careful training to prevent redundant focus on similar regions.

3. Method

The method builds video representations by discovering diverse salient regions spatially, aggregating each region across sampled frames temporally, and concatenating the resulting part features. Hellinger-distance regularization discourages redundant receptive fields so attention models focus on different regions.

  • Frame Sampling: Restricted random sampling selects one frame from each of N equal-duration video chunks to compactly represent long sequences.The ordered sampled frames retain information from across the video while avoiding redundancy between sequential frames.
  • Spatial Attention: Multiple spatial attention models use learned responses and softmax weights to localize distinctive body parts or accessories without imposing a predefined grid.Pooling over the full grid removes explicit image-location information while producing features that are roughly aligned across images for each attention model.
  • Image Features: A ResNet-50 extracts an 8×4 grid of 32 feature cells, each represented by a 2048-dimensional vector, from every sampled image.The feature extractor spans conv1 through res5c.
  • Temporal Aggregation: Temporal attention aggregates features separately for each spatial attention model, then concatenates the resulting spatiotemporal gated features into one video representation.The architecture groups spatial gated features across frames by attention model before temporal aggregation.
  • Diversity Regularization: Hellinger-distance regularization penalizes redundant receptive fields, encouraging attention models to focus on different salient regions instead of collapsing onto the same body part.The regularization term Q is added to the original OIM loss after measuring redundancy between receptive fields per image.
  • Diversity Regularization: The authors replace KL-divergence-based diversity with Hellinger regularization because small softmax attention values can make KL-based training unstable.The paper reports empirical evidence of instability when near-zero attention values are passed through the logarithm in KL divergence.

4. Experiments

The experiments evaluate the proposed framework on three video re-identification datasets and analyze its components, training choices, temporal aggregation, and number of spatial attention models.

  • Datasets and protocol: Evaluations use PRID2011, iLIDS-VID, and MARS, with rank-1 accuracy reported and mAP additionally reported for MARS.PRID2011 and iLIDS-VID use repeated random probe/gallery splits, while MARS uses its original training and testing identities.
  • Component analysis: The component analysis compares multi-region spatial attention, diversity regularization, temporal pooling, temporal attention, and dataset-specific fine-tuning.The final configuration fine-tunes the whole network independently on each video dataset.
  • Component analysis: SpaAtn improves rank-1 accuracy over the baseline by 1.5% on PRID2011, 3.7% on iLIDS-VID, and 1.1% on MARS.The comparison isolates the contribution of multiple spatial attention models relative to the ResNet-50 baseline.
  • Component analysis: Hellinger-distance regularization improves accuracy by encouraging multiple attention models to learn sufficiently large but minimally overlapping receptive fields.Temporal attention weights reflect the pertinence of each spatially attended region, such as whether a body part is fully visible and easy to detect.
  • Number of spatial attention models: Using K = 6 spatial attention models achieves maximum performance, although performance generally drops when K increases from 1 to 2.Increasing K enables discovery of more body parts while regularization makes receptive fields shrink as K grows.

K PRID2011 iLIDS-VID MARS

The experiments examine how the number and diversity of spatial attention models affect performance and compare the resulting framework with state-of-the-art methods across three datasets.

  • K PRID2011 iLIDS-VID MARS: K = 6 achieves maximum performance, whereas increasing K from 1 to 2 causes a general performance drop.The results suggest that treating a person as one region can outperform splitting the representation into two body parts, while sufficiently large K is beneficial.
  • Spatial attention visualization: The learned receptive fields focus primarily on foreground body parts with minimal overlap, unlike the compared method’s fields that include background clutter and substantially overlap.Receptive-field sizes vary according to the discovered concept.
  • State-of-the-art comparison: The proposed method attains the highest performance on PRID2011, iLIDS-VID, and MARS in the state-of-the-art comparison.The comparison reports rank-1 accuracy for all datasets and mAP in brackets for MARS.
  • State-of-the-art comparison: 11.7% is the maximum improvement over the state-of-the-art, achieved on MARS.MARS is described as the most challenging dataset because it contains distractor sequences and a substantially larger gallery set.

5. Summary

The paper represents videos with diverse spatial attention models that align salient local patches and identify occluded regions, then aggregates each spatial component across time. A Hellinger-distance diversity regularizer prevents redundant attention models, and evaluations on three datasets report large-margin improvements over state of the art.

  • A diverse set of spatial attention models extracts consistent local patches across frames despite pose, orientation, and occlusion changes.This addresses alignment and visibility challenges that make single per-frame feature vectors unreliable.
  • Hellinger-distance diversity regularization minimizes overlap between receptive fields so attention models discover nonredundant regions.The paper contrasts this image-focused regularizer with a text-embedding diversity term that experiments find less effective for images.
  • Temporal attention aggregates features separately for each spatial component, allowing each discovered body part to use its most pertinent regions across the video.For example, facial-region features are combined with one another rather than pooled indiscriminately across components.
  • Evaluations on three datasets and component analyses show large-margin improvements over state-of-the-art approaches.The reported result is presented as evidence of the framework’s effectiveness for video-based person re-identification.

Supplementary Material Diversity Regularized Spatiotemporal Attention

The supplementary material describes temporal feature enhancement using appearance and temporal similarities, followed by row-wise softmax weighting and residual feature updates. Visualizations show attention favoring correctly detected content over occluded or background regions, while spatial attention identifies salient foreground parts.

  • For each spatial component, features from all video frames are organized into a matrix for temporal pooling.The kth component collects x1,k through xN,k in Xk ∈ R^D×N.
  • The feature-similarity matrix Φk uses feature-vector inner products, while the temporal-similarity matrix Ψ encodes relative frame distance and positional information.The overall similarity combines appearance and location information before temporal weighting.
  • Ck is obtained by applying row-wise Softmax to Φk + Ψ, with each row describing contribution probabilities from all frames to one frame.The resulting weights support enhanced feature representations through a residual connection.
  • Temporal attention assigns low weights to occluded or background parts and high weights to correctly detected parts.The visualization is presented as confirmation that the temporal attention model learns frame importance from salient regions.
  • With K = 6, spatial attention models primarily focus on foreground regions and generally correspond to specific body parts.The receptive-field examples illustrate distinctive image regions considered useful for re-identification.
Loading 1803.09882v1…