PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Complementary Learning for Video Person Re-Identification

Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, Xilin Chen

arXiv:2007.09357v1cs.CV

TL;DR

Video person re-identification methods often produce redundant frame features that focus on similar local parts. TCLNet uses temporal saliency erasing and boosting to learn complementary features across frames, and experiments report favorable performance against state-of-the-art methods.

  • Problem

    Existing video re-identification methods often produce redundant features by repeatedly focusing on the same local salient parts across similar frames.

  • Method

    TCLNet uses Temporal Saliency Erasing to recursively discover complementary parts across frames and Temporal Saliency Boosting to propagate salient information among frames.

  • Results

    TCLNet outperforms state-of-the-art methods across three challenging video re-identification benchmarks and multiple evaluation metrics.

  • Takeaways & Limitations

    Complementary temporal feature learning can produce diverse frame features that cover the target body and improve distinction among identities with similar local appearances.

Abstract

from arXiv · show

This paper proposes a Temporal Complementary Learning Network that extracts complementary features of consecutive video frames for video person re-identification. Firstly, we introduce a Temporal Saliency Erasing (TSE) module including a saliency erasing operation and a series of ordered learners. Specifically, for a specific frame of a video, the saliency erasing operation drives the specific learner to mine new and complementary parts by erasing the parts activated by previous frames. Such that the diverse visual features can be discovered for consecutive frames and finally form an integral characteristic of the target identity. Furthermore, a Temporal Saliency Boosting (TSB) module is designed to propagate the salient information among video frames to enhance the salient feature. It is complementary to TSE by effectively alleviating the information loss caused by the erasing operation of TSE. Extensive experiments show our method performs favorably against state-of-the-arts. The source code is available at https://github.com/blue-blue272/VideoReID-TCLNet.

1 Introduction

Video person re-identification benefits from spatial-temporal cues, but frame-wise methods often produce redundant features focused on similar local regions. TCLNet addresses this with temporal complementary learning, saliency erasing, and saliency boosting.

  • 1 Introduction: Existing methods often attend to the same salient local parts across highly similar frames, making similarly appearing pedestrians difficult to distinguish.Upper clothing is given as an example of a salient but insufficiently discriminative region.
  • 1 Introduction: TCLNet extracts complementary features from consecutive video frames to form an integral characteristic of the target identity.The method is designed to exploit spatial-temporal information that existing frame-wise operations leave underused.
  • 1 Introduction: TSE recursively erases parts discovered in previous frames so ordered learners can mine diverse, complementary parts across consecutive frames.The resulting features can focus on diverse regions covering the target body.
  • 1 Introduction: TSB propagates salient information among video frames to strengthen salient features weakened by TSE's recursive erasing.This complements TSE by improving the representation of the most salient parts.
  • 1 Introduction: Across three challenging video re-identification benchmarks, TCLNet outperforms state-of-the-art methods under multiple evaluation metrics.The proposed modules can be inserted into deep CNNs for video-frame feature extraction.

2 Related Work

Prior video re-identification methods either aggregate independently extracted frame features or model temporal sequences, yet commonly produce redundant representations. TCLNet instead uses temporally guided complementary feature extraction rather than conventional random erasing.

  • 2 Related Work: Image-set methods independently extract frame features and aggregate them with pooling or temporal attention, but ignore temporal cues.These methods are described as efficient while treating video frames as disordered images.
  • 2 Related Work: Temporal-sequence methods exploit temporal cues through optical flow, recurrent architectures, or other sequence modeling approaches.The passage identifies these as a second major category of video re-identification methods.
  • 2 Related Work: Existing methods perform the same operation on each frame, producing highly redundant features and motivating temporal complementary learning.The proposed mechanism seeks an integral identity characteristic for better re-identification.
  • 2 Related Work: Unlike conventional erasing methods that randomly drop pixels or activations for regularization, TCLNet erases guided regions to extract complementary video-frame features during training and testing.Its mechanism is guided by activated regions rather than random masking.
  • 2 Related Work: Saliency-guided image-erasing methods operate on single images, whereas TCLNet performs feature erasing across video sequences to extract complementary features for consecutive frames.The distinction concerns both purpose and implementation.

3 Temporal Complementary Learning Network

TCLNet combines Temporal Saliency Erasing to mine complementary frame features with Temporal Saliency Boosting to restore salient information lost during erasing.

  • Temporal Saliency Erasing Module: TSE iteratively erases parts discovered in previous frames and uses ordered learners to discover new discriminative parts, producing complementary features across consecutive frames.The resulting frame features can be combined into an integral characteristic of the target person.
  • Temporal Saliency Erasing Module: TSE uses correlation maps and block binarization to identify and erase contiguous regions activated by previous-frame features.The correlation layer localizes relevant regions through semantic similarity, while block binarization avoids discontinuous erasing.
  • Temporal Saliency Boosting Module: TSB addresses TSE-induced information loss by propagating the most salient information among intermediate frame-level feature maps.This strengthens salient features by allowing them to capture visual cues across all video frames.
  • Temporal Saliency Boosting Module: TSB propagates salient information from memory frames to each query frame through query-memory attention and residual learning.Similarity-weighted memory descriptors are normalized, aggregated, and added back to the query after batch normalization.
  • Overall Architecture: TCLNet integrates TSE and TSB into a CNN-based architecture, with temporal average pooling shown in its overall design.The architecture uses a ResNet-50 backbone, TSE learners, and a final representation formed from learner features.

4 Experiments

Experiments on three video re-identification benchmarks show that TCLNet improves performance through complementary feature extraction and salient-feature propagation, with low computational overhead.

  • Comparison with State-of-the-art Methods: TCLNet outperforms the best existing methods on MARS, DukeMTMC-VideoReID, and iLIDS-VID under multiple evaluation metrics.The comparison includes image-set and temporal-sequence methods across all three benchmarks.
  • Ablation Study: 2.9% mAP and 1.4% top-1 accuracy gains show that TSE substantially improves the baseline with negligible computational overhead.The learners collaboratively mine complementary parts to form a more integral identity representation.
  • Ablation Study: TSE mainly gains from saliency erasing rather than extra parameters, because removing the erasing operation yields only a small improvement over the baseline.Without saliency erasing, different ordered learners capture nearly the same visual features.
  • Ablation Study: Performance increases with more ordered learners but drops at four learners, so the method uses N=2 to avoid erasing discriminative regions and activating background.The erased height performs best at he = 3; overly small or large erased regions are less effective.
  • Ablation Study: 2.7% mAP and 0.8% top-1 accuracy gains show that TSB strengthens salient features, while combining TSE and TSB adds about 1% on both metrics.TSB propagates salient information among video frames to compensate for information loss from erasing.
  • Ablation Study: TCLNet requires 16.259 GFLOPs, only a 0.08% increase over the original model, while adding 6.4M parameters.The added computation mainly comes from matrix-multiplication-based correlation and probability maps.

5 Conclusions

The paper concludes that TCLNet improves video person re-identification by learning complementary frame features and strengthening salient information across video sequences.

  • TCLNet combines Temporal Saliency Erasing and Temporal Saliency Boosting for video person re-identification.TSE discovers complementary cues, while TSB propagates salient information among frames.
  • TSE progressively discovers diverse and complementary visual cues across consecutive video frames.The module uses saliency erasing to support complementary feature learning.
  • TSB strengthens representation by propagating salient information among video frames, complementing TSE's erasing operation.The propagation alleviates information loss caused by saliency erasing.
  • Extensive experiments demonstrate that TCLNet outperforms current state-of-the-art methods.
Loading 2007.09357v1…