Source-linked AI summary
Revisiting Temporal Modeling for Video-based Person ReID
Jiyang Gao, Ram Nevatia
TL;DR
Video-based person re-ID lacks clean comparisons of temporal models because feature extractors and losses vary across methods. This paper standardizes those components, compares four temporal architectures on MARS, and proposes temporal-convolution attention, which achieves the strongest reported performance among the compared methods and outperforms state-of-the-art models by a large margin.
Problem
Different image-level feature extractors and loss functions substantially affect video-based person re-ID, making the influence of temporal modeling methods difficult to compare directly.
Method
The paper fixes ResNet-50 and triplet-plus-softmax losses, compares temporal pooling, temporal attention, RNN, and 3D convolution, and adds a temporal-convolution attention-generation network.
Results
The temporal-convolution attention model achieves the best performance among the tested temporal methods on MARS and outperforms state-of-the-art methods by a large margin.
Takeaways & Limitations
Standardizing the extractor and loss enables a direct comparison showing that temporal modeling choices differ substantially in effectiveness on MARS.
Abstract
from arXiv · showhide
Video-based person reID is an important task, which has received much attention in recent years due to the increasing demand in surveillance and camera networks. A typical video-based person reID system consists of three parts: an image-level feature extractor (e.g. CNN), a temporal modeling method to aggregate temporal features and a loss function. Although many methods on temporal modeling have been proposed, it is hard to directly compare these methods, because the choice of feature extractor and loss function also have a large impact on the final performance. We comprehensively study and compare four different temporal modeling methods (temporal pooling, temporal attention, RNN and 3D convnets) for video-based person reID. We also propose a new attention generation network which adopts temporal convolution to extract temporal information among frames. The evaluation is done on the MARS dataset, and our methods outperform state-of-the-art methods by a large margin. Our source codes are released at https://github.com/jiyanggao/Video-Person-ReID.
1 Introduction
Video-based person re-ID combines image-level extraction, temporal aggregation, and loss design, but differing choices make temporal methods difficult to compare directly. This paper standardizes the extractor and losses to compare four temporal architectures and introduces a temporal-convolution attention network evaluated on MARS.
- Motivation: Different image extractors and loss functions can substantially affect video-based person re-ID performance, confounding comparisons of temporal modeling methods.Prior methods used shallow CNNs, hand-crafted features, or VGG features, alongside differing training objectives.
- Approach: The study compares temporal pooling, temporal attention, RNNs, and 3D convolutional networks using the same ResNet-50 extractor and triplet-plus-softmax losses.The 3D network is kept at the same depth as the 2D CNN for fair comparison.
- Approach: A new attention-generation network uses temporal convolution to extract information across frames, and the evaluation is conducted on the MARS dataset.The paper reports that the proposed method outperforms state-of-the-art models by a large margin.
- Contributions: The paper comprehensively studies four temporal modeling methods on MARS and releases the source code alongside its proposed temporal-convolution attention model.The temporal-convolution model achieves the best performance among the compared temporal modeling methods.
2 Related Work
Related work covers temporal modeling for video re-ID, image-based person re-ID, and video temporal analysis. Figure 1 organizes pooling, RNN, and attention architectures around an image-level feature extractor.
- Video-based person re-ID: Prior video-based person re-ID methods primarily use RNNs or temporal attention to aggregate frame-level features.Examples include RNN representations based on averaged cell outputs or final hidden states, and quality-aware attention-weighted averaging.
- Image-based person re-ID: Image-based person re-ID research emphasizes spatial feature modeling and metric-learning loss functions.Reported spatial approaches include human-joint parsing and part-aligned representations.
- Video temporal analysis: Related video-analysis research applies temporal pooling and 3D CNNs to tasks including video classification and temporal action detection.These approaches motivate temporal feature aggregation and spatio-temporal convolutional modeling beyond person re-ID.
- Architectures: Figure 1 depicts temporal pooling, RNN, and temporal-attention architectures built on an image-level extractor, with attention generated by either spatial convolution plus fully connected layers or temporal convolution.RNN representations use a final hidden state or averaged cell outputs.
Methods
The system encodes video clips with either 3D CNNs or 2D CNNs followed by temporal aggregation, then averages clip representations into a video feature. It compares pooling, attention, RNN, and convolutional designs while training with combined triplet and softmax losses.
- Clip encoder: The encoder either directly maps a clip to a feature with a 3D CNN or extracts frame features with ResNet-50 before temporal aggregation.The 2D-CNN branch produces a T × D sequence and aggregates it into a D-dimensional clip vector.
- Temporal aggregation: Temporal pooling uses max or average aggregation, while temporal attention computes an attention-weighted average of frame features.Attention scores can be generated with spatial convolution plus a fully connected layer or with added temporal convolution across frames.
- Temporal aggregation: RNN variants encode frame sequences with recurrent hidden states and aggregate them either by the final hidden state or by averaging recurrent outputs.Both LSTM and GRU cells are evaluated.
- Loss functions: Training combines batch-hard triplet loss, which selects hardest positives and negatives within each batch, with softmax cross-entropy for identity classification.The batch contains P identities and K clips per identity, for a total of PK clips.
- Video pipeline: Videos are split into consecutive non-overlapping clips, whose clip features are averaged into a video-level representation for L2-based matching.Each clip contains T frames, and the resulting video representation is used to measure similarity between query and gallery videos.
4 Evaluation
The evaluation compares temporal modeling choices on MARS under shared experimental settings, including 3D CNNs, pooling, attention, and RNNs. Results favor short sequences and show that temporal attention and pooling outperform the image baseline, while RNN performance is lower.
- Evaluation setup: The evaluation uses MARS with mAP and CMC at ranks 1, 5, 10, and 20, alongside a ResNet-50 image baseline trained with triplet and cross-entropy losses.MARS contains 1261 identities and around 20000 tracklets with evenly split training and test sets.
- Temporal pooling: Average pooling consistently outperforms max pooling at T = 8.The pooling comparison is performed with the same sequence length.
- RNN experiments: Among RNN configurations, cell-output averaging generally performs better than using the final hidden state, Dh = 512 is best, and LSTM consistently outperforms GRU.The RNN experiments compare representation choices, hidden-state sizes, and recurrent cell types.
- Comparison with state-of-the-art methods: Temporal attention outperforms the image-based baseline by 3%, while RNN performance is lower than the image-based baseline.The comparison uses each model's selected best setting and includes temporal pooling, temporal attention, RNN, and the image baseline.
- Discussion: Mean pooling improves performance by 3% over the image baseline, whereas RNN aggregation performs worse than both alternatives.The discussion attributes the RNN result either to ineffective temporal capture or difficult training on MARS.
5 Conclusion
The paper compares four temporal modeling methods for video-based person re-ID under a fixed base network and loss setup. Its temporal-convolution attention model performs best among the evaluated temporal methods.
- The study compares temporal pooling, temporal attention, RNN and 3D convnets for video-based person re-ID.
- The temporal-convolution-based attention model achieves the best performance among all evaluated temporal modeling methods.
- The comparison fixes the base network architecture as ResNet-50 and the loss function as triplet loss plus softmax cross-entropy loss.