Source-linked AI summary
Jointly Attentive Spatial-Temporal Pooling Networks for Video-based Person Re-Identification
Shuangjie Xu, Yu Cheng, Kang Gu, Yang Yang, Shiyu Chang, Pan Zhou
TL;DR
Video-based person re-identification must match people despite substantial appearance variation, while prior methods often learn sequence representations independently of their matching counterparts. ASTPN uses similarity-guided joint spatial-temporal attention within a Siamese recurrent-convolutional architecture, and experiments on iLIDS-VID, PRID-2011, and MARS show improved performance over established methods and pooling baselines.
Problem
Video person re-identification is difficult under viewpoint, lighting, pose, and occlusion variation, and prior approaches rarely account for mutual influence between matching sequences.
Method
ASTPN learns similarity between paired video features and uses it to pool informative spatial regions and temporal frames in a Siamese recurrent-convolutional network.
Results
ASTPN outperforms established methods on iLIDS-VID, PRID-2011, and MARS, with joint pooling stronger than spatial or temporal pooling alone.
Takeaways & Limitations
Joint attentive pooling provides a simple, low-overhead design choice for deep RNN-CNN person re-identification systems.
Abstract
from arXiv · showhide
Person Re-Identification (person re-id) is a crucial task as its applications in visual surveillance and human-computer interaction. In this work, we present a novel joint Spatial and Temporal Attention Pooling Network (ASTPN) for video-based person re-identification, which enables the feature extractor to be aware of the current input video sequences, in a way that interdependency from the matching items can directly influence the computation of each other's representation. Specifically, the spatial pooling layer is able to select regions from each frame, while the attention temporal pooling performed can select informative frames over the sequence, both pooling guided by the information from distance matching. Experiments are conduced on the iLIDS-VID, PRID-2011 and MARS datasets and the results demonstrate that this approach outperforms existing state-of-art methods. We also analyze how the joint pooling in both dimensions can boost the person re-id performance more effectively than using either of them separately.
1. Introduction
Video-based person re-identification addresses challenging cross-camera matching by using temporal sequences, while ASTPN makes each sequence representation responsive to its matching counterpart. It jointly attends to informative regions and frames, achieving strong results across three datasets.
- Motivation: Video-based re-identification uses temporal information and multiple appearance samples to improve person representations beyond still-image matching.Prior methods extract sequence-level representations with temporal deep networks or recurrent feature aggregation.
- Motivation: Most existing approaches derive each sequence representation separately, neglecting the matching counterpart’s mutual influence during representation learning.The paper motivates pair-aware processing by noting that different comparison sequences can change which frames and regions matter.
- Proposed approach: ASTPN learns similarity between two input sequences and uses it to compute spatial and temporal attention vectors for pooling.Spatial attention selects regions within frames, while temporal attention selects informative frames over sequences.
- Proposed approach: The architecture combines recurrent-convolutional feature extraction, jointly attentive pooling, and a Siamese network trained end to end.The Siamese design allows the two inputs’ interdependence to influence representation learning and similarity measurement.
- Results: Experiments on iLIDS-VID, PRID-2011, and MARS show that ASTPN significantly outperforms established baselines and achieves new state-of-the-art performance.The cross-dataset test reaches the same overall conclusion.
2. Related Work
Related work progressed from handcrafted image-level features and metric learning toward video-based deep models. The paper identifies a remaining gap: most methods do not let matching inputs mutually influence representation learning, motivating joint attention.
- Image-based re-identification: Traditional person re-identification methods primarily learn image representations or robust distance metrics, despite challenges from viewpoint, illumination, occlusion, and similar appearance.Video offers motion-related temporal information and more appearance samples, but increases computation.
- Video-based re-identification: Video-based methods seek to exploit temporal information through video fragments, frame aggregation, and sequence-level representations.Prior approaches include discriminative fragment selection, bag-of-words encoding, and recurrent feature aggregation.
- Deep learning methods: Deep learning combines feature extraction and metric learning through CNN, RNN, and Siamese architectures for person re-identification.These models learn discriminative representations from image or frame-wise features while making final matching predictions.
- Attention-based matching: Most prior frameworks ignore the mutual influence of compared inputs during representation learning, leaving a gap addressed through attention mechanisms.The paper draws on comparative and two-way attention ideas for spatial and temporal pooling.
3. The Proposed Model Architecture
ASTPN uses a Siamese recurrent-convolutional architecture with jointly attentive spatial-temporal pooling to form sequence representations for video-based person re-identification. Spatial pooling emphasizes regions within frames, while temporal pooling emphasizes informative time steps using information shared between matched sequences.
- Overall architecture: ASTPN passes paired video sequences through Siamese convolutional and recurrent networks to produce sequence-level representations and their Euclidean distance.The network comprises convolutional layers, attentive spatial pooling, a recurrent layer, and attentive temporal pooling.
- Spatial pooling: Spatial pyramid pooling extracts multi-scale regions from each frame using 8×8, 4×4, 2×2, and 1×1 spatial bins, then concatenates the pooling outputs.The convolutional feature map from each frame is fed into the spatial pooling layer to obtain an image-level representation.
- Joint attention: Jointly attentive pooling replaces generic spatial and temporal pooling so the matching distance can guide attention toward relevant regions and effective time steps.The mechanism is intended to support image sequences with arbitrary resolution or length.
- Recurrent layer: The resulting image-level representations are forwarded through the recurrent network to capture information between time steps.The recurrent layer uses hidden states propagated across time and a tanh activation.
- Temporal pooling: Attentive temporal pooling computes pair-dependent frame weights from probe and gallery recurrent outputs, applies softmax, and combines features into sequence representations.Column-wise and row-wise max pooling produce temporal weight vectors, while dot products with the attention vectors yield the final representations.
- Inputs and training: The network uses three color channels and two optical-flow channels, combining appearance information with temporal motion information.The Siamese network shares convolutional-layer parameters across the processed inputs.
4. Experimental Results
Experiments evaluate ASTPN and its pooling variants on iLIDS-VID, PRID-2011, and MARS, including cross-dataset testing and comparisons with established methods. Results show strong performance, with joint spatial-temporal pooling especially beneficial on challenging data.
- Experiment Settings: Experiments evaluate video-based person re-identification on iLIDS-VID, PRID-2011, and MARS using CMC-based comparisons.The iLIDS-VID and PRID-2011 sequences are split into equal training and testing subsets, with average CMC curves reported over 10 trials.
- Results: On iLIDS-VID, ASTPN achieves 62% rank-1 accuracy, exceeding RNN-CNN by more than 4%.The attentive mechanism outperforms the compared networks by a large margin, while temporal pooling remains effective without attentive spatial pooling.
- Results: On PRID-2011, ASTPN reaches 77% rank-1 accuracy, 95% at rank 5, and 99% at rank 10, exceeding RNN-CNN by 7% at rank 1.The authors report overall higher matching rates on this less challenging dataset.
- MARS: On MARS, ASTPN achieves the best accuracy, with improvements of around 4% across ranks despite an overall performance drop relative to the other datasets.MARS contains substantially more identities and tracklets, with automatically generated tracklets introducing realistic detection and tracking errors.
- Control Experiments with Different Pooling Strategies: ATPN improves over RNN-CNN by almost 10% at rank 2 on PRID-2011 and by 5% at rank 3 on iLIDS-VID.The results support attentive temporal pooling as a stronger sequence-level aggregation strategy than simple max-pooling or mean-pooling.
- Control Experiments with Different Pooling Strategies: ASPN matches ATPN on iLIDS-VID but is less competitive on PRID-2011, whereas ASTPN exceeds ATPN by about 5% at rank 3 on MARS.The authors associate spatial attention with contextual information and report stronger robustness from combining spatial and temporal pooling on challenging MARS.
- Cross-Dataset Testing: In cross-dataset testing, ASTPN achieves 30% rank-1 accuracy on PRID-2011, close to SRID’s 35% trained on PRID-2011.The cross-dataset setup trains on iLIDS-VID and tests on 50% of PRID-2011; video-based re-identification improves rank-1 scores over single-shot re-identification by 100% for both models.
5. Conclusion
ASTPN jointly pools spatial and temporal information to learn video representations and similarity measurements for person re-identification. Experiments show it outperforms standard pooling approaches, with joint pooling outperforming either spatial or temporal pooling alone.
- ASTPN jointly learns input representations and their similarity measurement for video-based person re-identification.
- It decomposes pooling into spatial pooling over CNN feature maps and attentive temporal pooling over RNN outputs.These stages select key regions and frames for feature representation learning.
- Extensive experiments on iLIDS-VID, PRID-2011 and MARS show ASTPN significantly outperforms standard max and temporal pooling approaches.
- Control experiments show joint spatial-temporal pooling is more effective than using spatial or temporal pooling separately.
- ASTPN introduces little computational overhead compared with general max pooling, supporting its use in deep RNN-CNN person re-identification systems.