Source-linked AI summary
Appearance-Preserving 3D Convolution for Video-based Person Re-identification
Xinqian Gu, Hong Chang, Bingpeng Ma, Hongkai Zhang, Xilin Chen
TL;DR
Video-based ReID suffers from temporal appearance misalignment caused by imperfect detection and posture changes, which can damage appearance representations during 3D convolution. AP3D aligns adjacent feature maps with an Appearance-Preserving Module before convolution, and experiments show state-of-the-art performance across three widely used datasets.
Problem
Imperfect detection and posture changes create temporal appearance misalignment, so existing 3D convolution can destroy appearance representations important for video-based ReID.
Method
AP3D combines an Appearance-Preserving Module that aligns adjacent feature maps by semantic similarity with a subsequent 3D convolution kernel.
Results
AP3D surpasses state-of-the-art video-based ReID methods on three widely used datasets, including 85.1% mAP on MARS.
Takeaways & Limitations
AP3D preserves appearance representation while modeling temporal information and can be integrated into existing 3D ConvNets by replacing their convolution kernels.
Abstract
from arXiv · showhide
Due to the imperfect person detection results and posture changes, temporal appearance misalignment is unavoidable in video-based person re-identification (ReID). In this case, 3D convolution may destroy the appearance representation of person video clips, thus it is harmful to ReID. To address this problem, we propose AppearancePreserving 3D Convolution (AP3D), which is composed of two components: an Appearance-Preserving Module (APM) and a 3D convolution kernel. With APM aligning the adjacent feature maps in pixel level, the following 3D convolution can model temporal information on the premise of maintaining the appearance representation quality. It is easy to combine AP3D with existing 3D ConvNets by simply replacing the original 3D convolution kernels with AP3Ds. Extensive experiments demonstrate the effectiveness of AP3D for video-based ReID and the results on three widely used datasets surpass the state-of-the-arts. Code is available at: https://github.com/guxinqian/AP3D.
1 Introduction
Video-based ReID must model temporal relations while preserving appearance, but detector errors and posture changes misalign adjacent bounding-box features. AP3D addresses this by aligning feature maps before 3D convolution and achieves strong benchmark performance.
- Video-based ReID requires effective modeling of temporal relations between video frames.
- 3D convolution can harm ReID when detection errors and posture changes cause adjacent frames to represent different body parts or scales.Bounding-box resizing can misalign spatial positions and body-part sizes across frames.
- AP3D combines an Appearance-Preserving Module with 3D convolution to reconstruct adjacent feature maps using cross-pixel semantic similarity.The reconstruction acts as feature-map registration and preserves temporal appearance alignment before convolution.
- AP3D can replace original 3D kernels in existing 3D ConvNets and requires no extra correspondence annotations beyond identification supervision.
- AP3D achieves state-of-the-art results on two datasets using RGB information only and surpasses state-of-the-art video-based ReID methods.
2 Related Work
Related work addresses video-based ReID through temporal modeling, multi-frame integration, and image registration. AP3D connects these concerns by treating feature-map alignment across sequential frames as registration.
- Video-based ReID methods either model additional temporal information or extract frame features with image-based ReID models before integrating or matching them.
- AP3D uses an Appearance-Preserving Module to reconstruct adjacent feature maps relative to central maps before 3D convolution.
- Feature-map registration aligns sequential feature maps so corresponding spatial positions represent the same body part.The paper frames this as an image-registration problem across feature maps obtained at sequential times.
3 Appearance-Preserving 3D Convolution
AP3D addresses appearance destruction caused by applying 3D convolution to temporally misaligned person feature maps. Its APM registers adjacent maps using cross-pixel semantic similarity, suppresses unmatched regions with Contrastive Attention, and then applies 3D convolution while preserving spatiotemporal relations.
- Existing 3D convolution may destroy person appearance representations because imperfect detections and posture changes misalign corresponding spatial positions across frames.
- AP3D combines an Appearance-Preserving Module with a following 3D convolution to align adjacent feature maps before temporal modeling.
- Feature Map Registration: APM performs feature-map registration by locating corresponding adjacent-frame regions through cross-pixel cosine similarity rather than fixed spatial positions.
- Contrastive Attention: Contrastive Attention identifies unmatched regions between reconstructed and central maps and masks the reconstructed features to limit error propagation from imperfect registration.
- Discussion: APM preserves spatiotemporal relative relations for subsequent 3D convolution, unlike Non-local operation, while using O(N) rather than O(N^2) complexity.
- Integration with 3D ConvNets: AP3D versions of I3D and P3D residual blocks require replacing only the original temporal convolution kernels with AP3Ds of the same size.
4 AP3D for Video-based ReID
The video-based ReID evaluation uses a 2D ConvNet baseline and extends it with AP3D residual blocks for spatiotemporal representation learning. Training combines cross-entropy and cosine-distance triplet losses.
- The experiments use a 2D ConvNet, called C2D, as the baseline and extend it into an AP3D ConvNet by replacing selected 2D residual blocks.
- C2D baseline: C2D uses an ImageNet-pretrained ResNet-50 with stage-5 downsampling removed to preserve feature granularity.
- C2D baseline: For an input clip with T frames, C2D produces T × H × W × 2048 features before spatial max pooling and temporal average pooling yield a 2048-dimensional representation.
- Training combines cross-entropy loss with triplet loss and uses cosine distance for the triplet objective to maintain consistency with angular-space optimization.
5 Experiments
Experiments evaluate AP3D across datasets, temporal-modeling comparisons, block placements, backbones, attention variants, and visualizations. AP3D consistently improves video-based ReID performance, while its gains depend on placement and are supported by feature-alignment evidence.
- Experimental setup: Experiments use MARS, DukeMTMC-VideoReID, and iLIDS-VID, with CMC and mAP as evaluation metrics.Ablations mainly use MARS and DukeMTMC-VideoReID; final state-of-the-art comparisons also include iLIDS-VID.
- AP3D versus original 3D convolution: AP3D improves over original I3D and P3D blocks, which show close or lower results than the C2D baseline because of appearance destruction.AP3D aligns appearance representations before temporal convolution.
- Comparison with related approaches: AP3D outperforms Non-local operation with fewer parameters and lower computational complexity, while NL-AP3D provides further improvement.Contrastive Attention embedded in Non-local does not match AP3D, whereas AP3D and Non-local are complementary.
- Ablation study: Placing AP3D in stage2 or stage3 performs similarly and surpasses five P3D blocks, whereas placement in stage1 or stage4 performs worse than C2D.Increasing AP3D blocks generally improves performance, but ten blocks can reduce performance through overfitting.
- Backbone generalization: AP3D improves ResNet-18 and ResNet-34 consistently on both datasets, and AP3D-ResNet-18 surpasses deeper ResNet-34 on MARS.The comparison indicates that AP3D effectiveness does not rely on additional parameters or computational load.
- Attention and visualization: Contrastive Attention further improves AP-I3D and AP-P3D on MARS, while varying the scale factor consistently beats the baseline and performs best at s = 4.Visualizations show reconstructed feature maps becoming aligned with the central-frame foreground after APM.
- State-of-the-art comparison: Using RGB only and temporal average pooling, AP3D surpasses the compared methods on MARS, DukeMTMC-VideoReID, and iLIDS-VID, achieving 85.1% mAP on MARS.Combining AP3D with Non-local yields further improvement.
6 Conclusion
The paper concludes that AP3D preserves appearance alignment before 3D convolution, enabling temporal modeling without degrading appearance representation. It is easy to integrate into existing 3D ConvNets and surpasses state-of-the-art methods on three datasets.
- Conclusion: AP3D combines an Appearance-Preserving Module with a 3D convolution kernel to align adjacent feature maps before temporal modeling.This addresses the appearance destruction problem of original 3D convolution.
- Conclusion: AP3D is easy to combine with existing 3D ConvNets and demonstrates effectiveness and generalization across three widely used datasets.The paper identifies extending AP3D as a basic operation for other video recognition tasks as future work.