Source-linked AI summary
Multi-scale 3D Convolution Network for Video Based Person Re-Identification
Jianing Li, Shiliang Zhang, Tiejun Huang
TL;DR
Video person ReID needs both spatial appearance and temporal cues, while conventional 3D CNNs can be costly and difficult to optimize. The paper combines a compact M3D temporal network with residual attention and a 2D spatial stream, improving benchmark performance with modest parameter overhead.
Problem
Video ReID requires spatial and temporal representations, but conventional 3D CNNs use many parameters and are difficult to optimize.
Method
The method fuses 2D spatial features with M3D multi-scale temporal features and residual spatial-temporal attention masks.
Results
The method outperforms compared approaches across three video ReID benchmarks; on MARS, about 4MB of added parameters raise mAP from 0.625 to 0.699.
Takeaways & Limitations
M3D provides a compact temporal feature-learning alternative to traditional 3D CNNs while preserving complementary spatial information from a 2D stream.
Abstract
from arXiv · showhide
This paper proposes a two-stream convolution network to extract spatial and temporal cues for video based person Re-Identification (ReID). A temporal stream in this network is constructed by inserting several Multi-scale 3D (M3D) convolution layers into a 2D CNN network. The resulting M3D convolution network introduces a fraction of parameters into the 2D CNN, but gains the ability of multi-scale temporal feature learning. With this compact architecture, M3D convolution network is also more efficient and easier to optimize than existing 3D convolution networks. The temporal stream further involves Residual Attention Layers (RAL) to refine the temporal features. By jointly learning spatial-temporal attention masks in a residual manner, RAL identifies the discriminative spatial regions and temporal cues. The other stream in our network is implemented with a 2D CNN for spatial feature extraction. The spatial and temporal features from two streams are finally fused for the video based person ReID. Evaluations on three widely used benchmarks datasets, i.e., MARS, PRID2011, and iLIDS-VID demonstrate the substantial advantages of our method over existing 3D convolution networks and state-of-art methods.
Introduction
Video person ReID benefits from combining spatial appearance with temporal cues, but existing frame-wise and deep 3D approaches face limitations in temporal modeling, model size, and optimization. The paper proposes a compact M3D-based temporal stream with residual attention, paired with a 2D spatial stream, and reports improved benchmark performance.
- Motivation: Spatial and temporal cues are important for identifying people whose appearance or gait alone may be ambiguous.The motivating examples contrast similar clothing with different gaits and similar gaits with different appearance.
- Existing approaches: Existing video ReID methods commonly process frames independently before pooling features or applying recurrent networks.These approaches extract frame-level features and then generate video representations through pooling, weighting, or RNNs.
- 3D CNN limitations: 3D CNNs learn spatial-temporal representations across consecutive frames but require stacked kernels to capture longer temporal cues.A single 3D kernel covers only short temporal context, motivating deeper 3D architectures.
- Proposed method: The proposed M3D CNN inserts parallel temporal kernels with different ranges into a 2D CNN, adding multi-scale temporal modeling with few parameters.Residual attention layers further learn spatial-temporal masks to retain discriminative cues and suppress noise.
- Results: Adding about 4MB of parameters raises the 2D CNN’s MARS mAP from 0.625 to 0.699, while I3D reaches 0.628 with 186MB.The reported comparison presents M3D as more accurate and substantially smaller than I3D.
- Proposed method: The two-stream model combines M3D temporal features with 2D spatial and appearance features for video person ReID.The contribution description identifies M3D, residual attention, and complementary 2D features as the main components.
Related work
Related work spans image-based and video-based person ReID, including handcrafted spatiotemporal descriptors, frame aggregation, two-stream networks, and 3D CNNs. The paper distinguishes its approach by directly learning temporal features from video with a compact M3D network rather than relying on stacked optical flow or large traditional 3D CNNs.
- Image-based ReID: Person ReID research is broadly divided into image-based and video-based approaches, with image methods learning features or distance metrics.Recent image-based work has made substantial progress in discriminative representation learning and feature matching.
- Video-based ReID: Video ReID methods include handcrafted spatiotemporal descriptors and frame-feature accumulation, but handcrafted methods show limited robustness versus deep features.Other approaches treat video as a collection of still-frame features and aggregate them into video representations.
- Video modeling: Video tasks commonly use either two-stream networks for separate spatial and temporal features or 3D CNNs for joint spatiotemporal learning.Two-stream methods often use still images and stacked optical flow as inputs to their respective streams.
- Paper distinction: The proposed method directly extracts temporal features from video and fuses them with spatial features, avoiding optical-flow misalignment and using a more compact architecture than traditional 3D CNNs.The paper identifies these differences as its contribution to video person ReID.
Two-stream M3D Convolution Network
The two-stream network combines a 2D CNN for spatial features with an M3D CNN for multi-scale temporal features, then fuses both representations. M3D layers and residual attention refine temporal modeling while retaining a compact, 2D-CNN-initializable architecture.
- Two-stream architecture: Video representation combines complementary spatial and temporal features extracted by separate modules and concatenated for person ReID.The spatial feature is produced by a 2D CNN, while the temporal feature is learned directly from video sequences by the M3D network.
- Spatial stream: 2D CNN frame features are aggregated through average pooling to form the sequence-level spatial representation.F2d denotes the 2D CNN used for frame-feature extraction.
- Design rationale: The architecture addresses 3D CNN limitations by modeling longer temporal cues with a compact network that is easier to initialize and optimize than deeply stacked 3D convolutions.Deep 3D CNNs require many parameters and cannot directly leverage 2D ImageNet pretraining, whereas M3D layers are built within a 2D CNN framework.
- Temporal stream: M3D layers use parallel temporal kernels with different dilation ranges alongside spatial convolution to capture multi-scale temporal cues.The number of temporal kernels controls the temporal receptive field; with n = 3, the receptive field can cover 7 time dimensions.
- Temporal stream: M3D layers can replace 2D CNN layers through residual connections, enabling initialization from pretrained 2D CNN weights by setting temporal-kernel weights to 0.This design supports a larger temporal receptive field while preserving compatibility with well-trained 2D CNN layers.
- Residual attention: Residual Attention Layers learn spatial, temporal, and channel attention masks, preserve the input, and emphasize meaningful cues through residual convolution.The attention mask is factorized into low-dimensional spatial, channel, and temporal masks to reduce parameter requirements.
Experiment
Experiments compare M3D-based temporal modeling, residual attention, and two-stream fusion against 2D and 3D CNN alternatives across three video ReID datasets. The results show that M3D improves accuracy with low parameter overhead, RAL further improves performance, and fusion performs best overall.
- Experimental setup: The study evaluates PRID-2011, iLIDS-VID, and MARS using a ResNet50-based 2D baseline, 3D CNN variants, M3D CNN, and attention-enhanced two-stream models.PRID-2011 and iLIDS-VID use repeated random train/test identity splits, while MARS provides fixed training and testing sets.
- M3D convolution: M3D CNN outperforms the 2D CNN and other tested 3D CNNs, exceeding 2D CNN and P3D-C by about 7.4% and 2.9% in mAP, respectively.Its four M3D layers add only 4.2MB over the 2D CNN, compared with 90.6MB for I3D and 15.2MB for P3D.
- M3D convolution: 8-frame clips outperform 16-frame clips for the tested 3D CNNs, which the authors associate with lower memory use and larger training batches.The authors therefore use 8-frame clips in subsequent experiments.
- Residual attention: RAL improves M3D performance, with the complete attention layer raising PRID rank-1 accuracy from 87.64% to 91.03%.RAL includes spatial, temporal, and channel attention branches.
- Spatial-temporal fusion: Combining 2D CNN and M3D CNN features achieves the best performance in the component comparison by exploiting complementary spatial and temporal information.The two-stream evaluation includes M3D, RAL, and spatial-temporal feature fusion across three datasets.
- Comparison with recent work: On MARS, the method reaches 84.39% rank-1 accuracy and 74.06% mAP, exceeding DuATM and DRSA by 6.33% and 8.26% in mAP, respectively.The method uses global features and simple Euclidean-distance matching, unlike the compared methods’ more complex matching or local-part strategies.
- Comparison with recent work: On PRID and iLIDS-VID, the method has competitive rank-1 accuracy, although DRSA outperforms it on iLIDS-VID.The authors attribute this difference partly to iLIDS-VID’s small training set and DRSA’s multi-task and OIM-based training.
Conclusion
The proposed two-stream network combines M3D-based temporal representations, refined by residual attention, with 2D CNN spatial representations for video person ReID. Experiments on three video ReID datasets demonstrate superiority over existing state-of-the-art methods.
- The two-stream network explicitly leverages spatial and temporal cues for video person ReID.
- M3D layers learn multi-scale temporal cues with only a fraction of increased parameters.
- Residual Attention Layers refine M3D-learned temporal features in a residual manner before fusion with spatial representations.
- Experiments on three widely used video ReID datasets demonstrate superiority over current state-of-the-art methods.