Source-linked AI summary
Multi-Granularity Reference-Aided Attentive Feature Aggregation for Video-based Person Re-identification
Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, Zhibo Chen
TL;DR
Video re-identification can use information across frames, but must contend with redundancy, occlusion, and motion blur. MG-RAFA learns global attention from feature relations to representative reference nodes at multiple granularities. It achieves state-of-the-art performance on three benchmark datasets.
Problem
Video re-identification uses information across frames but must handle redundancy, occlusion, and motion blur when identifying people.
Source: §1 · p. 1Method
MG-RAFA infers global feature attention from relations to a small representative set of reference nodes and models those relations at multiple granularities.
Source: §1 · p. 2Results
MG-RAFA achieves state-of-the-art performance on three benchmark datasets.
Source: §5 · p. 8Takeaways & Limitations
The scheme enhances discriminative features and suppresses identity-irrelevant features in spatial and temporal representations.
Source: §5 · p. 8Results on DukeMTMC-VideoReID are not included because the dataset is no longer publicly released.
Source: §4.5 · p. 8
Abstract
from arXiv · showhide
Video-based person re-identification (reID) aims at matching the same person across video clips. It is a challenging task due to the existence of redundancy among frames, newly revealed appearance, occlusion, and motion blurs. In this paper, we propose an attentive feature aggregation module, namely Multi-Granularity Reference-aided Attentive Feature Aggregation (MG-RAFA), to delicately aggregate spatio-temporal features into a discriminative video-level feature representation. In order to determine the contribution/importance of a spatial-temporal feature node, we propose to learn the attention from a global view with convolutional operations. Specifically, we stack its relations, i.e., pairwise correlations with respect to a representative set of reference feature nodes (S-RFNs) that represents global video information, together with the feature itself to infer the attention. Moreover, to exploit the semantics of different levels, we propose to learn multi-granularity attentions based on the relations captured at different granularities. Extensive ablation studies demonstrate the effectiveness of our attentive feature aggregation module MG-RAFA. Our framework achieves the state-of-the-art performance on three benchmark datasets.
1. Introduction
Video re-identification draws on richer information across frames but must handle redundancy, occlusion, and motion blur while distinguishing useful features. MG-RAFA learns global, multi-granularity attention using representative reference nodes, and the final scheme outperforms state-of-the-art approaches on three benchmarks.
1. Introduction
- Video frames provide more comprehensive information than still images, but their overlapping and repetitive contents create redundancy alongside occlusion and motion blur.Some discriminative cues may appear only occasionally across frames.Source: §1 · p. 1, Figure 1 · p. 1
- The paper identifies a need for global attention that jointly assesses spatiotemporal feature importance and redundancy, and for mechanisms that capture semantics at different granularities.Its stated contributions are MG-RAFA, multi-granularity relation-based attention, and a compact representative reference set.Source: §1 · p. 2
- The method uses each feature node’s relations to representative reference nodes, together with the feature itself, to infer attention from a global view.The reference nodes represent global structural information, while the feature supplies appearance information.Source: §1 · p. 2
- MG-RAFA hierarchically models relations at multiple granularities to capture diverse semantics while using a small representative reference set instead of all feature nodes.The design is intended to ease attention learning and reduce computational complexity.Source: §1 · p. 2
- MG-RAFA significantly outperforms state-of-the-art approaches on three benchmark datasets, with less than 1% added computational complexity over the baseline.Its multi-granularity version also reduces complexity compared with the single-granularity version.Source: §1 · p. 2
2. Related Work
Prior video re-identification methods aggregate frame features in several ways, while attention approaches vary in whether they use spatial, temporal, or sequence-pair information. The paper positions MG-RAFA as a joint global spatiotemporal attention approach that models relations at multiple granularities.
2. Related Work
- Video re-identification methods include temporal pooling, recurrent aggregation, and 3D convolution, but these approaches treat feature positions as equally important.The cited works use frame-wise features or sequence-level representations in different ways.Source: §2 · p. 2
- Image-based re-identification attention uses human part, pose, or mask information, or learns spatial and channel attention.Relation-aware global attention learns attention using pairwise relations.Source: §2 · p. 2
- Some video attention methods condition one sequence’s temporal weights on its match with another sequence [36, 25, 4], requiring query-specific gallery features in one cited approach.That query-specific feature preparation is described as complicated and less practical.Source: §2 · p. 3
- Other video methods estimate frame quality, update frame features with temporal attention, or reweight attribute-related features, but do not jointly derive spatial and temporal attention globally.A predefined spatial-temporal map approach is described as having limited optimization space.Source: §2 · p. 3
- The paper addresses the lack of effective joint spatiotemporal attention for video feature aggregation with multi-granularity, reference-aided global attention.The proposed framework uses representative reference nodes to model global relations and capture semantics at different levels.Source: §2 · p. 3
3. Multi-Granularity Reference-aided Attentive Feature Aggregation
MG-RAFA aggregates video features using global relations to representative reference nodes, combining each node’s appearance and relational information to estimate its importance. It extends this process across granularities and applies retrieval and classification losses to the fused video representation.
3.1. Overview
- MG-RAFA samples T frames, extracts feature maps, and weights and sums their spatial-temporal nodes into a video-level representation for matching by L2 distance.A single-frame feature extractor, such as ResNet-50, produces the frame feature maps.Source: §3.1 · p. 3
3.2. Reference-aided Attentive Feature Aggregation
- To form the reference set, MG-RAFA temporally averages feature maps into a smaller set of representative nodes, reducing relation-modeling complexity while retaining appearance information.The reference map contains H × W nodes, rather than the H × W × T nodes in the full feature set.Source: §3.2 · pp. 3–4
- For each node, MG-RAFA combines its appearance feature with a relation vector of affinities to reference nodes, then normalizes attention across nodes and uses it to aggregate features.The relation vector captures global, clustering-like structure; the original feature represents local appearance.Source: §3.2 · pp. 4–5
3.3. Multi-Granularity Attention
- MG-RAFA models relations and derives attention at multiple feature-map resolutions, assigning separate channel groups to granularities and concatenating their fused features.The paper motivates this hierarchy by noting that different semantics, such as clothing details and body shape, may be captured at different granularities.Source: §3.3 · p. 5
3.4. Loss Design
- The training objective applies hard-mining triplet loss and label-smoothed cross-entropy to the final video feature and to each granularity’s aggregated feature.Source: §3.4 · p. 5
4.1. Datasets and Evaluation Metrics
The evaluation covers MARS, iLIDS-VID, and PRID2011 using CMC and mAP metrics.
4.1. Datasets and Evaluation Metrics
- Table 1 presents three public video-based person reID datasets.Source: Table 1 · p. 5, §4.1 · p. 5
- MARS uses its defined train/test split; iLIDS-VID and PRID2011 report CMC averaged over 10 random half-half train/test splits.The stated metrics are CMC at Rank-1 through Rank-20 and mean average precision (mAP).Source: §4.1 · p. 6
4.2. Implementation Details
The implementation uses a ResNet-50 backbone and applies MG-RAFA after the final residual block to produce the video feature.
4.2. Implementation Details
- The baseline uses global spatial-temporal average pooling, while MG-RAFA is applied after conv5 x to obtain the final feature vector.The last spatial down-sampling operation in conv5 x is removed for both systems.Source: §4.2 · p. 6
- Training samples one frame from each of 8 uniform video chunks and uses Adam with hard-mined triplet loss.The input resolution is 256×128; batches use P=16, Z=4 for MARS and P=8, Z=4 for iLIDS-VID and PRID2011.Source: §4.2 · p. 6
4.3. Ablation Study
The ablations test MG-RAFA’s relations, granularity choices, reference-node pooling, and extension to other attention designs. The reported comparisons favor multiple granularities and pooled reference nodes over the specified alternatives.
4.3.1 Effectiveness Analysis
- MG-RAFA improves Rank-1 over the baseline by 2.9%, 2.1%, and 3.4% on Mars, iLIDS-VID, and PRID2011, respectively.It exceeds MG-AFA without relations by 2.2%, 1.9%, and 3.3%, and SG-RAFA(S=1) by 1.0%, 1.5%, and 2.3% on those datasets; finer granularity performs better, while MG-RAFA uses 9.2% of SG-RAFA(S=1)'s FLOPs.Source: §4.3.1 · pp. 6–7
4.3.2 Selection of the Reference Feature Nodes
- The selected S-RFNs outperform ST(16×8×8) by 0.8% mAP while reducing aggregation FLOPs from 6.697G to 2.301G.They also outperform spatial-pooling alternatives by 1.2%, 1.2%, 1.0%, and 0.9% mAP, respectively; the authors attribute their lower performance to removing useful information.Source: §4.3.2 · p. 7
4.3.3 Comparison with Non-local
- SG-RAFA exceeds NL(ST) and NL(S) by 2.4% and 1.9% mAP, respectively.NL(ST) applies non-local refinement across all spatio-temporal features, while NL(S) applies it within each frame.Source: §4.3.3 · p. 7, Table 4 · p. 7
4.3.4 Extension of MG Design to Other Attention
- Multi-granularity design improves mAP over single-granularity RGA-SC, SE, and CBAM by 1.5%, 1.4%, and 1.7%, respectively.MG-RAFA also exceeds the multi-granularity versions of those methods by 0.9%, 1.6%, and 1.3% mAP, respectively.Source: Table 5 · p. 7, §4.3.4 · p. 8
4.4. Comparison with State-of-the-arts
MG-RAFA outperforms state-of-the-art methods on all three reported benchmark datasets.
4.4. Comparison with State-of-the-arts
- MG-RAFA improves mAP over STA by 5.1% on Mars and Rank-1 over the second-best method by 2.3% on iLIDS-VID and 0.4% on PRID2011.Source: §4.4 · p. 8
4.5. Visualization Analysis
MG-RAFA’s attention focuses on different semantic regions across frames, excludes interference such as occlusions, and captures different semantics at finer and coarser granularities.
4.5. Visualization Analysis
- Attention focuses on different semantic regions across frames, reducing repetition, and can exclude interference in occluded areas.The paper highlights occlusions in the third and fourth columns of Figure 3(a).Source: §4.5 · p. 8
- Finer granularities tend to capture more details, while coarser granularities tend to capture larger body parts.Figure 3 visualizes the four granularities at spatial resolutions of 16×8, 8×4, 4×2, and 2×1.Source: §4.5 · p. 8, Figure 3 · p. 8
- Results on DukeMTMC-VideoReID are not included because the dataset is no longer publicly released.Source: §4.5 · p. 8
- Figure 3(a) visualizes attention across frames at the second granularity.The attention masks are rescaled to the same spatial resolution for visualization.Source: §4.5 · p. 8, Figure 3 · p. 8
5. Conclusion
MG-RAFA aggregates spatial and temporal features for video-based person re-identification, using reference nodes and multi-granularity attention; the framework achieves state-of-the-art performance on three benchmark datasets.
5. Conclusion
- MG-RAFA enhances discriminative features and suppresses identity-irrelevant features in spatial and temporal representations.Source: §5 · p. 8
- A representative set of reference feature nodes models global relations to reduce optimization difficulty.Source: §5 · p. 8
- Multi-granularity attention explores relations at different levels to capture semantics at different levels.Source: §5 · p. 8
- MG-RAFA achieves state-of-the-art performance on three benchmark datasets.Source: §5 · p. 8