Source-linked AI summary
Learning Multi-Granular Hypergraphs for Video-Based Person Re-Identification
Yichao Yan, Jie Qin1, Jiaxin Chen, Li Liu, Fan Zhu, Ying Tai, Ling Shao
TL;DR
Video-based person re-ID needs representations that capture spatial and temporal clues while addressing misalignment and occlusion. MGH learns multi-granular hypergraphs with propagation, aggregation, and mutual-information minimization, achieving 90.0% top1 accuracy on MARS and favorable results across three benchmarks.
Problem
Video-based person re-ID requires joint modeling of spatial and temporal clues because misalignment and occlusion hinder robust identification.
Method
MGH constructs spatially multi-granular hypergraphs whose hyperedges capture temporal ranges, then propagates and aggregates features while preserving representation diversity through mutual information minimization.
Results
90.0% top1 accuracy on MARS is achieved by MGH, with favorable performance compared with recent state-of-the-art methods across three benchmarks.
Takeaways & Limitations
MGH provides a hypergraph-based representation that jointly exploits spatial and temporal dependencies for video-based person re-ID.
Abstract
from arXiv · showhide
Video-based person re-identification (re-ID) is an important research topic in computer vision. The key to tackling the challenging task is to exploit both spatial and temporal clues in video sequences. In this work, we propose a novel graph-based framework, namely Multi-Granular Hypergraph (MGH), to pursue better representational capabilities by modeling spatiotemporal dependencies in terms of multiple granularities. Specifically, hypergraphs with different spatial granularities are constructed using various levels of part-based features across the video sequence. In each hypergraph, different temporal granularities are captured by hyperedges that connect a set of graph nodes (i.e., part-based features) across different temporal ranges. Two critical issues (misalignment and occlusion) are explicitly addressed by the proposed hypergraph propagation and feature aggregation schemes. Finally, we further enhance the overall video representation by learning more diversified graph-level representations of multiple granularities based on mutual information minimization. Extensive experiments on three widely adopted benchmarks clearly demonstrate the effectiveness of the proposed framework. Notably, 90.0% top-1 accuracy on MARS is achieved using MGH, outperforming the state-of-the-arts. Code is available at https://github.com/daodaofr/hypergraph_reid.
1. Introduction
Video-based person re-ID must jointly exploit spatial and temporal clues to handle misalignment and occlusion. MGH addresses these challenges with multi-granular hypergraphs, propagation, aggregation, and mutual-information-based representation learning.
- Video-based person re-ID benefits from jointly modeling spatial and temporal clues because misalignment and occlusion complicate identification.
- MGH constructs hypergraphs at multiple spatial granularities, with hyperedges capturing temporal dependencies across different ranges.Graph nodes represent global or part-level features, while each hypergraph corresponds to a spatial granularity.
- Hypergraph propagation and feature aggregation address misalignment by associating related nodes and occlusion by modeling short- and long-term temporal dependencies.Hypergraphs model higher-order dependencies among multiple nodes rather than only pairwise relationships.
- MGH preserves and enhances diversity across graph representations using mutual information minimization.
- 90.0% top1 accuracy on MARS is achieved by MGH, outperforming state-of-the-art models.
2. Related Work
Prior video-based person re-ID research includes handcrafted descriptors, ranking methods, deep models, and graph-based approaches. Hypergraph learning extends conventional graphs by modeling higher-order relationships among multiple objects.
- Earlier video-based person re-ID methods used handcrafted spatiotemporal features, discriminative descriptors, or ranking algorithms.
- Deep learning models have more recently shown superior performance compared with handcrafted features in video-based person re-ID.
- Conventional graphs model pairwise relationships, limiting their scalability to data with more complex structures.
- Hypergraphs and HyperGraph Neural Networks model higher-order relationships and correlations among objects connected in hypergraphs.
3. Multi-Granular Hypergraph Learning
MGH learns video person re-ID representations by modeling spatial and temporal dependencies at multiple granularities with hypergraphs. It combines multi-granular feature extraction, hypergraph propagation, attentive aggregation, and mutual-information-based graph representation learning.
- Multi-Granular Feature Extraction: MGH extracts global and part-level features by hierarchically partitioning each frame's feature map into 1, 2, 4, and 8 horizontal parts.Average pooling produces Np = T × p part-level features for each spatial granularity.
- Hypergraph Construction: Separate hypergraphs represent spatial granularities, while hyperedges connect nearest-neighbor features across short-, mid-, and long-term temporal ranges.Three temporal thresholds T1, T2, and T3 define the temporal ranges.
- Hypergraph Propagation: Hypergraph propagation averages neighboring node features within each hyperedge, measures hyperedge importance by feature similarity, and updates nodes using aggregated messages.The update process repeats for L rounds through a fully connected layer combining prior node features and hyperedge messages.
- Attentive Hypergraph Feature Aggregation: Attentive aggregation weights node features before forming graph- or video-level representations, reducing the influence of occluded parts and background regions.The method explicitly accounts for unequal node importance within each hypergraph.
- Model Learning: The learning objective combines cross-entropy and triplet losses with mutual-information minimization across hypergraph representations.The final video representation concatenates graph-level features from different spatial granularities while encouraging their diversity.
4. Experimental Results
Experiments on MARS, iLIDS-VID, and PRID-2011 evaluate MGH’s components, granularity choices, hyperparameters, and comparisons with state-of-the-art methods. Results show that multi-granular hypergraph learning provides the main gains, while attention and mutual-information losses add further improvements.
- Experimental Setup: MGH is evaluated on MARS, iLIDS-VID, and PRID-2011 using CMC and mAP metrics under established dataset protocols.MARS uses a predefined train/test split, while iLIDS-VID and PRID-2011 average results over ten trials.
- Component Analysis: The hypergraph learning mechanism raises top-1 accuracy by 3% on MARS and 5% on iLIDS-VID and PRID-2011 over the baseline.The baseline uses cross-entropy and triplet losses with average pooling of frame-level features.
- Component Analysis: 90.0% top-1 accuracy and 85.8% mAP are achieved on MARS after adding mutual-information loss, improving from 85.7% and 78.3%.The attention modules additionally improve top-1 accuracy by 0.5%-1%.
- Multi-Granularity: Performance increases steadily as more detailed spatial and temporal granularities are captured.When the temporal range equals one, only adjacent nodes are connected.
- Sensitivity Analysis: Correlation aggregation outperforms max pooling, average pooling, and LSTM because those alternatives neglect hyperedge-feature dependency on the target node.The comparison is reported in Figure 5(a).
- Sensitivity Analysis: Two HGNN layers are selected because one layer has insufficient representational capability, whereas deeper networks introduce more parameters and training difficulty.Performance is relatively insensitive to layer count, with two layers slightly better than other settings.
- State-of-the-Art Comparisons: On MARS, MGH achieves 90% top-1 accuracy and 85.8% mAP without re-ranking, outperforming prior methods and improving mAP by 3.5% over the current best method.MGH dynamically constructs hypergraphs from feature affinities and models temporal dependencies across varying ranges.
- State-of-the-Art Comparisons: MGH achieves competitive results on iLIDS-VID and PRID-2011 while requiring only identity annotations, unlike methods using external attributes.It outperforms several recent models across all reported evaluation metrics on both datasets.
5. Conclusion
The proposed Multi-Granular Hypergraph framework addresses video-based person re-identification by jointly modeling spatial and temporal clues at multiple granularities. Experiments on three benchmarks showed favorable performance compared with recent state-of-the-art methods.
- MGH jointly exploits multi-granular spatial and temporal clues in video sequences through a hypergraph neural network.The framework models spatiotemporal dependencies across multiple granularities for video-based person re-identification.
- An attention mechanism aggregates node-level features into more discriminative graph representations.
- A mutual information loss learns more diversified multi-granular features.
- MGH achieved favorable performance compared with recent state-of-the-art methods on three person re-identification benchmarks.