Source-linked AI summary
Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification
Mang Ye, Jianbing Shen, David J. Crandall, Ling Shao, Jiebo Luo
TL;DR
VI-ReID requires matching visible and infrared person images despite large modality variation and noisy samples, while existing methods often rely on limited global representations. DDAG combines intra-modality weighted-part attention, cross-modality graph structured attention, and parameter-free dynamic aggregation. Experiments report that DDAG outperforms state-of-the-art models across various settings, often by a large margin.
Problem
VI-ReID makes discriminative part-feature learning difficult because of large intra-class and cross-modality variation, sample noise, and the limitations of global representations.
Method
DDAG uses IWPA for intra-modality part-context modeling, CGSA for cross-modality graph-context modeling, and parameter-free dynamic aggregation for progressive joint training.
Results
DDAG outperforms state-of-the-art models on various settings, usually by a large margin.
Takeaways & Limitations
Mining relationships among body parts and cross-modality neighboring images provides the basis for enhanced VI-ReID representations within the proposed framework.
Abstract
from arXiv · showhide
Visible-infrared person re-identification (VI-ReID) is a challenging cross-modality pedestrian retrieval problem. Due to the large intra-class variations and cross-modality discrepancy with large amount of sample noise, it is difficult to learn discriminative part features. Existing VI-ReID methods instead tend to learn global representations, which have limited discriminability and weak robustness to noisy images. In this paper, we propose a novel dynamic dual-attentive aggregation (DDAG) learning method by mining both intra-modality part-level and cross-modality graph-level contextual cues for VI-ReID. We propose an intra-modality weighted-part attention module to extract discriminative part-aggregated features, by imposing the domain knowledge on the part relationship mining. To enhance robustness against noisy samples, we introduce cross-modality graph structured attention to reinforce the representation with the contextual relations across the two modalities. We also develop a parameter-free dynamic dual aggregation learning strategy to adaptively integrate the two components in a progressive joint training manner. Extensive experiments demonstrate that DDAG outperforms the state-of-the-art methods under various settings.
1 Introduction
VI-ReID must match visible and infrared person images despite large modality differences, intra- and cross-modality variation, and noisy samples. DDAG addresses these challenges by combining intra-modality part aggregation with cross-modality graph attention and dynamically aggregating both components.
- Problem: VI-ReID matches visible-spectrum and infrared person images, but large visual differences and changing camera environments create substantial variation.Data collection and annotation difficulties also introduce noise from inaccurate detections and background clutter.
- Problem: Existing VI-ReID methods mainly learn sharable global features, which are sensitive to background clutter and do not explicitly handle modality discrepancy.These limitations make discriminative part-level feature learning difficult.
- Approach: DDAG mines contextual cues at both intra-modality part and cross-modality graph levels using a two-stream network.Its two components are IWPA and CGSA.
- Approach: CGSA incorporates relationships among person images across modalities, adaptively weighting intra- and cross-modality neighbors to reduce modality discrepancy and smooth training.The method uses a multi-head attentive graph scheme.
- Approach: IWPA learns discriminative part-aggregated representations by modeling contextual relations among body parts within each modality and adaptively weighting them.A residual BatchNorm connection reduces the impact of noisy body parts and handles part discrepancy.
- Results: DDAG establishes a new baseline on two VI-ReID datasets and outperforms state-of-the-art methods by a large margin.The reported conclusion covers various settings.
2 Related Work
Related work spans single-modality Re-ID, cross-modality person Re-ID, and visible near-infrared face recognition. The paper distinguishes its part-level attention from pixel-level non-local designs that are sensitive and inefficient for VI-ReID noise.
- Single-Modality Person Re-ID: Single-modality person Re-ID uses global or part-level features but is usually unable to handle VI-ReID’s modality discrepancy.This limits applicability in night-time surveillance scenarios.
- Cross-Modality Person Re-ID: Cross-modality person Re-ID research addresses modality differences through ranking losses, adversarial training, GANs, modality-specific classifiers, and modality-aware learning.Other approaches include collaborative ensembles, grayscale tri-modal learning, intermediate X-modalities, and non-local attention.
- Part Attention: DDAG’s part-attention design differs from pixel-level non-local networks by using learnable weighted part-level attention with a BatchNorm residual connection.The paper states that pixel-level designs are sensitive and inefficient for VI-ReID noise.
3 Proposed Method
DDAG is a two-stream VI-ReID framework that combines intra-modality part aggregation with cross-modality graph attention. A parameter-free dynamic strategy progressively integrates these components for joint training.
- Framework: DDAG uses a two-stream network with modality-specific early parameters and shared deep convolutional blocks for visible and infrared inputs.The baseline combines identity loss with online hard-mining triplet loss.
- IWPA: IWPA mines within-modality part relationships to learn discriminative weighted-part features instead of relying only on global representations.Feature maps are divided into non-overlapping parts, and a modified non-local module learns part attention.
- IWPA: IWPA uses a p×p part-attention map and learnable weighted aggregation with a residual BatchNorm connection to improve efficiency and reduce noisy-part accumulation.Part-level attention is smaller than pixel-level HW × HW attention and is described as more robust to noisy regions and local clutter.
- CGSA: CGSA reinforces node representations by assigning adaptive weights to intra- and cross-modality neighbors in a graph built from person-image identity relations.The graph attention uses transformed concatenated features and a learnable weighting vector to exploit same-identity context across modalities.
- Dynamic Dual Aggregation: Dynamic dual aggregation separates instance-level part learning LP from graph-level global learning Lg and progressively adds Lg because direct joint optimization can be unstable.The two components target different objectives, while early cross-modality feature differences can destabilize graph attention.
4 Experimental Results
Experiments on SYSU-MM01 and RegDB evaluate DDAG’s components, design choices, and comparisons with state-of-the-art VI-ReID methods. The results support benefits from weighted-part aggregation, graph structured attention, and their dynamic combination across testing settings.
- Experimental Settings: Experiments use SYSU-MM01 and RegDB, reporting rank-k matching accuracy and mean Average Precision (mAP).SYSU-MM01 includes all-search and indoor-search settings; RegDB evaluates visible-to-infrared and infrared-to-visible queries.
- Evaluation of Each Component: The SYSU-MM01 component study finds that weighted-part aggregation and graph structured attention each improve the two-stream baseline.The graph module uses relationships between person images across modalities to reduce modality discrepancy.
- Evaluation of Each Component: Dynamic dual aggregation further improves performance, indicating that the two attention modules are mutually beneficial.The strategy aggregates the intra-modality and cross-modality attention components.
- Why Weighted Part Attention with RBN?: Learnable weighted-part attention outperforms two alternative designs, while residual BatchNorm performs better than a general residual connection.The weighted aggregation also produces a smaller final feature dimension than concatenation, and removing the residual connection substantially reduces performance.
- Parameter Evaluation: Increasing the number of body parts or graph attention heads generally helps, but excessive values reduce performance because of insufficient part information or optimization difficulty.The selected settings are p = 3 and L = 4 for all experiments.
- Comparison with State-of-the-Art Methods: DDAG significantly outperforms AlignGAN on both datasets and performs better under both RegDB query directions without image generation or adversarial training.The authors also report that DDAG is robust to visible-to-infrared and infrared-to-visible query settings.
5 Conclusion
DDAG addresses VI-ReID by combining intra-modality part relationships with cross-modality neighborhood information, then dynamically aggregating both components. It outperforms state-of-the-art models across various settings and may extend to single-modality person re-identification.
- DDAG combines IWPA part relationships within each modality with CGSA neighborhood information across modalities to enhance representations and reduce the modality gap.Its dynamic dual aggregation strategy seamlessly integrates the two components.
- DDAG outperforms state-of-the-art models across various settings, usually by a large margin.
- The authors suggest applying relation mining across body parts and contextual images to general single-modality person re-identification.