Source-linked AI summary
Neural Feature Search for RGB-Infrared Person Re-Identification
Yehansen Chen, Lin Wan, Zhihang Li, Qianyan Jing, Zongyuan Sun
TL;DR
RGB-IR person re-identification must match identities across visible and infrared images despite modality discrepancy, while existing feature selection often depends on manual design. NFS automates this process by searching spatial and channel-wise cues with differentiable, contrastive optimization, and it outperforms prior methods on standard benchmarks. The paper reports gains including 11.20% Rank-1 on RegDB and 12.01% Rank-1 on SYSU-MM01 under specified settings.
Problem
RGB-IR ReID must identify people across modalities whose visual discrepancy makes appearance cues unreliable and complicates feature alignment.
Method
NFS formulates feature selection as bilevel optimization over spatial and channel-wise features, using differentiable search and cross-modality contrastive supervision.
Results
NFS outperforms previous state-of-the-art methods on standard RGB-IR ReID benchmarks, including a 11.20% Rank-1 gain on RegDB and a 12.01% Rank-1 gain on SYSU-MM01 under specified settings.
Takeaways & Limitations
Automated feature selection adaptively focuses the network on informative human-body parts, filters sample noise, and searches modality-invariant cues.
Abstract
from arXiv · showhide
RGB-Infrared person re-identification (RGB-IR ReID) is a challenging cross-modality retrieval problem, which aims at matching the person-of-interest over visible and infrared camera views. Most existing works achieve performance gains through manually-designed feature selection modules, which often require significant domain knowledge and rich experience. In this paper, we study a general paradigm, termed Neural Feature Search (NFS), to automate the process of feature selection. Specifically, NFS combines a dual-level feature search space and a differentiable search strategy to jointly select identity-related cues in coarse-grained channels and fine-grained spatial pixels. This combination allows NFS to adaptively filter background noises and concentrate on informative parts of human bodies in a data-driven manner. Moreover, a cross-modality contrastive optimization scheme further guides NFS to search features that can minimize modality discrepancy whilst maximizing inter-class distance. Extensive experiments on mainstream benchmarks demonstrate that our method outperforms state-of-the-arts, especially achieving better performance on the RegDB dataset with significant improvement of 11.20% and 8.64% in Rank-1 and mAP, respectively.
1. Introduction
RGB-IR ReID must match identities across visible and infrared views despite modality discrepancy and large intra-class variation. NFS automates feature selection through a dual-level search space and contrastive supervision, outperforming prior state-of-the-art methods on mainstream benchmarks.
- RGB-IR ReID matches people across visible and infrared views, where imaging differences make colors and textures unreliable for retrieval.
- Manual feature-selection modules use cues such as partition stripes, human landmarks, parsing maps, and body contours but are difficult and time-consuming to design across variations.
- NFS formulates feature selection as bilevel optimization and uses data-driven search neurons to discover discriminative feature subsets.
- 11.20% Rank-1 accuracy gain was reported on RegDB visible-to-infrared mode, while SYSU-MM01 multi-shot all-search mode gained 12.01%.
- NFS jointly selects spatial and channel-wise features while contrastive optimization discourages irrelevant cues and encourages modality-invariant features.
- NFS outperforms previous state-of-the-art methods on two mainstream RGB-IR ReID benchmarks.
2. Related Work
Related work addresses ReID variation and modality discrepancy through RGB-based methods, RGB-IR synthesis or shared feature learning, and neural architecture search. NFS is positioned as automated feature selection rather than automated network topology design.
- RGB-based Person ReID: RGB-based ReID methods address pose, scale, and background-clutter variation in visible images using deep feature learning and alignment.
- RGB-Infrared Person ReID: RGB-IR ReID additionally addresses modality discrepancy from the different wavelength ranges of visible and infrared cameras.
- Neural Architecture Search: NFS differs from NAS by searching spatial and channel-wise features within a two-stream CNN feature space.
- RGB-Infrared Person ReID: RGB-IR methods commonly use GAN-based synthesis or shared feature learning to handle intra- and inter-modality variation.
- Neural Architecture Search: Neural architecture search automatically discovers task-oriented network operations or topologies from a predefined search space.
3. Methodology
NFS automates RGB-IR ReID feature selection by searching spatial pixels and channels within a CNN feature space. Its differentiable bilevel search is supervised by baseline ReID and cross-modality contrastive objectives to select informative, modality-invariant cues.
- Differentiable Feature Search: NFS formulates feature selection as bilevel optimization, updating search gates with validation loss and network weights with training loss.After deriving optimal gates, the full network is trained with the selected features and evaluated using the standard RGB-IR ReID procedure.
- Dual-level Feature Search Space: NFS searches spatial and channel-wise features within CNN feature maps instead of searching network topology or operations.Its search space decomposes features into pixel-level local patches and depth-level detector response maps.
- Dual-level Feature Search Space: Modality-aware search cells assign selection probabilities separately across spatial positions and channels, then use sigmoid-activated fields and binary gates to select features.The pixel-level space covers spatial positions, while the depth-level space contains all channels of each feature map.
- Differentiable Feature Search: Continuous Bernoulli reparameterization and a straight-through estimator make the discrete, large-scale search space continuously optimizable with stochastic gradient descent.This avoids inefficient brute-force enumeration of feature subsets.
- Cross-Modality Contrastive Optimization: RGB-IR open-set recognition and modality divergence motivate selecting invariant features that reduce feature-distribution variance and maximize inter-class distance.Testing identities differ from training identities, while appearance discrepancies can enlarge within-class variance and blur identity boundaries.
- Cross-Modality Contrastive Optimization: The cross-modality contrastive criterion pulls positive RGB-IR pairs together while pushing negative pairs apart with an explicit margin.The overall NFS objective combines this contrastive loss with the baseline ReID loss using a trade-off coefficient.
4. Experiments
Experiments on SYSU-MM01 and RegDB show that NFS improves RGB-IR ReID through automated feature selection and cross-modality contrastive optimization. Ablations and visualizations indicate that its gains arise from focusing on informative human regions, filtering noise, and reducing modality discrepancies.
- Comparison with State-of-the-art Methods: NFS achieves 56.91% Rank-1 and 55.45% mAP on SYSU-MM01 in the single-shot, all-search setting.
- Comparison with State-of-the-art Methods: 11.20% Rank-1 and 8.64% mAP gains over DDAG are achieved on RegDB in visible-to-infrared evaluation.
- Ablation Study: 1.91% Rank-1 and 1.46% mAP improvements over the baseline show that NFS focuses on informative body regions and filters sample noise.
- Ablation Study: Contrastive optimization adds 5.29% Rank-1 and 4.45% mAP in all-search evaluation and gives NFS an additional 4.62% Rank-1 and 4.54% mAP over the baseline.
- Ablation Study: Searching stages 1, 2, and 3 produces the best result, while broader search spaces can make optimal feature discovery more difficult.
- Influence of Hyperparameters: NFS performs best at contrastive margin T = 15 and trade-off coefficient λ = 0.04, while remaining consistently better across the tested settings.
5. Conclusion
The paper proposes Neural Feature Search (NFS) to automatically discover identity-related, modality-invariant features for RGB-IR person re-identification. Experiments on two standard benchmarks demonstrate that NFS surpasses previous state-of-the-art methods.
- NFS automatically discovers identity characteristics for RGB-IR person re-identification.The method frames feature selection as an automated search rather than relying solely on manually designed modules.
- Its dual-level search space jointly performs global channel-wise and local spatial feature selection.This design supports searching both coarse-grained channels and fine-grained spatial regions.
- Cross-modality contrastive optimization guides the search toward high-quality invariant feature subsets for matching and retrieval.The objective is used to improve the quality of selected features across modalities.
The Architecture of Our Baseline Network
The baseline uses a commonly used two-stream network with ResNet-50 as its backbone. Its architecture is detailed in Table 5, with images resized to 288 × 144 and fine-grained feature maps produced by reducing the final convolutional stride.
- The baseline network adopts a commonly used two-stream structure.
- ResNet-50 serves as the baseline network's backbone.
- Input images are resized to 288 × 144 for network processing.
- The last convolutional block uses stride 1 to obtain fine-grained feature maps.
Visualization of Retrieved Examples
The visualization presents top-5 NFS retrievals for randomly selected SYSU-MM01 queries, ordered by cosine similarity. Correct and incorrect matches are marked separately, illustrating retrieval under modality variation and noisy conditions.
- Retrieved samples are ordered from left to right by descending cosine similarity scores.The numerical values report cosine similarities between image pairs.
- NFS retrieves correct results in difficult cases involving missing IR color information, background clutter, partial occlusions, and clothing changes.The examples include queries that are difficult even for humans and demonstrate robustness to several sources of noise.
- Top-5 retrieval results are shown for 16 randomly selected SYSU-MM01 query samples.The visualization includes eight samples per query setting.
- Correct retrievals are marked with green boxes, while wrong matchings are marked with red boxes.