Source-linked AI summary
CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification
Chaoyou Fu, Yibo Hu, Xiang Wu, Hailin Shi, Tao Mei, Ran He
TL;DR
VI-ReID must match visible and infrared pedestrian images despite large modality discrepancies, while manually designed two-stream architectures require time-consuming experimentation. CM-NAS searches BN-layer separation schemes automatically using a BN-oriented search space, and improves Rank-1 and mAP over state-of-the-art methods on SYSU-MM01 and RegDB.
Problem
VI-ReID faces large modality discrepancies, and manually designing two-stream separation schemes is time consuming and labor intensive.
Method
CM-NAS automatically searches whether each Batch Normalization layer should separate using a BN-oriented neural architecture search space.
Results
CM-NAS improves Rank-1/mAP by 6.70%/6.13% on SYSU-MM01 and 12.17%/11.23% on RegDB over state-of-the-art methods.
Takeaways & Limitations
Appropriately separating BN layers is identified as important for cross-modality matching, motivating automatic search for the separation scheme.
Abstract
from arXiv · showhide
Visible-Infrared person re-identification (VI-ReID) aims to match cross-modality pedestrian images, breaking through the limitation of single-modality person ReID in dark environment. In order to mitigate the impact of large modality discrepancy, existing works manually design various two-stream architectures to separately learn modality-specific and modality-sharable representations. Such a manual design routine, however, highly depends on massive experiments and empirical practice, which is time consuming and labor intensive. In this paper, we systematically study the manually designed architectures, and identify that appropriately separating Batch Normalization (BN) layers is the key to bring a great boost towards cross-modality matching. Based on this observation, the essential objective is to find the optimal separation scheme for each BN layer. To this end, we propose a novel method, named Cross-Modality Neural Architecture Search (CM-NAS). It consists of a BN-oriented search space in which the standard optimization can be fulfilled subject to the cross-modality task. Equipped with the searched architecture, our method outperforms state-of-the-art counterparts in both two benchmarks, improving the Rank-1/mAP by 6.70%/6.13% on SYSU-MM01 and by 12.17%/11.23% on RegDB. Code is released at https://github.com/JDAI-CV/CM-NAS.
1. Introduction
VI-ReID matches visible and infrared pedestrian images to address the inability of visible cameras to capture people clearly in dark environments. CM-NAS identifies effective Batch Normalization separation schemes automatically, improving cross-modality matching on two benchmarks.
- VI-ReID matches visible and infrared pedestrian images because visible cameras cannot image clearly in dark environments.
- Existing two-stream architectures separate some layers for modality-specific representations and share others for modality-sharable representations, but the appropriate layers remain unclear.
- 195 manually designed architectures show that separating BN layers within blocks outperforms separating entire blocks, while separating two BN blocks generally outperforms separating one.
- CM-NAS automatically determines whether each BN layer separates within a BN-oriented search space, addressing the large number of possible separation schemes.
- 6.70%/6.13% Rank-1/mAP improvements are achieved on SYSU-MM01, and 12.17%/11.23% on RegDB, compared with state-of-the-art methods.
2. Related Works
Related work covers single-modality person ReID, VI-ReID methods for cross-modal matching, and neural architecture search strategies. These lines position CM-NAS within established approaches to person matching and architecture optimization.
- Single-Modality Person ReID: Single-modality person ReID matches pedestrian images across non-overlapping visible cameras using handcrafted descriptors, metric learning, or deep learning.
- Single-Modality Person ReID: Visible cameras cannot image clearly in dark environments, limiting the application of single-modality person ReID.
- Visible-Infrared Person ReID: VI-ReID matches pedestrian images from different modalities and has motivated datasets, zero-padding, and generative approaches to address modality differences.
- Neural Architecture Search: NAS methods include micro search over reusable cells and macro search over whole architectures, with differentiable search reducing computational overhead relative to traditional methods.
3. Method
CM-NAS searches how each Batch Normalization layer should separate or share parameters across visible and infrared modalities. The method builds on systematic architecture comparisons and optimizes a BN-oriented search space with cross-modality objectives.
- Analyses of Manually Designed Architectures: Systematic comparisons examine block-level and BN-layer-level separation in ResNet50 two-stream architectures.The backbone uses ResNet50 pretrained on ImageNet, with modality-specific branches separated at selected architectural units.
- Analyses of Manually Designed Architectures: Separating two BN-layer blocks generally outperforms separating a single block, motivating broader BN separation searches.The comparison particularly favors BN-layer separation over separating all layers in a block.
- Cross-Modality NAS: CM-NAS treats every backbone BN layer as a searchable unit with separate VIS/IR parameters or shared parameters as candidate operations.The search space directly encodes the choice between modality-specific and modality-shared normalization.
- Cross-Modality NAS: Architecture parameters are relaxed with softmax probabilities, and each BN layer ultimately keeps the operation with the larger probability.The resulting discrete architecture selects either separate or shared BN parameters layer by layer.
- Cross-Modality NAS: The search uses bi-level optimization in which network weights minimize training loss while architecture parameters minimize validation loss.The training objective combines classification and triplet losses with cross-modality discrepancy and correlation-consistency terms.
- Cross-Modality NAS: Cross-modality objectives combine class-specific maximum mean discrepancy with correlation consistency to align visible and infrared representations.The discrepancy term uses a polynomial-kernel mapping, while correlation consistency compares normalized feature-similarity matrices.
4. Experiment
Experiments evaluate CM-NAS on SYSU-MM01 and RegDB, analyzing normalization, loss components, parameter sensitivity, searched architectures, and comparisons with prior methods. CM-NAS consistently improves cross-modality retrieval, including cross-dataset transfer, while requiring only small additional parameters and no extra computational costs.
- Experimental Settings: SYSU-MM01 contains 287,628 visible images and 15,792 near-infrared images, while RegDB contains 4,120 paired visible-thermal images from 412 identities.SYSU-MM01 provides daytime visible and dark-environment near-infrared images; RegDB uses synchronized visible and thermal-IR cameras.
- Experimental Settings: Evaluation uses CMC and mAP, with SYSU-MM01 and RegDB results averaged over 10 repeated random splits.SYSU-MM01 averages probe/gallery splits, whereas RegDB averages random training/testing splits.
- Experimental Analyses: Separating only BN layers outperforms separating entire blocks, supporting the importance of BN-layer separation for VI-ReID.The comparison also reports that two-stream methods outperform one-stream methods.
- Experimental Analyses: Adding LC3MMD improves all six baselines; for BN-oriented two-stream training on SYSU-MM01, Rank-1 and mAP increase by 1.0% and 1.06%.Using LC3MMD in both search and training phases performs better for the search method than using it in only one phase.
- Experimental Analyses: Removing either LCMMD or LCC degrades performance, while replacing LCMMD with LMMD also produces inferior results.The paper attributes the difference to LCMMD incorporating class labels.
- Experimental Analyses: CM-NAS is relatively insensitive to λ1 and λ2 over a broad range; the recommended setting is λ1 = 0.05 and λ2 = 5.0.When λ1 changes from 0.01 to 0.1, Rank-1 changes by only 0.67%, while overly large values reduce performance.
- Experimental Analyses: The searched architectures usually share high-level BN layers, but retain complex mixed separation patterns that are difficult to design manually.All BN layers in stage5 are shared, consistent with experiments showing that separating stage5 layers hurts performance.
- Comparisons with State-of-the-Art Methods: On SYSU-MM01, CM-NAS improves over HAT by 6.70% Rank-1 and 6.13% mAP under the single-shot&all-search setting.CM-NAS uses 256×128 inputs, smaller than HAT's 288×144 inputs; CM-NAS† also surpasses counterparts after cross-dataset architecture transfer.
5. Conclusion
CM-NAS addresses challenging VI-ReID by automatically searching BN-layer separation schemes after analysis showed that appropriate BN separation improves performance. Experiments on two datasets demonstrate its superiority and motivate its use as a foundation for future VI-ReID research.
- CM-NAS automatically decides BN-layer separation to search for an optimal architecture for VI-ReID.
- Appropriately separating BN layers yields better performance than manually designed architectural alternatives.
- Experiments on two popular datasets demonstrate CM-NAS's superiority.
- The authors present CM-NAS as a simple, effective foundation for future VI-ReID research.