PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared Person Re-Identification

Zhipeng Huang, Jiawei Liu, Liang Li, Kecheng Zheng, Zheng-Jun Zha

arXiv:2203.01735v2cs.CV

TL;DR

RGB-infrared person re-identification must address modality discrepancy between visible and infrared images. MID combines actor-critic-guided adaptive mixup with convolution decomposition, and experiments on two datasets show its effectiveness.

  • Problem

    RGB-infrared person re-identification must bridge modality discrepancy between RGB and infrared images to learn modality-invariant, discriminative representations.

    Source: Introduction · p. 1

  • Method

    MID uses actor-critic-guided modality-adaptive mixup to reduce pixel-level discrepancy and decomposes convolution into modality-specific bases and shared coefficients for feature-level learning.

    Source: Introduction · p. 2, Modality-Adaptive Conv Decomposition · p. 4

  • Results

    Extensive experiments on two datasets demonstrate the effectiveness of MID.

    Source: Introduction · p. 2, Conclusions · p. 7

  • Takeaways & Limitations

    MID combines pixel-level mixed-modality image generation with feature-level decomposition to learn modality-shared representations.

    Source: Conclusions · p. 7

    The formulation assumes RGB-IR training images have identity labels drawn from a defined set of persons.

    Source: Problem Definition and Overview · p. 3

Abstract

from arXiv · show

RGB-infrared person re-identification is an emerging cross-modality re-identification task, which is very challenging due to significant modality discrepancy between RGB and infrared images. In this work, we propose a novel modality-adaptive mixup and invariant decomposition (MID) approach for RGB-infrared person re-identification towards learning modality-invariant and discriminative representations. MID designs a modality-adaptive mixup scheme to generate suitable mixed modality images between RGB and infrared images for mitigating the inherent modality discrepancy at the pixel-level. It formulates modality mixup procedure as Markov decision process, where an actor-critic agent learns dynamical and local linear interpolation policy between different regions of cross-modality images under a deep reinforcement learning framework. Such policy guarantees modality-invariance in a more continuous latent space and avoids manifold intrusion by the corrupted mixed modality samples. Moreover, to further counter modality discrepancy and enforce invariant visual semantics at the feature-level, MID employs modality-adaptive convolution decomposition to disassemble a regular convolution layer into modality-specific basis layers and a modality-shared coefficient layer. Extensive experimental results on two challenging benchmarks demonstrate superior performance of MID over state-of-the-art methods.

Introduction

RGB-infrared person re-identification must bridge modality differences between visible and infrared images as well as appearance differences. MID combines modality-adaptive image mixup with convolution decomposition to learn modality-invariant, discriminative representations.

  • RGB-infrared Re-ID seeks matching images of the same pedestrian across RGB and infrared cameras, where different imaging processes create a prominent modality discrepancy.Source: Introduction · p. 1
  • Existing pixel-level translation methods can be difficult to optimize and introduce noisy samples, while one- and two-stream feature methods have distinct representation drawbacks.One-stream methods may neglect modality-specific characteristics; two-stream methods separate modality-specific and shared processing, potentially damaging shared semantics.Source: Introduction · pp. 1–2
  • MID uses actor-critic-driven, data-adaptive image mixup at the pixel level and modality-adaptive convolution decomposition at the feature level.The mixup policy interpolates regions of RGB-IR image pairs, while decomposition separates modality-specific basis layers from a modality-shared coefficient layer.Source: Introduction · p. 2
  • Unlike XIV, which generates an auxiliary X modality from RGB alone, MID accounts for IR characteristics when generating mixed-modality samples.The passage describes XIV's RGB-only generation as failing to produce high-quality samples for reducing the modality gap.Source: Introduction · p. 2
  • MID's stated contributions are pixel-level modality-adaptive mixup and feature-level convolution decomposition to capture invariant visual semantics.Source: Introduction · p. 2

Related work

Prior RGB-infrared Re-ID methods address modality discrepancy through image translation or feature-level networks. Existing mixup schemes use global random interpolation, while MID's architecture combines pixel-level mixup with feature-level convolution decomposition.

RGB-Infrared Person Re-ID

  • Feature-level RGB-infrared Re-ID methods use modality-specific and shared metrics, dual-path networks, contextual aggregation, or hetero-center loss with part pooling.Source: RGB-Infrared Person Re-ID · p. 2
  • Pixel-level prior methods generate cross-modality images or paired images using generative adversarial approaches.Source: RGB-Infrared Person Re-ID · p. 2

Domain Mixup

  • Mixup methods interpolate samples globally and randomly, but existing schemes are not suitable for RGB-infrared person Re-ID.Examples include MixUp and domain-mixup approaches for regularization or domain adaptation.Source: Domain Mixup · pp. 2–3
  • MID's architecture pairs a pixel-level modality-adaptive mixup module with a decomposed convolution network for feature-level discrepancy reduction.Source: Figure 1 · p. 3

Method

MID learns modality-invariant, discriminative representations for RGB-infrared person re-identification by combining adaptive image mixing with decomposed convolution features. Its training alternates between the decomposed network and an actor-critic agent.

Problem Definition and Overview

  • MID feeds RGB, IR, and mixed images into a decomposed convolution network, whose first three residual blocks use MACD to align invariant semantics; identification and center triplet losses optimize it.The network learns modality-invariant and discriminative features for re-identification.Source: Problem Definition and Overview · p. 3

Modality-Adaptive Mixup Scheme

  • Global random mixup can produce low-quality samples that intrude on the data manifold, so MAM learns dynamic, local interpolation between cross-modality image regions.An actor-critic agent uses intermediate RGB and IR features as its state and a continuous mix-ratio vector as its action; its reward is relative re-identification performance improvement.Source: Modality-Adaptive Mixup Scheme · pp. 3–4

Modality-Adaptive Conv Decomposition

  • MACD decomposes convolution filters into modality-specific dictionary bases and a shared coefficient, jointly addressing modality variation and promoting cross-modality shared semantics.The bases model modality variation; the shared coefficient weights corrected feature channels, and the network takes RGB, IR, and mixed images as input.Source: Modality-Adaptive Conv Decomposition · pp. 4–5

Loss Function and Optimization

  • MID alternates optimization of the decomposed network and actor-critic agent, combining supervised and reinforcement learning.The network uses identification and center triplet losses; the actor-critic agent uses a one-step process with continuous actions, and the actor is updated to achieve higher Q-values.Source: Loss Function and Optimization · p. 5

Experiments

MID was evaluated on RegDB and SYSU-MM01, where it led the reported state-of-the-art comparisons. Ablations examined its components, mixup choices, local-region count, and decomposition depth.

Experimental Setting

  • Evaluation used RegDB’s visible-to-thermal and thermal-to-visible modes and SYSU-MM01’s all-search and indoor-search modes, with CMC at Rank-k and mAP.Source: Experimental Setting · p. 5

Comparison to State-of-the-Art Methods

Ablation Study

  • Adding M improved rank-1 accuracy and mAP by 7.09% and 8.32% on RegDB and 6.76% and 7.80% on SYSU-MM01; adding D improved them by 4.02% and 5.72% and 4.05% and 2.28%, respectively.Source: Ablation Study · pp. 6–7, Table 2 · p. 6
  • Data-independent Fix and Beta mixup performed worse than data-dependent MAM, which adapts its mixup ratio to differences between RGB–IR image pairs.Source: Table 3 · p. 7, Ablation Study · p. 7
  • Local-region interpolation outperformed global interpolation, and six partitioned regions produced the best re-identification performance on RegDB.Source: Table 4 · p. 7, Ablation Study · p. 7
  • Increasing decomposed residual blocks from 0 to 3 improved rank-1 accuracy by 4.02% on RegDB and 4.05% on SYSU-MM01; increasing from 3 to 5 reduced it by 4.87% and 5.76%.MID performed best with three decomposed blocks.Source: Table 5 · p. 7, Ablation Study · p. 7
  • Visualizations show MAM images without the occlusion, blurring, and ghosting seen with vanilla mixup, alongside retrieval examples demonstrating identification and distinction of pedestrians.Source: Figure 2 · p. 7, Ablation Study · p. 7

Conclusions

MID addresses RGB–infrared modality discrepancy at both the pixel and feature levels. It combines actor-critic-guided modality-adaptive mixup with modality-adaptive convolution decomposition, and experiments on two datasets demonstrate superiority over other methods.

  • MID uses an actor-critic agent for modality-adaptive mixup, generating mixed images to reduce pixel-level modality gaps and promote a continuous modality-invariant latent space.Source: Conclusions · p. 7
  • MID applies modality-adaptive convolution decomposition to counter feature-level modality discrepancy and enforce cross-domain shared semantics.Source: Conclusions · p. 7
  • Experiments on two cross-modality person re-identification datasets demonstrate MID’s superiority over other methods.Source: Conclusions · p. 7

Related papers

Loading 2203.01735v2…