PaperlayerView exact revision on arXiv

Source-linked AI summary

SFANet: A Spectrum-aware Feature Augmentation Network for Visible-Infrared Person Re-Identification

Haojie Liu, Shun Ma, Daoxun Xia, Shaozi Li

arXiv:2102.12137v1cs.CV

TL;DR

VI-ReID must match people across visible and infrared cameras, where modality differences complicate shared feature learning and existing approaches may miss pixel-level variation. SFANet uses grayscale-spectrum augmentation and complementary feature-level constraints. In an ablation, grayscale augmentation outperforms the RGB-infrared strategy on both reported SYSU-MM01 metrics.

  • Problem

    VI-ReID must handle large visible-infrared modality differences, while prior feature-level methods overlook pixel-level variation.

    Source: §I · p. 1

  • Method

    SFANet replaces RGB inputs with grayscale-spectrum images and combines shared feature extraction with BTTR loss and dual-linear batch-normalized ID embedding.

    Source: §I · p. 2

  • Results

    On SYSU-MM01, grayscale augmentation reaches 43.70% rank-1 accuracy and 40.36% mAP, versus 38.81% rank-1 accuracy and 35.30% mAP without it.

    Source: B. Ablation Study · p. 8

  • Takeaways & Limitations

    The grayscale-spectrum approach approximates infrared style while retaining visible-image structure, providing image-level augmentation without GAN-based image generation.

    Source: §I · p. 2, A. Grayscale-spetrum Images Generation · p. 3

Abstract

from arXiv · show

Visible-Infrared person re-identification (VI-ReID) is a challenging matching problem due to large modality varitions between visible and infrared images. Existing approaches usually bridge the modality gap with only feature-level constraints, ignoring pixel-level variations. Some methods employ GAN to generate style-consistent images, but it destroys the structure information and incurs a considerable level of noise. In this paper, we explicitly consider these challenges and formulate a novel spectrum-aware feature augementation network named SFANet for cross-modality matching problem. Specifically, we put forward to employ grayscale-spectrum images to fully replace RGB images for feature learning. Learning with the grayscale-spectrum images, our model can apparently reduce modality discrepancy and detect inner structure relations across the different modalities, making it robust to color variations. In feature-level, we improve the conventional two-stream network through balancing the number of specific and sharable convolutional blocks, which preserve the spatial structure information of features. Additionally, a bi-directional tri-constrained top-push ranking loss (BTTR) is embedded in the proposed network to improve the discriminability, which efficiently further boosts the matching accuracy. Meanwhile, we further introduce an effective dual-linear with batch normalization ID embedding method to model the identity-specific information and assits BTTR loss in magnitude stabilizing. On SYSU-MM01 and RegDB datasets, we conducted extensively experiments to demonstrate that our proposed framework contributes indispensably and achieves a very competitive VI-ReID performance.

I. INTRODUCTION

VI-ReID must match identities across visible and infrared cameras despite modality and appearance differences, while prior approaches leave pixel-level variation insufficiently addressed or risk damaging image structure. SFANet combines grayscale-spectrum augmentation with a shared feature network, BTTR loss, and dual-linear batch-normalized ID embedding.

I. INTRODUCTION

  • SFANet uses grayscale-spectrum images instead of RGB for cross-modality learning, approximating infrared style while preserving visible-image structure.The authors contrast this with GAN-based image generation, which can destroy local structure.Source: §I · pp. 1–2
  • The sharable dual-path network shares shallow convolutional parameters to learn common features while preserving spatial structure.The authors report that it outperforms the first parameter-sharing work for VI-ReID.Source: §I · p. 2
  • BTTR constrains relative distances across same-modality and cross-modality samples to improve feature discriminability.Its inter-modality regularizer considers relationships between cross-modality and intra-modality distances, addressing a difficulty in discriminative embedding.Source: §I · p. 2
  • Dual-Linear with Batch Normalization ID Embedding models identity-specific information and is designed to improve classification accuracy and stabilize embedding magnitudes with BTTR.The method is intended to improve robustness to modality variation at the classifier level.Source: §I · p. 2

II. RELATED WORK

Prior VI-ReID methods addressed modality differences mainly through feature-level constraints, while image-generation methods used GANs. SFANet instead augments visible inputs with grayscale images designed to resemble infrared images while preserving structure.

II. RELATED WORK

  • Visible-camera Re-ID methods address appearance changes but are designed for a single modality and cannot perform well on cross-modality Re-ID.The reviewed methods cover feature extraction and metric learning for variations such as viewpoint, pose, occlusion, and illumination.Source: §II · p. 2
  • VI-ReID methods addressed cross-modality differences mainly with feature-level constraints, leaving pixel-level spectral variation unaddressed.GAN-based image generation offered another approach but required substantial resources and raised generator–discriminator balancing challenges.Source: §II · p. 3

III. OUR APPROACH

  • SFANet combines grayscale-spectrum augmentation, a partially shared feature network, and identity and ranking losses for VI-ReID.The approach section describes grayscale and infrared inputs entering a sharable dual-path network supervised by cross-modality sharable identity loss and BTTR loss.Source: §III · p. 3, Figure 3 · p. 3

A. Grayscale-spetrum Images Generation

  • The grayscale transform combines red, green, and blue pixel values with weights 0.299, 0.587, and 0.114, respectively.The resulting single-channel image is copied across three channels for model input; the authors report that it resembles infrared imagery while retaining structure and avoiding introduced noise.Source: A. Grayscale-spetrum Images Generation · pp. 3–4

B. Sharable Dual-path Information-preserving Network.

The sharable dual-path network separates modality-specific low-level extraction from shared high-level embedding. Its partially shared design retains 3D feature maps to preserve spatial structure.

B. Sharable Dual-path Information-preserving Network.

  • The network uses separate grayscale and infrared paths for modality-specific low-level features, then shares upper convolutional parameters to learn common high-level features.ResNet50 is the backbone, with Res-layer 0 and Res-layer 1 unshared.Source: B. Sharable Dual-path… · p. 4
  • Unlike a fully two-stream network, the partially shared design outputs 3D common feature maps rather than 1D maps from a shared fully connected layer.The authors state that this preserves spatial structure information and cross-view consistency.Source: B. Sharable Dual-path… · p. 4
  • During testing, pooled 2048-dimensional features are used as the final Re-ID features.During training, each mini-batch includes images from different modalities and the network extracts 2048-dimensional features for embedding.Source: B. Sharable Dual-path… · p. 4

C. Dual-Linear with Batch Normalization ID Embedding.

The section identifies limitations of a shared single-linear identity classifier and introduces dual-linear ID embedding with batch normalization. The added layers are intended to improve modality discrimination and stabilize embedding magnitudes.

C. Dual-Linear with Batch Normalization ID Embedding.

  • A shared single-linear classifier can hinder convergence and cross-modality discrimination when pooled features from different modalities feed directly into it.The authors attribute this to lost detail and internal covariate shift.Source: C. Dual-Linear with Batch… · p. 4
  • DL-IDE adds a linear layer after pooling and batch normalization before the final fully connected layer to model a more discriminative common feature space.The authors describe the added linear layer as a modality-alignment subspace projection.Source: C. Dual-Linear with Batch… · pp. 4–5
  • The added linear layer projects modality-specific features before the final classifier, which produces identity classification outputs.The authors contrast this with a single-linear classifier that must average multiple same-identity modality patterns.Source: C. Dual-Linear with Batch… · p. 5
  • A shared one-dimensional batch-normalization layer is applied to the common embedding space to balance losses and stabilize magnitude.Its scaling and shift parameters recalibrate embedding channels; the figure caption identifies last-layer outputs as final classification results.Source: C. Dual-Linear with Batch… · p. 5, Figure 5 · p. 5

D. Learning with Bi-directional Tri-constrained Top-Push Ranking Loss.

BTTR extends a bidirectional ranking loss with cross-modality, intra-modality, and inter-modality constraints. The section first revisits the bidirectional visible-to-infrared and infrared-to-visible ranking setup.

D. Learning with Bi-directional Tri-constrained Top-Push Ranking Loss.

  • BTTR adds an inter-modality constraint to cross-modality and intra-modality regularizers in the earlier bidirectional ranking loss.The authors present it as an improved version of the loss in.Source: D. Learning with Bi-directional… · p. 5

1) Revisit Bi-directional Ranking Loss.:

  • The bidirectional ranking setup uses visible anchors with infrared positives and negatives, and infrared anchors with visible positives and negatives.The two directions are represented by visible-anchor and infrared-anchor triplet terms, with a margin parameter.Source: 1) Revisit Bi-directional Ranking Loss · p. 6

2) Bi-directional Tri-constrained Top-Push Ranking Loss.:

BTTR combines cross-modality, intra-modality, and inter-modality top-push ranking constraints to shape discriminative features across grayscale and infrared images. Its inter-modality constraint further relates cross- and intra-modality distances.

2) Bi-directional Tri-constrained Top-Push Ranking Loss.:

  • BTTR uses three-fold triplet constraints: cross-modality, intra-modality, and inter-modality top-push ranking losses.The combined objective also includes identity loss, with tradeoff parameters balancing the losses.Source: 2) Bi-directional Tri-constrained… · pp. 6–7
  • The cross-modality loss uses positive grayscale-infrared pairs and negative infrared-grayscale pairs to enhance feature discriminability.It is designed for the grayscale-infrared matching protocol used at testing.Source: 2) Bi-directional Tri-constrained… · p. 6
  • The intra-modality loss reduces distances between same-person images and enlarges distances between different-person images within each modality.It ensures the closest intra-modality negative is farther away than the farthest intra-modality positive.Source: 2) Bi-directional Tri-constrained… · p. 6
  • The inter-modality regularizer relates cross-modality and intra-modality distances to encourage modality-invariant features.For a grayscale anchor, the method mines the farthest grayscale-infrared positive pair and the closest grayscale-grayscale negative pair.Source: 2) Bi-directional Tri-constrained… · p. 6
  • The added inter-modality constraint makes same-label samples closer and different-label samples farther regardless of modality.The authors state this also mitigates modality biases while retaining top-push ranking's representation and discriminative ability.Source: Figure 6 · p. 6, 2) Bi-directional Tri-constrained… · p. 7

IV. EXPERIMENTS

The experiments evaluate the proposed approach through component ablations and comparisons with state-of-the-art methods, asking how components contribute and how the framework compares with peer methods.

IV. EXPERIMENTS

  • The evaluation includes comparisons between the proposed framework and state-of-the-art methods.The section frames this as assessing the framework against peer methods.Source: §IV · p. 7
  • The experiments use ablation studies to assess how individual components contribute to performance.Source: §IV · p. 7
  • The section describes experimental settings as part of evaluating the proposed approach.Source: §IV · p. 7

A. Experimental Settings

Experiments on SYSU-MM01 and RegDB evaluate the proposed components, with SYSU-MM01 ablations showing gains from grayscale augmentation, ID embedding, and BTTR. The study also specifies the datasets, evaluation protocol, and weighting parameters.

A. Experimental Settings

  • SYSU-MM01 and RegDB are evaluated using CMC rank-n accuracy and mAP, with cross-modality query and gallery sets.SYSU-MM01 uses all-search and indoor-search modes; RegDB uses visible-to-thermal query-to-gallery evaluation in the supplied protocol.Source: A. Experimental Settings · p. 7

B. Ablation Study

  • Grayscale augmentation raises rank-1 accuracy from 38.81% to 43.70% and mAP from 35.30% to 40.36% versus the RGB-infrared baseline.The grayscale strategy also significantly outperforms GAN-based methods in both SYSU-MM01 search modes.Source: B. Ablation Study · p. 8, Table II · p. 8
  • 60.45% rank-1 and 53.87% mAP are reported when batch normalization and dual-linear ID embedding are combined.Batch normalization improves the baseline by 9%–11%, and adding a fully connected layer increases rank-1 and mAP by 3%–4%.Source: B. Ablation Study · p. 8, Table III · p. 8
  • The cross-modality regularizer improves performance by 2%–3%, while intra- and inter-modality regularizers each yield smaller, consistent improvements of 1%–2%.The authors describe the latter two as relatively weaker constraints for cross-modality metric learning.Source: B. Ablation Study · pp. 8–9, Table IV · p. 8
  • 65.74% rank-1 accuracy and 60.83% mAP with BTTR, up from 60.45% and 53.87% (+5.29% and +6.96%), respectively.The authors attribute BTTR’s improvement over BDTR to its inter-modality regularizer.Source: Table IV · p. 8, B. Ablation Study · p. 9
  • On SYSU-MM01, the reported BTTR weighting setting is λ1 = 0.1 and λ2 = 0.5.λ1 weights intra-modality regularization and λ2 weights inter-modality regularization.Source: B. Ablation Study · p. 9

2) Margin Parameters:

The margin analysis varies δ1, δ2, and δ3 for the cross-, intra-, and inter-modality terms on SYSU-MM01. With δ1 = 0.5 and δ2 = 0.1, performance peaks at δ3 = 0.9.

2) Margin Parameters:

  • With δ1 = 0.5 and δ2 = 0.1, performance peaks at δ3 = 0.9; the authors link the larger inter-modality margin to faster convergence.They explain that cross-modality positive and same-modality negative pairs can have very close distances.Source: 2) Margin Parameters · p. 9

C. Comparison with State-of-the-Arts

SFANet reports strong state-of-the-art comparisons on SYSU-MM01 and RegDB, while grayscale-spectrum augmentation is presented as an alternative to GAN-based image translation. Its reported margins vary by dataset, query setting, and metric.

1) Comparisons on SYSU-MM01 Dataset:

  • 65.74% rank-1 accuracy on SYSU-MM01 is 10.99 percentage points above DDAG; 60.83% mAP exceeds most methods but remains below cm-SSFT.The authors attribute the cm-SSFT advantage in part to its use of both shared and sample-specific information.Source: Table V · p. 10, 1) Comparisons on SYSU-MM01 Dataset · p. 10

2) Comparisons on RegDB Dataset:

  • On RegDB thermal-to-visible evaluation, SFANet exceeds the best rank-1 accuracy by 6.25% and the best mAP by 2.27%; visible-to-thermal rank-1 and mAP rise from DDAG’s 69.34% and 63.46% to 76.31% and 68.00%.The reported comparisons cover both query settings.Source: Table VI · p. 10, 2) Comparisons on RegDB Dataset · p. 10

A. Analysis of feature augmentation strategy via grayscalespretrum images.

  • GAN-generated infrared images show indistinctness, distorted body structure, and added noise, which can enlarge intra-person discrepancy despite reducing pixel-level modality discrepancy.The authors illustrate these issues with CycleGAN samples.Source: A. Analysis of feature augmentation… · p. 11, Figure 9 · p. 11
  • Grayscale-spectrum images replace RGB inputs for feature learning, preserve visible-image structure and semantic cues, and approximate infrared style; grayscale-infrared learning outperforms RGB-infrared training in Table II.The authors describe the remaining grayscale-to-infrared difference as appearance discrepancy addressed by feature-level constraints.Source: A. Analysis of feature augmentation… · p. 11, Table VII · p. 11

B. Analysis of the sharable dual-path information-preserving architecture.

The sharable dual-path architecture keeps early convolutional layers modality-specific and shares later layers to retain spatial structure while learning shared features. On SYSU-MM01, it outperforms the compared network structures.

B. Analysis of the sharable dual-path information-preserving architecture.

  • Two convolutional layers learn modality-specific feature maps, while conv3–conv5 are shared to capture modality-shared information and preserve person spatial structure.The authors contrast this with separately optimized two-stream blocks and fully shared one-stream structures.Source: B. Analysis of the sharable dual-path… · p. 11
  • The sharable dual-path architecture improves on the second-best MSTN by 2.57% in rank-1 accuracy and 4.54% in mAP.One-stream and MSTN backbones also improve performance, but less than the proposed structure.Source: B. Analysis of the sharable dual-path… · p. 11

C. Analysis of the Batch Normalization Operation.

The section examines batch normalization’s role in reconciling identity and BTTR objectives, then reports triplet-variant, sampling, visualization, and overall benchmark findings.

C. Analysis of the Batch Normalization Operation.

  • Batch normalization suppresses the inconsistency between identity and BTTR objectives, smooths the loss curve, and improves optimization performance.The authors describe the two losses as using different embedding-space objectives; BN recalibrates channels and helps stabilize their interaction.Source: C. Analysis of the Batch Normalization… · p. 12

D. Comparsion with Other Triplet Variants.

  • Integrating the proposed triplet reformulations improves performance, and BTTR with hard mining achieves higher performance by explicitly optimizing hard cross- and intra-modality samples.The comparison covers soft-margin, weighted-regularized triplet, and lifted-embedding losses on SYSU-MM01.Source: D. Comparsion with Other Triplet… · p. 12, Table VIII · p. 12

E. Batch sampling strategy

  • Each training iteration samples N identities and one visible plus one infrared image per identity, yielding 2*N images for the two-stream network.The random sampling strategy is intended to traverse possible assemblies toward a global optimum.Source: E. Batch sampling strategy · p. 13

F. Visualization of results

  • SYSU-MM01 retrieval visualizations include infrared-to-visible and visible-to-infrared queries; the authors report that structural cues such as bags, stripes, and logos support correct top-ranked retrievals despite some errors.Five randomly selected queries per setting show the top ten cross-modality results.Source: F. Visualization of results · p. 13

VI. CONCLUSION

  • Across two standard benchmarks, SFANet learns modality-invariant representations and outperforms state-of-the-art methods.The framework combines grayscale-spectrum augmentation, parameter sharing, dual-linear batch-normalized ID embedding, and BTTR.Source: §VI · p. 13

Related papers

Loading 2102.12137v1…