PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Aware Interaction Network for RGB-T Semantic Segmentation

Ying Lv, Zhi Liu, Gongyang Li

arXiv:2401.01624v1cs.CV

TL;DR

RGB-T segmentation methods have not effectively explored complementary relationships between modalities during multilevel interaction. CAINet constructs an interaction space with context-aware modules and auxiliary supervision, and is reported to achieve state-of-the-art performance on MFNet and PST900. On PST900, it outperforms LASNet by 0.34% in mIoU.

  • Problem

    Existing RGB-T methods do not effectively explore complementary relationships between modalities during multilevel interaction, despite RGB imagery’s difficulty in unfavorable lighting.

    Source: §I · p. 1

  • Method

    CAINet combines direct and feedback fusion in an interaction space, using CACR, GCM, DA, and auxiliary supervision for multilevel feature interaction and refinement.

    Source: §I · p. 2

  • Results

    CAINet achieves state-of-the-art performance on MFNet and PST900; on PST900, it outperforms LASNet by 0.34% in mIoU.

    Source: §I · p. 2, 2) Evaluation on the PST900 Dataset · p. 8

  • Takeaways & Limitations

    The authors report that cooperation between CACR and GCM fully utilizes long-range dependencies and multimodal complementary information.

    Source: §I · p. 2

Abstract

from arXiv · show

RGB-T semantic segmentation is a key technique for autonomous driving scenes understanding. For the existing RGB-T semantic segmentation methods, however, the effective exploration of the complementary relationship between different modalities is not implemented in the information interaction between multiple levels. To address such an issue, the Context-Aware Interaction Network (CAINet) is proposed for RGB-T semantic segmentation, which constructs interaction space to exploit auxiliary tasks and global context for explicitly guided learning. Specifically, we propose a Context-Aware Complementary Reasoning (CACR) module aimed at establishing the complementary relationship between multimodal features with the long-term context in both spatial and channel dimensions. Further, considering the importance of global contextual and detailed information, we propose the Global Context Modeling (GCM) module and Detail Aggregation (DA) module, and we introduce specific auxiliary supervision to explicitly guide the context interaction and refine the segmentation map. Extensive experiments on two benchmark datasets of MFNet and PST900 demonstrate that the proposed CAINet achieves state-of-the-art performance. The code is available at https://github.com/YingLv1106/CAINet.

I. INTRODUCTION

RGB-T segmentation addresses RGB imagery’s difficulty in challenging lighting, but existing fusion methods leave multilevel complementary interaction and explicit guidance unresolved. CAINet combines direct and feedback fusion with context-aware modules and auxiliary supervision.

  • RGB imagery can fail to identify pedestrians and vehicles in unfavorable lighting, motivating thermal input and effective cross-modal fusion.Existing methods also leave complementary relationships between modalities underexplored across levels, with no explicit guidance for multilevel feature interaction.Source: §I · pp. 1–2
  • Direct fusion often repeats one fusion module across levels, while feedback fusion risks an information bottleneck and still lacks effective multilevel complementary reasoning.The resulting multilevel fused-feature representations can remain ambiguous and uncertain.Source: §I · pp. 1–2
  • CAINet combines advantages of direct and feedback fusion in a new paradigm for exploring complementary relationships across modalities and multiple feature levels.It also explores low-level features for boundary detail and applies different auxiliary supervision to improve representations at multiple levels.Source: §I · p. 2, Figure 2 · p. 2
  • CACR models multimodal complementarity and long-range dependencies across spatial and channel dimensions; GCM guides multilevel interactions with global context, and DA aggregates detail for refinement.Specific auxiliary supervision guides feature representations at each level; the paper also introduces residual learning to retain high-level global context.Source: §I · p. 2
  • CAINet is reported to achieve state-of-the-art performance on the MFNet and PST900 datasets.Source: §I · p. 2

II. RELATED WORK

Related work covers unimodal RGB segmentation and RGB-T methods, progressing from feature fusion toward interaction while identifying gaps in multilevel complementary reasoning.

A. RGB Semantic Segmentation

  • RGB segmentation work includes multiscale context, attention, and transformer-based methods, while visible-light systems can struggle in adverse weather and low-light settings.Thermal imaging is presented as a needed supplement in those settings.Source: A. RGB Semantic Segmentation · pp. 2–3

B. RGB-T Semantic Segmentation

  • Early RGB-T methods such as MFNet, RTFNet, and FuseSeg use simple fusion operations that may ignore cross-modal differences and introduce information redundancy.Later work explores specialized fusion, multiscale features, attention, and boundary-assisted supervision.Source: B. RGB-T Semantic Segmentation · p. 3
  • Interaction methods address feature exchange, but prior approaches are described as insufficient in spatial detail or multimodal complementarity.CAINet combines direct and feedback fusion to reason about complementarity at multiple levels, with CACR, GCM, DA, and auxiliary supervision.Source: B. RGB-T Semantic Segmentation · p. 3

III. PROPOSED METHOD

CAINet combines RGB and thermal encoders with multilevel interaction, global context, and detail aggregation. Its CACR module reasons over multimodal complementarity and long-range context in spatial and channel dimensions.

A. Architecture Overview

  • CAINet uses RGB and thermal encoders, CACR, GCM, three-decoder supervision, DA, and residual learning with auxiliary tasks; P4 is the sole prediction retained at inference.The other supervised branches can be removed without added inference cost.Source: Figure 3 · p. 4

B. Context-Aware Complementary Reasoning

  • CACR constructs an interaction space to reason about complementary multimodal features and exchange long-range context across spatial and channel dimensions.It operates at three high-level stages, producing complementary features CR3, CR4, and CR5.Source: B. Context-Aware Complementary Reasoning · p. 4
  • CACR converts input features into a feature set representing spatial and attribute information, then builds regional similarity correlations in the interaction space.The interaction-space correlation captures relationships between regions and supports reasoning over the complete multimodal feature map.Source: B. Context-Aware Complementary Reasoning · p. 5
  • Two fully connected layers infer along different dimensions to adaptively assign interaction-space correlations to regional features, enabling multimodal features to exchange long-range context.The resulting similarity correlations let features communicate according to their complementary relationship.Source: B. Context-Aware Complementary Reasoning · pp. 5–6

C. Global Context Modeling

GCM performs higher-level global reasoning over complementary multilevel features. It produces global context features that guide feature interactions and joint auxiliary supervision with ARLM.

C. Global Context Modeling

  • GCM performs higher-level global reasoning over complementary relationships among multilevel features and yields global context feature G for guidance with ARLM.The module is presented as going beyond global-context paradigms such as pooling and self-attention.Source: C. Global Context Modeling · p. 6
  • GCM concatenates three high-level complementary features into semantic guidance CRc before modeling their relationships.The passage identifies Conv2D, channel-wise Softmax β, and Conv1D as operations in the relationship-modeling process.Source: C. Global Context Modeling · p. 6
  • Global context G guides auxiliary and target supervision with ARLM, while its decoder output is a coarse segmentation map that may lose detail information.The method therefore follows this interaction with DA for feature-map refinement.Source: C. Global Context Modeling · p. 6

D. Detail Aggregation Module

DA combines low-level RGB and thermal features to use boundary and texture detail, then refines the final segmentation result through subsequent ARLM steps.

D. Detail Aggregation Module

  • Low-level features contain boundary information and abundant texture; DA uses them with auxiliary boundary-detail supervision to promote segmentation performance.The passage describes these features as complementary to higher-level representations.Source: D. Detail Aggregation Module · p. 6
  • In the AD module, RGB and thermal features are concatenated along channels and combined with depthwise separable convolution to reduce computational costs; thermal features use global max pooling and a 7 × 7 convolution for spatial information.A channel-attention layer obtains detail-aggregation features.Source: D. Detail Aggregation Module · p. 6
  • After detail-aggregation features are obtained, subsequent ARLM steps produce boundary and binary maps to refine the final semantic segmentation result.The channel-attention layer is used to obtain the detail-aggregation features.Source: D. Detail Aggregation Module · p. 6

E. Residual Learning with Auxiliary and Target Tasks

Residual learning uses global context to guide stepwise interactions among complementary multimodal features and produces segmentation, attention, binary, and boundary predictions. Auxiliary and target losses supervise these outputs and the backbone decoders.

E. Residual Learning with Auxiliary and Target Tasks

  • Global context G guides multi-level complementary-feature interactions step by step, producing segmentation and attention predictions alongside binary and boundary maps.ARLM1 takes G and complementary features CR5 and CR4; subsequent steps produce P2, Att2, P3, binary, P4, and boundary predictions.Source: E. Residual Learning with Auxiliary… · p. 6
  • Attention-map supervision uses a target Q derived from the binary-map ground truth by corrosion, expansion, and Gaussian blurring.Its loss combines mean squared error with a modified linear correlation coefficient metric.Source: E. Residual Learning with Auxiliary… · p. 6
  • Lovász-Softmax supervises P1 and P2, while weighted cross-entropy supervises P3 and P4; the four segmentation losses form Ltarget.Binary and boundary predictions also receive weighted binary-cross-entropy losses, and coarse decoder maps use ordinary cross-entropy.Source: E. Residual Learning with Auxiliary… · pp. 6–7

A. Experimental Protocol

Experiments use MFNet and PST900, RGB-T datasets with different class counts, image resolutions, and train/test splits. Performance is evaluated using mean accuracy and mean intersection over union.

1) Datasets:

  • MFNet contains 1,569 image pairs across nine categories at 480×640 resolution; its split follows the dataset's proposed scheme.The pairs include 820 daytime and 749 nighttime images; PST900 has 894 aligned pairs across five categories at 1280×720, split into 597 training and 288 test pairs.Source: 1) Datasets · pp. 7–8

2) Evaluation Metrics:

  • Mean accuracy (mAcc) and mean intersection over union (mIoU) are the two metrics used to evaluate semantic-segmentation performance.The passage defines each as an average across classes, using class-wise pixel accuracy and intersection-over-union, respectively.Source: 2) Evaluation Metrics · p. 8

3) Implementation Details:

CAINet is trained in stages from pretrained MobileNet-V2 branches to joint training with residual learning. It reports leading MFNet mIoU and strong PST900 rankings, with the authors also reporting parameter and FLOP efficiency.

3) Implementation Details:

  • Training proceeds from separately trained RGB and thermal branches to the GCM decoder, then joint training with the residual-learning branch.The model starts from ImageNet-pretrained MobileNet-V2; training uses batch size 8, learning rate 5×10−4, Adam, and early termination.Source: 3) Implementation Details · p. 8

1) Evaluation on MFNet dataset:

  • CAINet achieves 58.6% mIoU on MFNet, 0.4 percentage points above the second-best methods CCAFFMNet and CMX-SegF-B2 at 58.2%.It also exceeds the second-best results in bike and bump categories by the reported mAcc and mIoU margins.Source: 1) Evaluation on MFNet dataset · p. 8, Table II · p. 8
  • For small, obscured objects, CAINet may segment a contiguous target shape, and some roadblocks may go undetected.The authors also describe favorable results under high background lighting and attribute thermal imaging's consistent heat representation across lighting conditions.Source: 1) Evaluation on MFNet dataset · p. 8

2) Evaluation on the PST900 Dataset:

3) Computational Complexity:

The ablations compare CAINet modules individually and in combination on MFNet. The full model reaches 58.6%, while the authors report gains for module combinations over the baseline and single-module variants.

3) Computational Complexity:

  • The computational-complexity analysis compares model parameter counts and floating-point operations across methods.The supplied passage does not state the numerical comparison results.Source: 3) Computational Complexity · p. 9

1) The Individual and Joint Contributions of Four Modules:

  • Adding DA, CACR, or GCM alongside ARLM improves performance considerably over the baseline.ARLM is required in this framework; the other modules are replaced with ASPP modules in the deletion variants.Source: 1) The Individual and Joint… · pp. 10–11
  • The complete CAINet reaches 58.6% in the module ablation, while the baseline reaches 51.9% mIoU, 6.7 percentage points lower.The table assesses individual and joint contributions of the four CAINet modules.Source: 1) The Individual and Joint… · pp. 10–11, Table IV · p. 10
  • Among the tested two-module combinations, CACR with GCM performs better than the other two pairings; the authors link this to explicit auxiliary supervision with attention maps.The three combinations are CACR+GCM, DA+GCM, and DA+CACR.Source: 1) The Individual and Joint… · p. 11

2) The Effectiveness of Auxiliary Supervision and Target Supervision:

The paper reports CAINet’s overall benchmark comparison, evaluates the supervision design and tests generalization to RGB-D data. It also identifies model size and computational complexity as areas for improvement.

2) The Effectiveness of Auxiliary Supervision and Target Supervision:

  • Joint supervision yields CAINet’s final segmentation results, reported as up to 58.6% outperforming other results.Discarding one or all supervision was detrimental to final segmentation performance.Source: 2) The Effectiveness of Auxiliary… · p. 11

D. Generalization to RGB-D Data

V. CONCLUSION

  • CAINet outperforms state-of-the-art methods on MFNet and PST900, while parameter count and computational complexity remain improvement targets.The authors say a lighter model is needed for intended mobile-device deployment.Source: §V · pp. 11–12
Loading 2401.01624v1…