PaperlayerView exact revision on arXiv

Source-linked AI summary

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, Rainer Stiefelhagen

arXiv:2203.04838v5cs.CVcs.ROeess.IV

TL;DR

RGB-X semantic segmentation lacks a unified model that generalizes across sensors with differing characteristics and uncertainties. CMX addresses this gap with interactive feature rectification and fusion, and demonstrates broad performance across five complementary modalities, including a new RGB-Event benchmark.

  • Problem

    A modality-agnostic RGB-X segmentation model remains unresolved because supplementary sensors differ in characteristics and may contain uncertainties.

  • Method

    CMX uses cross-modal feature rectification to calibrate paired RGB-X features and feature fusion to exchange long-range context before mixing.

  • Results

    CMX generalizes across RGB-Depth, Thermal, Polarization, Event, and LiDAR segmentation, outperforming specialized architectures and setting new state-of-the-art on RGB-Event.

  • Takeaways & Limitations

    The results support comprehensive cross-modal interaction as a unified strategy for diverse RGB-X semantic segmentation.

Abstract

from arXiv · show

Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality (X-modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for modality fusion. Specifically, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate bi-modal features by leveraging the features from one modality to rectify the features of the other modality. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to perform sufficient exchange of long-range contexts before mixing. To verify CMX, for the first time, we unify five modalities complementary to RGB, i.e., depth, thermal, polarization, event, and LiDAR. Extensive experiments show that CMX generalizes well to diverse multi-modal fusion, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available at https://github.com/huaaaliu/RGBX_Semantic_Segmentation.

I. INTRODUCTION

CMX addresses the difficulty of unifying RGB-X semantic segmentation across diverse sensing modalities by using comprehensive cross-modal interaction. It combines cross-modal feature rectification with feature fusion and evaluates one framework across five modality combinations.

  • Motivation: RGB-X segmentation supplements RGB with complementary sensing information, including depth, thermal, polarization, event, and LiDAR data.These modalities provide geometric, infrared, specular-scene, dynamic-scene, or spatial information.
  • Motivation: Existing input- and feature-fusion methods are typically tailored to a single modality pair and generalize poorly across sensing combinations.Prior global interaction strategies are also reported to be less effective across different sensing data.
  • Method: CMX uses a two-stream architecture with Cross-Modal Feature Rectification and Feature Fusion modules for interactive RGB-X fusion.CM-FRM calibrates bi-modal features using spatial- and channel-wise correlations, while FFM exchanges long-range context before combining features.
  • Evaluation: CMX is evaluated on five RGB-X tasks spanning RGB-Depth, RGB-Thermal, RGB-Polarization, RGB-Event, and RGB-LiDAR segmentation.The evaluation involves nine datasets and includes a newly established RGB-Event benchmark based on EventScape.
  • Results: CMX outperforms specialized architectures across five segmentation tasks and achieves new state-of-the-art performance on the RGB-Event benchmark.The experiments also examine CNN- and Transformer-based architectures and representations for polarization and event data.

II. RELATED WORK

Prior multi-modal segmentation methods use modality-specific fusion designs, while CMX targets broader cross-modal generalization. The related work motivates interactive fusion for diverse sensing data and introduces the CMX framework overview.

  • Semantic Segmentation: Dense semantic segmentation uses multi-scale feature methods, contrastive learning, prototype retrieval, and transformer self-attention to improve scene parsing.These approaches address feature representation, class separation, semantic retrieval, and long-range dependencies.
  • Multi-modal Semantic Segmentation: RGB-Depth and RGB-Thermal segmentation are widely studied, while polarization, event, and LiDAR cues support perception under challenging driving conditions.The related work frames multi-modality as useful for reliable scene understanding in real-world settings.
  • Multi-modal Semantic Segmentation: Existing fusion methods commonly embed cross-modal information in modality-specific layers or operators, limiting transfer beyond the modality for which they were designed.Multi-task frameworks are also discussed as another line of multi-modal work.
  • CMX Motivation: Simple global channel-wise interaction performs less satisfactorily across sensing data, motivating comprehensive cross-modal interactions for RGB-X segmentation.ACNet and SA-Gate, designed for RGB-D segmentation, are cited as examples that perform less satisfactorily in RGB-T parsing.
  • CMX Framework: CMX uses parallel RGB and X-modal branches for RGB-Depth, Thermal, Polarization, Event, and LiDAR inputs.Its framework overview presents the branches and their interactive processing.

III. PROPOSED FRAMEWORK: CMX

CMX uses parallel RGB and X-modal branches with interactive feature extraction, rectifying noisy features across modalities before fusion. Its CM-FRM combines channel-wise and spatial-wise calibration to improve cross-modal feature interaction.

  • Framework Overview: CMX processes RGB and supplementary inputs through two parallel but interactive branches for RGB-X semantic segmentation.
  • Cross-Modal Feature Rectification: CM-FRM uses features from each modality to rectify the other modality’s noisy measurements.
  • Channel-wise Feature Rectification: Channel-wise rectification learns global weights from pooled RGB and X features, then applies them reciprocally across modalities.
  • Spatial-wise Feature Rectification: Spatial-wise rectification embeds concatenated bimodal features into two spatial weight maps for local calibration.
  • Rectified Feature Output: The rectified RGB and X features combine original features with channel-wise and spatial-wise corrections before entering the next stage.The correction weights λ_C and λ_S are both set to 0.5 by default.

C. Feature Fusion

The Feature Fusion Module exchanges information between RGB and X branches through cross-attention before producing fused features. Its efficient global-context formulation avoids the full N×N attention map.

  • Information Exchange Stage: FFM first maintains two modal branches and exchanges information symmetrically through cross-attention between interactive vectors.Features are flattened from H×W positions into N tokens before interaction.
  • Cross-Attention: The cross-attention mechanism forms global context vectors from each modality’s keys and values, then uses them to attend to the other modality.This replaces the traditional attention map of size R^N×N with compact context vectors.

{ U }_{RGB

After cross-modal exchange, FFM combines attended and residual features and then merges both modal paths into a single feature representation.

  • Information Exchange Stage: FFM concatenates each attended result with its residual vector, applies linear embedding, and restores the feature map to H×W×C.
  • Fusion Stage: The fusion stage merges the two paths through 1×1 convolution-based channel embedding and adds depth-wise convolution to exploit surrounding areas.

D. Multi-modal Data Representations

CMX represents diverse supplementary sensors in modality-specific forms while retaining a common RGB-X segmentation framework. The representations cover depth, thermal, polarization, event, and LiDAR inputs.

  • RGB-Depth: RGB-Depth uses HHA encoding to represent depth geometry through horizontal disparity, height above ground, and angle.
  • RGB-Thermal: RGB-Thermal uses the single-channel infrared image copied into three channels to match the backbone input.
  • RGB-Polarization: RGB-Polarization studies DoLP and AoLP, derived from Stokes vectors describing the polarization state of light.
  • RGB-Polarization: The experiments compare monochromatic and trichromatic polarization representations, with trichromatic cues computed separately for RGB channels.
  • RGB-LiDAR: LiDAR points are projected from world coordinates into 2D image coordinates using rotation, translation, and camera projection parameters.The rotation and translation matrices are obtained from KITTI-360.

IV. EXPERIMENT DATASETS AND SETUPS

CMX is evaluated across diverse RGB-X semantic segmentation datasets, including five RGB-Depth datasets and RGB-Thermal, RGB-Polarization, RGB-Event, and RGB-LiDAR combinations.

  • The evaluation uses five RGB-Depth datasets plus RGB-Thermal, RGB-Polarization, RGB-Event, and RGB-LiDAR datasets.
  • NYU Depth V2 contains 1,449 RGB-D images, 40 semantic categories, and 795/654 training/testing images.
  • SUN-RGBD contains 10,335 RGB-D images with 37 classes and 5,285/5,050 training/testing samples.
  • Stanford2D3D, ScanNetV2, and Cityscapes provide additional RGB-D settings with 13, 20, and urban road-scene categories, respectively.
  • KITTI-360 supplies suburban driving scenes with 49,004/12,276 training/validation images and 19 semantic classes.

B. Implementation Details

Experiments use standardized transformer-based training settings and compare CMX across indoor, outdoor, thermal, and RGB-Depth benchmarks.

  • Training uses an ImageNet-pretrained MiT encoder, an MLP decoder with embedding dimension 512, AdamW, cross-entropy loss, and polynomial learning-rate scheduling.
  • The experiments cover RGB-Depth, RGB-Thermal, and multiple indoor and outdoor semantic segmentation benchmarks.
  • On NYU Depth V2, CMX with MiT-B2 attains 54.4% mIoU and exceeds previous methods.
  • Table II reports results on five RGB-Depth datasets, with pixel accuracy and multi-scale testing identified by the table notation.
  • On the RGB-T benchmark, CMX improves daytime mIoU by 2.7%∼3.1% over RGB-only baselines.

C. Results on RGB-Polarization Dataset

CMX demonstrates strong cross-modal performance on polarization, event, and LiDAR combinations, while representation and backbone analyses examine its generality.

  • C. Results on RGB-Polarization Dataset: CMX outperforms the previous best RGB-Polarization method by >6.0% in mIoU on ZJU-RGB-P.
  • C. Results on RGB-Polarization Dataset: Polarization-aware classes improve over RGB-only MiT-B2, including glass by >8.0% and car by >2.5% IoU.
  • D. Results on RGB-Event Dataset: On RGB-Event segmentation, CMX with MiT-B4 reaches 64.28% mIoU and sets the state of the art among more than 10 methods.
  • D. Results on RGB-Event Dataset: For RGB-Event, CMX improves vehicle, pedestrian, and traffic-light IoU by +2.1%, +11.7%, and +7.0%, respectively.
  • CMX improves mIoU over RGB-only DeepLabV3+, Swin-s, and Swin-b by +1.26%, +8.37%, and +7.90%, respectively.
  • E. Results on RGB-LiDAR Dataset: On RGB-LiDAR segmentation, CMX reaches 64.31% mIoU, a +9.76% gain over MiT-B2 TokenFusion.

F. Ablation Study

Ablations show that comprehensive cross-modal rectification and feature exchange improve segmentation, while efficiency and qualitative results reveal practical operating trade-offs.

  • RGB-only Baseline and CMX: With the same MiT-B2 backbone, CMX improves over RGB-only by +6.1%, +0.6%, +5.0%, +2.6%, +3.2%, and +3.0% across six datasets.
  • Ablation of CM-FRM: Using only channel-wise or spatial-wise rectification produces sub-optimal accuracy compared with the complete CM-FRM.
  • Ablation of FFM: Removing information exchange in FFM variants is less effective than the complete module, supporting cross-attention across feature levels.
  • Computational Complexity: CMX with MiT-B2 has similar parameters and lower FLOPs than SA-Gate while achieving higher mIoU; MiT-B4 reaches 56.0% mIoU.
  • Visualization of Segmentation Results: Qualitative results show modality-specific benefits: depth helps low-texture recognition, thermal clarifies nighttime boundaries, and event data improves moving-object segmentation.
  • Visualization of Segmentation Results: LiDAR supplies spatial cues for wall recognition, while CM-FRM reduces the impact of LiDAR noise in truck-glass regions.

VI. CONCLUSION

CMX is a transformer-based cross-modal fusion architecture designed to generalize across diverse RGB-X sensing combinations. It combines feature rectification and fusion modules, establishes an RGB-Event benchmark, and reports state-of-the-art results across nine benchmarks.

  • CMX is a universal transformer-based architecture for cross-modal RGB-X semantic segmentation across diverse sensing combinations.
  • Visualization results compare RGB-only and RGB-X methods using the same backbone across five complementary modalities.
  • CM-FRM performs channel- and spatial-wise feature rectification, while FFM uses cross-attention and mixed channel embedding for global information exchange.
  • CMX establishes an RGB-Event semantic segmentation benchmark to assess generalization to dense-sparse data fusion.
  • CMX sets the state of the art on nine benchmarks spanning RGB-D, RGB-Thermal, RGB-Polarization, RGB-Event, and RGB-LiDAR combinations.

APPENDIX A MORE IMPLEMENTATION DETAILS

The appendix details training settings and visual analyses for CMX across RGB-X segmentation tasks. Results emphasize comparisons with RGB-only baselines, feature rectification, and performance under noisy depth measurements.

  • Experiments use PyTorch, a poly learning-rate schedule, 10 warm-up epochs, and dataset-specific backbones, hardware, image sizes, and training durations.
  • Segmentation results on the Cityscapes dataset: Cityscapes visualizations compare RGB-only and RGB-X SegFormer-B4 results through ground-truth difference maps and HHA depth encoding.
  • Segmentation results on the Cityscapes dataset: Despite noisy depth measurements, CMX benefits from HHA-encoded depth through cross-modal feature rectification and fusion.
  • Failure-case visualizations cover RGB-Depth, RGB-Thermal, RGB-Polarization, and RGB-Event segmentation using matched RGB and RGB-X backbones.
  • Layer-1 visualizations show extracted features, rectified features, and their difference map.
  • The RGB stream is rectified toward informative features and complementary discontinuities for more accurate semantic understanding.
Loading 2203.04838v5…