PaperlayerView exact revision on arXiv

Source-linked AI summary

Delivering Arbitrary-Modal Semantic Segmentation

Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon Reiß, Kunyu Peng, Haodong Fu, Kaiwei Wang, Rainer Stiefelhagen

arXiv:2303.01480v1cs.CV

TL;DR

Arbitrary-modal semantic segmentation remains underexplored despite the potential of multimodal fusion to improve robustness. The paper introduces the DELIVER benchmark and CMNeXt, whose Self-Query Hub and Parallel Pooling Mixer support flexible modality fusion. CMNeXt achieves state-of-the-art performance across six datasets and scales from 1 to 81 modalities, including 66.30% mIoU on DELIVER with a +9.10% gain over the RGB-only baseline.

  • Problem

    Most existing multimodal segmentation methods and benchmarks focus on predefined sensor pairs, limiting arbitrary-modal fusion.

  • Method

    CMNeXt combines an asymmetric Hub2Fuse architecture with a Self-Query Hub for informative feature selection and PPX for harvesting discriminative auxiliary cues.

  • Results

    CMNeXt achieves state-of-the-art performance on six datasets, scales from 1 to 81 modalities, and reaches 66.30% mIoU on DELIVER with a +9.10% gain over the RGB-only baseline.

  • Takeaways & Limitations

    The benchmark and model support arbitrary-modal segmentation under adverse weather and partial sensor failures.

Abstract

from arXiv · show

Multimodal fusion can make semantic segmentation more robust. However, fusing an arbitrary number of modalities remains underexplored. To delve into this problem, we create the DeLiVER arbitrary-modal segmentation benchmark, covering Depth, LiDAR, multiple Views, Events, and RGB. Aside from this, we provide this dataset in four severe weather conditions as well as five sensor failure cases to exploit modal complementarity and resolve partial outages. To make this possible, we present the arbitrary cross-modal segmentation model CMNeXt. It encompasses a Self-Query Hub (SQ-Hub) designed to extract effective information from any modality for subsequent fusion with the RGB representation and adds only negligible amounts of parameters (~0.01M) per additional modality. On top, to efficiently and flexibly harvest discriminative cues from the auxiliary modalities, we introduce the simple Parallel Pooling Mixer (PPX). With extensive experiments on a total of six benchmarks, our CMNeXt achieves state-of-the-art performance on the DeLiVER, KITTI-360, MFNet, NYU Depth V2, UrbanLF, and MCubeS datasets, allowing to scale from 1 to 81 modalities. On the freshly collected DeLiVER, the quad-modal CMNeXt reaches up to 66.30% in mIoU with a +9.10% gain as compared to the mono-modal baseline. The DeLiVER dataset and our code are at: https://jamycheung.github.io/DELIVER.html.

1. Introduction

The paper addresses arbitrary-modal semantic segmentation by introducing the DELIVER benchmark and CMNeXt, which selects and fuses complementary information across changing modality sets. Experiments cover adverse conditions, sensor failures, and six benchmarks.

  • Most existing segmentation methods and benchmarks target specific sensor pairs rather than arbitrary modality combinations.
  • Adding modalities can improve segmentation through more diverse complementary information, as shown across RGB, polarization, near-infrared, event, LiDAR, depth, and light-field settings.
  • DELIVER covers Depth, LiDAR, Views, Events, and RGB under four adverse weather conditions and five sensor failure modes.
  • CMNeXt uses Hub2Fuse with asymmetric RGB and supplementary-modality branches, avoiding the computational cost of separate branches and information loss from a single joint branch.
  • The Self-Query Hub dynamically selects informative auxiliary features, while PPX harvests discriminative cues with approximately 0.01M additional parameters per modality.
  • 66.30% mIoU and a +9.10% gain over the RGB-only baseline are reported on DELIVER, while CMNeXt achieves state-of-the-art performance across six datasets.

2. Related Work

Prior work has advanced semantic segmentation through stronger feature extraction and multimodal fusion, but the cited literature spans modality-specific fusion and broader applications beyond segmentation.

  • Semantic segmentation research has advanced through multi-scale features, attention blocks, context priors, edge cues, and vision-transformer-based dense prediction.
  • Multimodal segmentation harvests complementary information from depth, thermal, polarization, events, LiDAR, and optical flow modalities.
  • CMX uses multi-level cross-modal interactions for RGB-X segmentation, while later work explores channel and token exchanges and multimodal fusion in other tasks.

3. CMNeXt: Proposed Framework

CMNeXt uses a dual-branch Hub2Fuse architecture for arbitrary-modal segmentation, selecting informative auxiliary features before fusing them with RGB. Its SQ-Hub supports scalable modality extension, while PPX extracts multi-scale and channel-enhanced complementary cues.

  • 3.1. CMNeXt Architecture: CMNeXt uses a dual-branch encoder in which RGB forms the primary branch and other modalities form the secondary branch within a Hub2Fuse paradigm.The encoder has four stages, and fused features are decoded for segmentation prediction.
  • 3.2. Self-Query Hub: SQ-Hub adds only ∼0.01M parameters per additional modality, supporting extension to an arbitrary number of modalities.The hub merges M auxiliary feature maps into one feature before RGB fusion.
  • 3.2. Self-Query Hub: The Self-Query Hub dynamically selects informative features from supplementary modalities before their fusion with RGB.It computes modality-specific score masks and chooses the highest-scoring patch across modalities.
  • 3.3. Parallel Pooling Mixer: PPX applies a 7×7 depth-wise convolution, three parallel pooling layers, residual summation, 1×1 mixing, and sigmoid attention to harvest multi-scale modal cues.Its weighted feature is subsequently fused with RGB through FRM and FFM modules.
  • 3.3. Parallel Pooling Mixer: PPX further uses squeeze-and-excitation and a feed-forward network to enhance channel information in cross-modal features.The design highlights fused features spatially through pooling and channel-wise through enhancement.
  • 3.3. Parallel Pooling Mixer: Compared with MSCA, MetaFormer, and FAN, PPX combines parallel pooling for attention weighting with channel-wise enhancement during feature mixing.These are identified as PPX's two advances over the compared architectures.

4. The DELIVER Multimodal Dataset

DELIVER is a CARLA-based multimodal segmentation dataset combining RGB, Depth, LiDAR, Events, and six orthogonal views. It includes adverse conditions, sensor-failure scenarios, multiple annotations, and 47,310 frames across six views.

  • Sensor settings and modalities: DELIVER is built with RGB, Depth, LiDAR, Views, and Event data using six mutually orthogonal views arranged as a panoramic cubemap.The views are front, rear, left, right, up, and down, each with a 91°×91° field of view and 1042×1042 resolution.
  • Sensor settings and modalities: The dataset covers five conditions—cloudy, foggy, night-time, rainy, and sunny—with five corner cases including motion blur and sensor-specific failures.The listed corner cases are Motion Blur, Over-Exposure, UnderExposure, LiDAR-Jitter, and Event Low-resolution.
  • Statistics and annotations: 47,310 frames are provided across six views, including 7,885 front-view samples split into 3,983 training, 2,005 validation, and 1,897 testing samples.The front-view samples contain semantic and instance segmentation labels.
  • Statistics and annotations: DELIVER includes semantic and instance annotations, while the work mainly discusses front-view semantic segmentation.The dataset also contains a 25-class semantic distribution described in Figure 5.

5. Experiments

Experiments across six multimodal segmentation datasets show that CMNeXt scales across modality counts, improves segmentation under adverse conditions, and achieves state-of-the-art or competitive results. Ablations support the contributions of Hub2Fuse, SQ-Hub, and PPX.

  • Datasets and implementation details: CMNeXt is evaluated on six multimodal semantic segmentation datasets spanning up to 81 modalities, including RGB-Thermal, RGB-Depth, light-field, and material segmentation settings.The datasets include KITTI-360, DeLiVER, MFNet, NYU Depth V2, UrbanLF, and MCubeS.
  • Comparison against the state of the art: CMNeXt reaches 66.30% mIoU on DeLiVER after progressively fusing RGB, Depth, Events, and LiDAR, improving 9.10% over the RGB-only baseline.Its mIoU increases from 57.20% to 63.58%, 64.44%, and 66.30% as modalities are added.
  • Comparison against the state of the art: CMNeXt achieves state-of-the-art performance on MFNet and NYU Depth V2, surpasses prior methods on UrbanLF and MCubeS, and outperforms modality-specific methods on KITTI-360.Reported gains include +3.90% on UrbanLF-Real and +8.68% on MCubeS.
  • Ablation studies: Ablations show that removing SQ-Hub decreases mIoU by 1.89%, while replacing PPX with MSCA also reduces accuracy; PPX performs best among compared token-mixing blocks.The study also finds that pooling sizes {3,7,11} yield the best mIoU in PPX.

6. Conclusion

The paper addresses arbitrary-modal semantic segmentation with the DeLiVER dataset and the CMNeXt model. CMNeXt uses Hub2Fuse, SQ-Hub, and PPX, achieves state-of-the-art results across six datasets, and scales from 1 to 81 modalities, while relying on RGB primacy and using only front-view semantics from DeLiVER.

  • Conclusion: DeLiVER provides four modalities and partial sensor failures under varied weather conditions for arbitrary-modal semantic segmentation.The dataset includes Depth, LiDAR, Views, Events, and RGB data, while the paper focuses on front-view semantic segmentation.
  • Conclusion: CMNeXt combines asymmetric Hub2Fuse branches with SQ-Hub for dynamic complementary-feature selection and PPX for efficient cross-modal feature harvesting.The framework is designed for arbitrary-modal fusion and supports negligible parameter growth per added modality as described earlier in the paper.
  • Limitations: The asymmetric architecture assumes RGB representations are essential, and the work exploits only front-view semantics despite DeLiVER providing multi-view data and instance labels.The authors identify multimodal pretraining and fusion of 3D LiDAR and Event representations as future directions.

A.1. Detailed settings in data collection

The data-collection settings describe preprocessing and geometric projection for DELIVER’s depth, event, and LiDAR data. Depth maps are converted to frame representations, while LiDAR points are projected into image coordinates using calibrated transformations.

  • Depth and event representations: Depth2Frames converts grayscale depth maps into logarithmically scaled depth frames to reduce discontinuity and quantization errors.The transformation provides milimetric granularity and better precision at close ranges.
  • Depth and event representations: Event2Frames records raw event point clouds between adjacent frames, with positive- and negative-event thresholds both set to 0.3.
  • LiDAR projection: LiDAR points are projected to 2D image coordinates using the point coordinates, rotation matrix, and translation matrix.The CARLA simulator sets the rotation and translation matrices as unit matrices.

A.2. Dataset structure

DELIVER organizes multimodal semantic-segmentation data across four adverse road conditions and five sensor-failure cases. The dataset contains four sensing modalities and 25 semantic classes, with failure cases included across conditions.

  • Dataset structure: DELIVER contains Depth, LiDAR, Event, and RGB modalities collected under rainy, sunny, foggy, and night conditions.
  • Sensor failures: The five sensor-failure cases are Motion Blur, Over-Exposure, Under-Exposure, LiDAR-Jitter, and Event Low-resolution.These cases evaluate robustness and stability under sensor failures.
  • Annotations: DELIVER defines 25 semantic classes spanning infrastructure, traffic participants, vehicles, terrain, and environmental elements.Examples include Building, Road, Pedestrian, Cars, TrafficLight, Water, Bus, and Truck.
  • Dataset structure: The condition-based and failure-based dataset partitions are mutually inclusive, with each failure case represented within every condition using different samples.

A.4. Dataset comparison

Compared with prior multimodal datasets, DELIVER combines four sensing modalities with adverse conditions and sensor-failure cases. This design supports multimodal semantic segmentation involving up to four modalities while providing greater data diversity than datasets using only one or a few sensors.

  • Dataset comparison: DELIVER includes RGB, Depth, Event, and LiDAR, enabling multimodal semantic segmentation with up to 4 modalities.
  • Dataset comparison: Unlike real-scene datasets such as WildDash and Waymo, which use only one or a few sensors, DELIVER provides four modalities for greater data diversity.
  • Dataset comparison: DELIVER covers adverse conditions and sensor-failure cases alongside its multimodal sensor coverage.The comparison discusses datasets with adverse conditions and cases, while DELIVER combines these with RGB, Depth, Event, and LiDAR data.

B. Implementation Details

The implementation uses modality-specific preprocessing and dataset-specific training configurations, while visual evaluations compare RGB-only, RGB-D, and quad-modal predictions across adverse conditions. The visual results show stronger fusion performance overall, including a 94.8% accuracy score for the quad-modal model in a rainy-scene sidewalk example.

  • Implementation Details: Depth images are converted to HHA representations, while DELIVER LiDAR and Event data are transformed into frame formats for training.HHA contains horizontal disparity, height above ground, and norm angle.
  • Implementation Details: The DELIVER dataset statistics include four adverse conditions and five failure cases per condition.The conditions are cloudy, foggy, rainy, and night; failure cases include MB, OE, UE, LJ, and EL.
  • Implementation Details: The multimodal-dataset comparison table defines abbreviations for daytime, sunset, night, random sampling, semantic segmentation, and instance segmentation.
  • Implementation Details: KITTI-360 training uses 40 epochs, 1408×376 resolution, batch size 4 per GPU, and an MiT-B2 backbone.
  • Implementation Details: NYU Depth V2 uses 500 epochs, 640×480 images, batch size 4 per GPU, an MiT-B4 backbone, and multi-scale flip testing.
  • Implementation Details: MFNet uses an MiT-B4 backbone and 500 epochs, while MCubeS uses MiT-B2 with 512×512 training and 1024×1024 evaluation images.
  • Implementation Details: UrbanLF comparison uses an MiT-B4 CMNeXt configuration, while visualizations cover cloudy, foggy, night, and rainy scenes.
  • Visual results: 94.8% accuracy is achieved by quad-modal CMNeXt for the sidewalk class in a rainy-scene failure example.RGB-only segmentation is better for that class in the shown example, although the quad-modal model has the higher reported accuracy score.
Loading 2303.01480v1…