PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Model Adaptation for Multimodal Semantic Segmentation

Abhinav Valada, Rohit Mohan, Wolfram Burgard

arXiv:1808.03833v3cs.CV

TL;DR

Semantic segmentation must handle diverse objects, appearance changes, and unreliable visual sensing, while conventional multimodal fusion cannot selectively exploit complementary information. The paper introduces adaptive multimodal fusion alongside the efficient AdapNet++ unimodal architecture, reporting state-of-the-art results across several benchmarks and adverse conditions. The main scope boundary is that some benchmark datasets are highly unbalanced and contain irregular or incomplete boundary annotations.

  • Problem

    Semantic segmentation remains challenging under varied objects, illumination, weather, occlusions, and sensor noise, while direct feature concatenation cannot dynamically select relevant modality information.

  • Method

    The paper combines SSMA, which adapts multimodal feature fusion by object category, spatial location, and scene context, with AdapNet++, an efficient architecture using multiscale residual units, eASPP, and refined decoding.

  • Results

    The unimodal and multimodal architectures achieve state-of-the-art performance across Cityscapes, Synthia, SUN RGB-D, ScanNet, and Freiburg Forest, with multimodal fusion outperforming CMoDE across datasets.

  • Takeaways & Limitations

    Adaptive multimodal fusion can exploit complementary features while targeting robustness, computational efficiency, and fast inference for resource-constrained scene understanding.

  • Takeaways & Limitations

    Evaluated indoor benchmark datasets are highly unbalanced, with rare classes and irregular or unassigned boundary pixels that make segmentation more challenging.

Abstract

from arXiv · show

Learning to reliably perceive and understand the scene is an integral enabler for robots to operate in the real-world. This problem is inherently challenging due to the multitude of object types as well as appearance changes caused by varying illumination and weather conditions. Leveraging complementary modalities can enable learning of semantically richer representations that are resilient to such perturbations. Despite the tremendous progress in recent years, most multimodal convolutional neural network approaches directly concatenate feature maps from individual modality streams rendering the model incapable of focusing only on relevant complementary information for fusion. To address this limitation, we propose a mutimodal semantic segmentation framework that dynamically adapts the fusion of modality-specific features while being sensitive to the object category, spatial location and scene context in a self-supervised manner. Specifically, we propose an architecture consisting of two modality-specific encoder streams that fuse intermediate encoder representations into a single decoder using our proposed self-supervised model adaptation fusion mechanism which optimally combines complementary features. As intermediate representations are not aligned across modalities, we introduce an attention scheme for better correlation. In addition, we propose a computationally efficient unimodal segmentation architecture termed AdapNet++ that incorporates a new encoder with multiscale residual units and an efficient atrous spatial pyramid pooling that has a larger effective receptive field with more than 10x fewer parameters, complemented with a strong decoder with a multi-resolution supervision scheme that recovers high-resolution details. Comprehensive empirical evaluations on several benchmarks demonstrate that both our unimodal and multimodal architectures achieve state-of-the-art performance.

1 Introduction

The paper targets unreliable semantic segmentation under diverse scenes, changing appearance, and sensor conditions by adapting multimodal fusion to object, spatial, and contextual factors. It also introduces efficient unimodal and multimodal architectures designed for accurate, robust, and resource-conscious deployment.

  • Motivation: Real-world segmentation remains difficult because scenes contain varied object types and scales, clutter, occlusions, and changing outdoor appearance.These changes occur throughout the day and across seasons.
  • Motivation: Existing methods can confuse visually inconspicuous objects and fail under camera overexposure, motivating complementary modalities such as depth and infrared.Depth provides geometry, while infrared provides reflectance-related information.
  • Motivation: The open fusion problem is adapting modality weighting to object category, spatial location, scene context, sensor noise, and possible sensor failure.Depth can be informative nearby but degrade rapidly at longer distances, while noisy mandatory fusion can worsen performance.
  • Multimodal framework: SSMA dynamically recalibrates and probabilistically fuses modality-specific intermediate features according to object class, spatial location, and scene context.Its fully convolutional design produces activation-level probabilities that amplify or suppress modality representations.
  • Multimodal framework: The multimodal architecture uses modality-specific encoders, mid-level and late fusion, an attention mechanism, and a shared decoder for upsampling and boundary refinement.The attention mechanism correlates fused mid-level features with high-level decoder features; segmentation remains supervised while SSMA is self-supervised.
  • Efficient architectures: AdapNet++ combines multiscale residual units, efficient ASPP, skip-refinement decoding, multiresolution supervision, and holistic pruning for efficient segmentation.The eASPP reduces parameters by 87% relative to the original ASPP, while pruning further reduces parameters and operations.
  • Evaluation: Evaluations on Cityscapes, Synthia, SUN RGB-D, ScanNet, and Freiburg Forest report state-of-the-art performance, 72ms inference, and robustness in adverse conditions.The reported adverse conditions include fog, snow, rain, and nighttime.

2 Related Works

Related work progresses from hand-engineered and patch-based segmentation toward end-to-end convolutional architectures using multiscale context and learned upsampling. Multimodal fusion methods differ in when streams are combined and how representations are integrated, but prior approaches do not dynamically adapt fusion to scene context.

  • Semantic segmentation: Classical semantic segmentation used hand-engineered features with flat classifiers, whereas later approaches adopted end-to-end deep convolutional neural networks.Examples of flat classifiers include support vector machines, boosting, and random forests.
  • Semantic segmentation: Early neural segmentation systems trained patch-wise or aggregated low-resolution predictions, while later architectures introduced learned or index-based upsampling and refinement stages.DeconvNet learned nonlinear upsampling, whereas SegNet reused encoder pooling indices.
  • Multiscale context: Dilated convolutions enlarge receptive fields without reducing resolution or increasing parameters, enabling pyramid modules to aggregate multiscale global context.DeepLab and PSPNet build on this strategy, although such approaches can incur high computational complexity and inference time.
  • Multimodal fusion: Multimodal fusion research includes handcrafted RGB-D methods and deep networks using early, hierarchical, or late fusion to learn joint representations.Approaches combine modalities through channel stacking, intermediate transformations, recurrent context fusion, graph structures, concatenation, or summation.
  • Multimodal fusion: Late fusion commonly combines independently trained modality streams near the network end, but concatenation or summation does not adapt fusion to changing scene context.This limitation motivates fusion mechanisms that account for the conditions of the scene.

3 AdapNet++ Architecture

AdapNet++ is an efficient fully convolutional encoder–decoder architecture that combines multiscale feature extraction, long-range context, boundary refinement, and multiresolution supervision for semantic segmentation. Its design also supports holistic pruning across encoder and decoder connections.

  • Efficient Atrous Spatial Pyramid Pooling: The eASPP module captures multiscale features and long-range context using cascaded atrous convolutions and bottleneck branches.Its topology has five parallel branches, replacing standard 3×3 atrous branches with cascaded bottleneck branches.
  • Efficient Atrous Spatial Pyramid Pooling: 87.87% fewer parameters and 89.53% fewer FLOPS are reported for eASPP compared with ASPP, while the ablation study reports improved performance.The eASPP uses 2.04M parameters and 3.62B FLOPS, compared with the original ASPP’s 15.53M parameters and 34.58B FLOPS.
  • Multiresolution Supervision: Two auxiliary softmax losses provide multiresolution supervision during training, balance gradient flow, and are removed during testing.The auxiliary branches are placed after the first and second upsampling stages, while only the main decoder stream remains at test time.
  • Network Compression: A holistic pruning technique removes feature maps across convolutional layers while masking shortcut-connected indices to preserve residual and skip connections.The pruned network is fine-tuned and pruning is reiterated toward a desired accuracy–complexity trade-off.

4 Self-Supervised Model Adaptation

The SSMA framework adaptively recalibrates and fuses modality-specific features according to scene content, while a correlation mechanism aligns fused skip features with decoder representations. A two-stream AdapNet++ architecture combines mid-level and late fusion for multimodal segmentation.

  • 4.1 SSMA Block: SSMA explicitly models correlations between modality-specific feature maps to emphasize informative complementary features before fusion.Its fully convolutional design adapts emphasis across spatial locations, object categories, and scene context.
  • 4.1 SSMA Block: The SSMA block concatenates modality features, applies bottleneck recalibration, rescales activations element-wise, and produces a channel-reduced fused output.The bottleneck uses convolutional layers, a nonlinearity, and sigmoid scaling before fusion.
  • 4.2 Fusion Architecture: The multimodal architecture uses modality-specific encoder streams with SSMA fusion at latent and skip-refinement stages before decoding.The latent SSMA uses η = 16, while the two skip SSMAs use η = 6 after channel reduction to 24.
  • 4.2 Fusion Architecture: Directly concatenating fused mid-level and high-level features can create localization mismatch and hamper convergence when encoders are uninitialized.The authors address this with multimodal pretraining of the encoders and attention-based correlation before skip concatenation.
  • 4.2 Fusion Architecture: The decoder correlates fused mid-level skip features with spatially aggregated high-level decoder features before concatenation.This mechanism weights skip features using decoder-derived descriptors and improves segmentation-output resolution over direct concatenation.

5 Experimental Results

The experiments benchmark unimodal and multimodal segmentation models using IoU-based evaluation, specified training procedures, and data augmentation. The supplied passages describe the evaluation setup but do not report quantitative performance outcomes.

  • Experimental scope: The experimental section covers datasets, unimodal AdapNet++ results, model compression, ablations, qualitative results, and multimodal fusion benchmarks.This establishes a broad evaluation structure across architecture and fusion analyses.
  • Evaluation metric: Performance is primarily quantified with the Jaccard Index, or intersection-over-union, computed as TP/(TP + FP + FN) for each object class.TP, FP, and FN denote true positives, false positives, and false negatives, respectively.
  • Training setup: Models use 768×384-pixel inputs, ImageNet-pretrained encoder weights, He initialization for other layers, and the Adam solver.RGB images are resized with bilinear interpolation, while other modalities and labels use nearest-neighbor interpolation.
  • Training setup: Multimodal training first fits each modality-specific AdapNet++ model, then initializes the joint SSMA model with the trained encoder weights.This is a two-stage transfer-learning protocol for the fusion model.
  • Data augmentation: Training augmentation randomly applies rotation, skewing, scaling, vignetting, cropping, brightness and contrast modulation, and flipping.The stated ranges include rotation from −13° to 13° and scaling from 0.5 to 2.0.

5.2 Datasets

The paper evaluates AdapNet++ on diverse urban, indoor, and unstructured forest benchmarks containing challenging perceptual conditions and multiple modalities. These datasets include depth, infrared-derived representations, semantic masks, and substantial annotation or sensing challenges.

  • Benchmark scope: The evaluation spans urban driving, synthetic outdoor, cluttered indoor, and unstructured forest scenes with challenging perceptual conditions and multiple modalities.The selected conditions include rain, snow, fog, nighttime, glare, motion blur, and seasonal appearance changes.
  • Urban datasets: Cityscapes provides complex urban RGB-D scenes collected across more than 50 cities, seasons, lighting conditions, and weather conditions.Images have resolution 2048×1024 and the dataset contains 5000 finely annotated images.
  • Depth preprocessing: Depth inputs can contain noise, holes, occlusions, and missing values, so the pipeline applies fast depth completion before segmentation.The paper also evaluates HHA encoding as an additional depth representation.
  • Indoor datasets: SUN RGB-D contains 10,335 indoor RGB-D images from four camera types, while ScanNet contains 2.5M RGB-D frames from 1512 scans across 707 spaces.The datasets represent varied indoor viewpoints and sensing configurations.
  • Dataset limitations: SUN RGB-D and ScanNet are highly unbalanced and include occluded, irregularly annotated, or unlabeled object regions that make segmentation more difficult.SUN RGB-D has 37 categories, with 16 rarely present and approximately 0.25% of pixels unassigned; ScanNet has 21 categories with irregular boundaries and distant unlabeled parts.
  • Forest dataset: Freiburg Forest targets unstructured environments whose objects vary greatly and whose appearance changes substantially across months because of seasonal variation.The dataset uses multispectral and depth modalities, including NIR, NDVI, NRG, EVI, and depth.

5.3 AdapNet++ Benchmarking

AdapNet++ is benchmarked against established segmentation architectures under matched evaluation settings across urban, synthetic, indoor, and forest datasets. It achieves strong performance across these benchmarks while emphasizing compactness and fast inference.

  • Evaluation protocol: AdapNet++ is compared with established models using official implementations, matched input resolutions, and mIoU plus per-class IoU metrics.Cityscapes and ScanNet results also include official benchmark leaderboard evaluations.
  • Cityscapes: 3.24%: AdapNet++ improves over the highest baseline on the 11-class Cityscapes validation set.It also improves thin-structure categories, including poles by 5.42%, fences by 5.05%, and signs by 7.29%.
  • Synthia: 3.87%: AdapNet++ improves the overall mIoU over baselines on Synthia, with the largest gains for signs at 13.14% and poles at 7.8%.The reported comparison uses the validation set at 768×384 input resolution without flips or multiscale testing.
  • Indoor benchmarks: 2.83%: AdapNet++ outperforms the state of the art on the ScanNet validation set.It performs significantly better in 16 of 20 object categories; the largest gains are 5.70% for toilet and 5.34% for bed.
  • Freiburg Forest: 0.82%: AdapNet++ improves over the state of the art on Freiburg Forest, where its decoder improves segmentation resolution for large objects.The dataset lacks thin structures and objects at multiple scales, limiting the types of improvements evaluated.
  • Efficiency: 72.77ms: AdapNet++ performs inference on an NVIDIA TITAN X while using fewer parameters and less time than most compared architectures.The paper presents this as a performance-versus-compactness trade-off for resource-constrained applications.

5.4 AdapNet++ Compression

The paper evaluates structured pruning of AdapNet++ feature maps to reduce model size and computation while preserving segmentation performance. Its shortcut-invariant strategy provides stronger compression–accuracy trade-offs than the compared pruning approaches.

  • Pruning strategy: The pruning strategy removes entire convolutional filters and corresponding feature maps and kernels while remaining invariant to identity and projection shortcut connections.This design avoids the restrictions that prevent many existing approaches from pruning residual feature maps.
  • Initial compression: 6.82%: Pruning the Res5 block reduces parameters and 3.3B FLOPS with a slight increase in mIoU.The compared baseline achieves less parameter reduction while also lowering mIoU.
  • Aggressive compression: 23.31%: Pruning Res2, Res3, Res4, and Res5 reduces parameters with only a 0.19% mIoU decrease.The Oracle approach produces lower parameter reduction and a larger performance drop in this experiment.
  • Compression trade-off: Regular pruning of convolutional feature maps yields higher parameter and FLOPS reduction with minor mIoU loss than selectively pruning layers at different stages.The experiments extend pruning to deconvolutional feature maps in later stages.

5.5 AdapNet++ Ablation Studies

Ablations show that AdapNet++ gains performance from multiscale encoder features, stronger decoder refinement, multiresolution supervision, and efficient atrous context aggregation. The resulting design improves boundary quality while substantially reducing eASPP computation and parameters.

  • Encoder and decoder: The base ResNet-50 model reaches 75.22% mIoU, while multiscale residual units improve performance by 1.7% without increasing parameters.Subsequent decoder and pre-activation configurations provide additional improvements.
  • eASPP: The eASPP bottleneck reduces computation by 13.6M parameters and 31.41B FLOPS, while cascading atrous convolutions recover performance to 80.77% mIoU.The cascaded design adds only 0.1M parameters relative to the bottleneck model.
  • eASPP: The proposed eASPP reduces parameters by 86.86% and FLOPs by 89.53% while increasing mIoU over the previously best-performing ASPP v3 topology.ASPP v3 reaches 80.67% mIoU before the more efficient configuration.
  • Boundary quality: Trimap evaluation shows that the new decoder improves performance on pixels near object boundaries compared with the earlier decoder.The experiment computes mIoU within a morphological band around void-class labels.
  • Decoder comparison: AdapNet++ reaches 80.77% mIoU and outperforms the reference, LRR, and RefineNet decoder configurations in the decoder comparison.RefineNet requires substantially more parameters and FLOPs than LRR.

5.6 Qualitative Results of Unimodal Segmentation

AdapNet++ qualitatively improves segmentation over dataset-specific state-of-the-art baselines, especially for inconspicuous classes, thin structures, and difficult object boundaries across outdoor and indoor scenes.

  • Cityscapes: AdapNet++ better differentiates inconspicuous Cityscapes classes such as sidewalk versus road and pole versus sign.The improvement is attributed primarily to eASPP's large receptive field and larger object context, with better segmentation of thin poles.
  • Synthia: On Synthia, AdapNet++ more accurately captures bicycles, cars, and people that the baseline misses.Pole-like structures and trees are also more continuous and well defined in the AdapNet++ output.
  • SUN RGB-D: On SUN RGB-D, AdapNet++ reduces confusion among visually similar indoor objects such as chairs, tables, desks, sofas, and wall pictures.The baseline misclassifies parts of these objects, whereas AdapNet++ more completely and precisely segments them.
  • ScanNet: On ScanNet, AdapNet++ improves wall-floor boundaries and avoids misclassifying cabinets, counters, and desks as other furniture.The examples show substantial qualitative improvement over the DeepLab v3 baseline.
  • Freiburg Forest: On Freiburg Forest, AdapNet++ better delineates boundaries between grass, vegetation, trail, and grass than the DeepLab v3 baseline.These class boundaries are difficult to estimate even for humans in the shown unstructured forest scenes.

5.7 Multimodal Fusion Benchmarking

The paper benchmarks SSMA multimodal fusion against established and topology-matched baselines across outdoor, indoor, synthetic, and forest datasets. SSMA generally achieves the strongest multimodal results and remains robust across changing conditions and modality combinations.

  • Benchmark setup: SSMA is compared with LFC, FuseNet, CMoDE, Late Fusion, Stacking, Average, Maximum, and unimodal AdapNet++ models using a common backbone.The benchmark covers Cityscapes, Synthia, SUN RGB-D, ScanNet, and Freiburg Forest.
  • Cityscapes: 80.80% mIoU is achieved by RGB on Cityscapes, where stereo depth degrades with increasing distance from the camera.The Cityscapes validation benchmark evaluates RGB, depth, and HHA modalities and their fusion approaches.
  • Synthia: 49.27% ± 4.04% overall mIoU is achieved by unimodal RGB across the 12 Synthia-Sequences conditions, while RGB-D fusion consistently outperforms unimodal models.Models are trained on Synthia-Rand-Cityscapes and tested separately across seasons and weather conditions in Synthia-Sequences.
  • ScanNet: ScanNet evaluates multimodal fusion on higher-quality depth data, with unimodal RGB reaching 52.92% mIoU and 77.70% pixel accuracy.The supplied passage introduces the comparison and reports the RGB baseline; the accompanying table contains the full benchmark.
  • SUN RGB-D: 66.34% mIoU is achieved by SSMA RGB-HHA on SUN RGB-D, improving 13.42% over unimodal RGB; SSMA msf further reaches 67.52%.The SSMA RGB-HHA model outperforms CMoDE, while SSMA msf provides the highest reported score in this comparison.
  • Freiburg Forest: 84.18% mIoU is achieved by SSMA msf with RGB-EVI on Freiburg Forest, exceeding the SSMA RGB-EVI score of 83.90%.Both SSMA models outperform existing multimodal fusion methods and set the reported state of the art on this dataset.

5.8 Multimodal Fusion Discussion

Fusion performance depends on modality quality, object class, spatial location, and scene context. The authors report that SSMA adapts semantically mature representations to these factors and outperforms competing fusion methods across datasets.

  • Modality conditions: RGB models generally perform best among unimodal alternatives, but dense, less noisy depth can outperform RGB indoors or in synthetic scenes.Outdoor depth is noisy and lacks information for distant objects, whereas confined indoor scenes provide denser depth coverage.
  • Fusion baselines: Stacking performs better outdoors, while Late Fusion performs better indoors when depth representations are dense and informative.Late Fusion can leverage semantically mature representations, but noisy outdoor depth limits its gains.
  • SSMA discussion: SSMA outperforms CMoDE on all evaluated datasets and sets the reported state of the art in multimodal semantic segmentation.The discussion attributes this to dynamic adaptation based on object class, spatial location, scene context, and modality properties.

5.9 Generalization of SSMA fusion to Other Tasks

The paper tests SSMA beyond semantic segmentation on ScanNet scene-type classification using RGB and top-down depth projections. The multimodal SSMA model achieves state-of-the-art performance in that task.

  • Task and architecture: ScanNet scene-type classification assigns indoor scans to 13 categories and uses RGB plus a top-down 2D depth projection as modalities.The multimodal network uses two modality-specific SE-ResNetXt-101 streams fused after block 5 with SSMA.
  • Results: The multimodal SSMA model achieves state-of-the-art performance on the ScanNet validation set.The unimodal depth model beats RGB in mIoU and mean recall, while Late Fusion improves mIoU only slightly and lowers mean recall.
  • Backbone evaluation: Table 23 reports SSMA performance with different real-time backbone networks on Cityscapes validation.This extends the evaluation to backbone variants while retaining the multimodal fusion setting.

5.10 Multimodal Fusion Ablation Studies

The ablations examine fusion placement, attention, bottleneck settings, supervision, and backbone efficiency. Results favor independently weighted encoder-stage SSMA fusion, while attention improves multimodal performance and activation maps show modality-selective recalibration.

  • Real-time backbones: MobileNet v2 SSMA fusion outperforms ERFnet fusion by 12.57% mIoU at 73.62ms, while AdapNet++ fusion reaches 82.84% mIoU at 99.96ms.The ERFnet fusion model reaches 64.60% mIoU at 66.06ms and outperforms both unimodal ERFnet models.
  • Fusion placement: 81.55% mIoU from encoder-end SSMA fusion improves 0.78% over the unimodal F0 model, while adding skip fusion yields 81.75%.The study attributes limited mid-level fusion gains to misaligned representations across modality-specific streams.
  • Channel attention: 82.64% mIoU with channel attention improves 1.09%, compared with 0.2% without channel attention.The larger multimodal-fusion increase is reported on indoor or synthetic datasets.
  • SSMA configuration: Independent SSMA configurations consistently outperform dependent configurations, with encoder-stage placement outperforming the other tested configurations.The fusion scheme learns independent probability weightings in a self-supervised manner.
  • Activation analysis: Activation visualizations show SSMA suppressing noisy modality features and retaining complementary structure across person, pole, table, bathtub, and trail examples.For the table category, fused activations accompany a 4.32% IoU improvement.

5.11 Qualitative Results of Multimodal Segmentation

Qualitative comparisons show multimodal fusion correcting RGB errors across outdoor, adverse-weather, indoor, and forest scenes by exploiting complementary depth, HHA, and EVI information. The examples also expose failures involving thin structures and visually similar classes.

  • Cityscapes: Depth features improve cyclist, distant-car, sidewalk, and fence-boundary segmentation over unimodal AdapNet++ outputs in Cityscapes examples.The multimodal model captures more fence structure even when thin wires are not represented in depth.
  • Synthia: In Synthia rainfall and nighttime scenes, multimodal fusion identifies objects that RGB misclassifies, including people on a road hidden by darkness.The nighttime multimodal prediction is reported with almost no error.
  • SUN RGB-D: HHA depth improves discrimination and structural segmentation in cluttered SUN RGB-D scenes, correcting bed-chair, pillow-bed, table-desk, and door-hatch confusions.The window in one scene is mislabeled as a desk in the ground truth.
  • ScanNet: HHA depth avoids RGB overexposure and motion-blur errors in ScanNet, enabling more accurate object-class predictions.The RGB model assigns blurred regions to the scene’s largest object, whereas depth emphasizes object structure.
  • Freiburg Forest: RGB-EVI fusion recovers boulders absent from oversaturated RGB segmentation in Freiburg Forest by using complementary EVI features.The comparison covers oversaturation, trail, and vegetation confusion.
  • Failure modes: Failure cases remain for extremely thin fences and adjacent chair-sofa instances with similar appearance, causing incorrect or merged predictions.The multimodal model captures more fence structure, but ground-truth labeling and class adjacency still produce errors.

5.12 Visualizations Across Seasons and Weather Conditions

The cross-season evaluation tests models trained on Synthia-Rand-Cityscapes against Synthia-Sequences across twelve seasons and weather conditions. Qualitative scenes span varied environments and RGB degradations such as rain, fog, snow, glare, and darkness.

  • Evaluation setting: The evaluation transfers models trained on Synthia-Rand-Cityscapes to Synthia-Sequences without training there.The target video sequences contain twelve seasons and weather conditions.
  • Scene diversity: The visual scenes cover highways, skyscraper-lined inner cities, small cities, changing vegetation, snow, sunrise glare, sunset hues, nighttime, rain, and fog.These conditions create diverse RGB appearance changes across the qualitative examples.

6 Conclusion

The paper concludes that self-supervised SSMA dynamically fuses modality-specific features across categories, locations, contexts, modalities, and environments. It also presents AdapNet++ as a computationally efficient unimodal alternative with strong benchmark performance and reduced complexity.

  • Multimodal architecture: SSMA dynamically adapts multimodal fusion to exploit relevant complementary information while attending to object category, spatial location, and environmental context.A channel attention mechanism correlates mid-level encoder features with high-level decoder features for boundary refinement.
  • Generalization: Self-supervised fusion generalizes beyond RGB-D to different modalities and environments, including urban, indoor, and unstructured forest settings.The conclusion explicitly describes this as effective generalization.
  • Unimodal architecture: AdapNet++ combines multiscale residual units, efficient atrous spatial pyramid pooling, skip refinement, and pruning for efficient deployment.Its efficient atrous spatial pyramid pooling reduces parameters by 10× while increasing performance versus standard pooling.
  • Benchmark outcomes: AdapNet++ achieves state-of-the-art performance on Synthia, ScanNet, and Freiburg Forest, with comparable Cityscapes and SUN RGB-D performance using fewer parameters and faster inference.The conclusion reports exhaustive evaluation across five datasets.
Loading 1808.03833v3…