PaperlayerView exact revision on arXiv

Source-linked AI summary

FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

Xin Zhang, Robby T. Tan

arXiv:2610.21779v1cs.CV

TL;DR

Multi-modal segmentation must handle sensors that are degraded or missing, while existing fusion designs can rely on dedicated modules or multi-branch encoders. FuseAdapt adapts a frozen VFM through modality-private attention updates and routed FFN residuals, and PMM predicts masked-modality representations during training. Across three benchmarks, it reports gains under both full and missing modalities.

  • Problem

    Multi-modal segmentation benefits from complementary sensors, but degradation or missing inputs make reliable fusion across complete and incomplete modalities a central challenge.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    FuseAdapt adapts a shared frozen VFM with modality-private Q/V residuals and routed FFN residuals, while PMM predicts a masked modality's representation from visible modalities using a full-modality teacher target.

    Source: §1 · p. 2

  • Results

    +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art across MCubeS, DELIVER, and MUSES.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Same-backbone controls support the adaptation-path separation and PMM choices, with improvements across the evaluated MiT scales.

    Source: §5 · p. 10

    PMM does not natively align different fields of view or temporal offsets.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose \textbf{FuseAdapt}, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs \emph{adaptation-space fusion} by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce \textbf{Predictive Modality Modeling (PMM)}, a direct representation-level objective for cross-modal redundancy. During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input. PMM is used only during training and adds no inference-time overhead. Across three multi-modal segmentation benchmarks (MCubeS, DELIVER, MUSES), FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art. Project page: https://devinxzhang.github.io/FuseAdapt-Page/.

1 Introduction

Multi-modal segmentation must fuse sensors effectively when all are present and remain reliable when only a subset is available. FuseAdapt adapts a frozen vision foundation model through modality-specific branches and routed cross-modal residuals, while PMM predicts masked-modality representations during training.

1 Introduction

  • Multi-sensor benefits are fragile when modalities are noisy, degraded, spatially imperfect, or unavailable, motivating both complete- and incomplete-modality evaluation.The introduction identifies autonomous-driving and embodied settings as contexts where sensors can be affected unequally.Source: §1 · pp. 1–2
  • Separate-encoder approaches can increase computation, while shared or late-fusion designs can favor the dominant or easiest modality; text anchors and subset consistency provide indirect dense-prediction supervision.The cited limitations include coarse text anchors and cross-modal mismatch.Source: §1 · p. 2, Figure 2 · p. 3
  • FuseAdapt keeps modality-specific Q/V low-rank residuals private and routes receiver-conditioned FFN low-rank residual activations across modalities within a frozen backbone.Routing is adjusted to the visible sensor subset, while the frozen dense FFN transformation remains on the receiving stream.Source: §1 · p. 2
  • PMM masks a target modality during training and predicts its representation from visible modalities using a teacher target computed under full-modality input.The teacher and predictor are used only during training.Source: §1 · pp. 2–3
  • +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art across MCubeS, DELIVER, and MUSES.The evaluation covers both modality-complete and modality-incomplete segmentation.Source: §1 · p. 2

2 Related Work

Prior work addresses multi-modal fusion with modality-pair, RGB-centric, shared-backbone, and feature-transfer designs, while missing-modality methods use masking, distillation, or consistency. FuseAdapt instead communicates between visible sensors through constrained adaptation paths in a shared frozen VFM.

2 Related Work

  • Earlier segmentation work includes fixed RGB-depth pairs, RGB-centric fusion, and shared-backbone fusion with auxiliary alignment, whereas FuseAdapt uses a frozen VFM without a standalone fusion network.The cited examples include RGB-centric fusion [Zhang et al., 2023b] and shared-backbone approaches [Zheng et al., 2024b,c,a].Source: §2 · p. 3, Figure 3 · p. 4
  • For missing modalities, CRM uses complementary RGB-thermal masking and clean-to-masked distillation, while M3L supervises masked-modality segmentation from a full-modality teacher.PMM instead predicts the designated missing sensor's full-context latent representation.Source: §2 · p. 3
  • FuseAdapt keeps Q/V residuals private and restricts directed source-to-receiver communication to FFN low-rank residuals among currently visible sensors.The paper contrasts this with routing all low-rank paths or complete FFN features.Source: §2 · p. 3, Figure 3 · p. 4

3 Proposed Method

FuseAdapt adapts a frozen vision foundation model for modality-complete and modality-incomplete segmentation using lightweight modality-specific paths. It combines private attention adaptation and cross-modal FFN exchange with a training-only objective that predicts masked-modality representations from visible modalities.

3 Proposed Method

  • FuseAdapt adapts a frozen VFM for full and partial inputs, combining modality-specific paths with FFN residual exchange and training-only masked-modality prediction.The pretrained backbone remains fixed, and PMM is removed at inference.Source: §3 · p. 4

3.1 Adaptation-Space Fusion

  • Each modality keeps its Q/V adaptation private, while receiver-conditioned routing exchanges FFN low-rank residual activations across available modalities.The FFN dense transformations remain frozen, and visible outputs are symmetrically aggregated for the shared decoder.Source: §3.1 · pp. 4–5

3.2 Predictive Modality Modeling

  • PMM trains the model to predict a masked modality’s representation from visible modalities, using a full-input EMA teacher target and no missing-target loss when all modalities are visible.The query-based predictor can retrieve from any visible position on a common grid, but does not natively align different fields of view or temporal offsets.Source: §3.2 · pp. 5–6

4 Experiments

Experiments evaluate FuseAdapt on MCubeS, DELIVER, and MUSES under modality-complete and modality-incomplete settings. It reports stronger results across the benchmarks, then examines design choices, robustness scope, and deployment cost.

4 Experiments

  • The experiments cover three benchmarks under MCSS and MISS protocols, with all modalities available at inference for MCSS and modality subsets for MISS.DELIVER, MCubeS, and MUSES cover different sensor sets and segmentation tasks; backbones and training settings vary by benchmark, and cross-backbone comparisons reflect both method and pretrained representation.Source: §4 · pp. 6–7

4.1 Modality-Complete Semantic Segmentation (MCSS) Results

  • On MCubeS, DELIVER, and MUSES, FuseAdapt reports 57.62 mIoU with all four modalities and 56.30 mean mIoU, 64.97 mean mIoU, and 76.14 mean mIoU, respectively, in MCSS.MCubeS mean mIoU compares with 53.29 for StitchFusion and 52.43 for MAGIC; DELIVER compares with 63.68 for StitchFusion and 64.18 for Any2Seg.Source: §4.1 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.2 Modality-Incomplete Semantic Segmentation (MISS) Results

  • In MISS, FuseAdapt achieves 52.89 mean mIoU on MCubeS, 56.22 on DELIVER, and 61.30 on MUSES, outperforming reported baselines across the three benchmarks.On MCubeS, it compares with 38.00 for MAGIC and 41.64 for Zheng et al.; on DELIVER, with 45.04 for Any2Seg, 44.78 for MAGIC++, and 43.91 for StitchFusion.Source: §4.2 · pp. 7–8, Table 4 · p. 8, Table 5 · p. 9

4.3 Ablation Studies

  • Within the frozen-VFM framework, Full FuseAdapt performs best among the tested MCSS variants in both comparison tables, supporting contributions from modality-specific adaptation and routed adaptation exchange.On MCubeS, FuseAdapt achieves 56.02 mIoU with EVA-CLIP-B and 56.28 with NeCo-B, compared with 57.62 using DINOv2-B.Source: §4.3 · pp. 8–9

4.4 Controlled Comparisons

  • On MCubeS, FuseAdapt reaches 57.62 mIoU versus 54.55 for late-logit fusion; it also outperforms dense-feature routing and all-path routed LoRA on MCubeS and MUSES.The comparisons are internal mechanism controls rather than reproductions of complete prior systems.Source: §4.4 · p. 9, Table 7 · p. 10
  • On MCubeS MISS, PMM obtains 52.89 mean mIoU versus 49.84 for output-logit KD and 50.57 for fused-feature KD, improving over output KD in all 15 subsets and fused-feature KD in 14/15.The controls keep the backbone, masking, EMA teacher, segmentation supervision, and schedule the same while varying the auxiliary objective or target.Source: §4.4 · p. 9, Table 8 · p. 10

4.5 Robustness and Deployment Cost

  • On DELIVER’s ten predefined weather and degradation conditions, FuseAdapt scores 67.36 mean mIoU versus 67.19 for Any2Seg, leading in eight conditions but trailing at night and under-exposure.A separate MCubeS test finds lower shift sensitivity for query prediction than per-token regression, not universal reconstruction superiority or robustness to arbitrary sensor misalignment.Source: §4.5 · p. 10
  • Three-modality MiT-B0 FuseAdapt takes 21.05 ms on an idle H100, versus 19.63 ms for CMNeXt and 26.69 ms for StitchFusion, retaining 3.91M deployed parameters.The deployment removes both PMM components; Any2Seg and MAGIC are faster in the reported profile.Source: §4.5 · p. 10

5 Discussion and Conclusion

FuseAdapt combines modality-private attention adaptation with FFN-residual exchange, and PMM predicts masked-modality latents. Same-backbone controls support both design choices across evaluated MiT scales; deployment and evaluation boundaries remain.

5 Discussion and Conclusion

  • Same-backbone controls support both design choices, with improvements across the evaluated MiT scales.FuseAdapt separates modality-private attention adaptation from cross-modal exchange through FFN low-rank residuals, while PMM adds target-conditioned missing-modality latent prediction.Source: §5 · p. 10
  • Inference cost grows with the number of sensor streams, and routing and averaging require a common projected grid rather than arbitrary native token lengths.The shift experiment measures latent-prediction sensitivity, not robustness to arbitrary calibration, field-of-view, or temporal errors; main segmentation comparisons are single runs without a statistical-significance claim.Source: §5 · p. 10

A Controlled Comparisons and Experimental Protocols

The controlled comparisons hold training and model components fixed to test adaptation, fusion, and backbone choices. Their conclusions are limited to the evaluated MiT scales and specified setups.

A Controlled Comparisons and Experimental Protocols

  • The MUSES MiT-B0/B2 controls share a frozen encoder, SegFormer decoder, data split, training recipe, and independently selected best full-modality validation checkpoints.These controls test backbone generality on evaluated hierarchical MiT scales, not arbitrary architectures.Source: A Controlled Comparisons and… · p. 17, Table 9 · p. 17, Table 10 · p. 17
  • Deep Prompt, Shared LoRA, Adapter, Separate LoRA, and FuseAdapt vary prompt, shared or private adaptation, and routed FFN exchange while averaging visible features before the decoder.The variants use 64 prompts per block, rank-4 LoRA, or width-32 residual bottlenecks as specified.Source: A Controlled Comparisons and… · p. 17
  • DINOv2-S/B achieve 77.28/80.16 full-modality best-validation mIoU on MUSES under the specified shared setup, a 2.88-point capacity difference.DINOv2-S was selected to keep the trainable budget comparable to MAGIC/Any2Seg, but does not match their backbone family.Source: A Controlled Comparisons and… · p. 17
  • The MCubeS fusion controls compare frozen feature averaging, concatenation, token attention, late-logit fusion, shared or separate LoRA, and routed FuseAdapt.Table 11 reports four-modality RADN fusion/adaptation mIoU; its caption gives the scope, not comparative values.Source: A Controlled Comparisons and… · p. 17, Table 11 · p. 18
  • Information-flow alternatives test routing Q/V together with FFN residuals or exchanging complete FFN features instead of low-rank residuals.The controlled MUSES MiT-B0 and MCubeS/DINOv2-B comparisons isolate architectural choices rather than reproducing StitchFusion or MLE-SAM in full.Source: A Controlled Comparisons and… · p. 17

B PMM Objective Controls and Teacher Reliability

PMM’s controls compare target-conditioned latent prediction with other consistency objectives and teacher choices. Full-context teacher targets help on average in the measured distribution but are not a per-sample oracle.

B PMM Objective Controls and Teacher Reliability

  • PMM exceeds output KD by 3.05 mean mIoU and fused-feature KD by 2.32.It improves output KD in all 15 subsets and fused-feature KD in 14/15; query prediction improves the linear control by 3.18, and EMA improves the no-EMA query by 2.30.Source: B PMM Objective Controls and Teacher… · p. 18, Table 12 · p. 18
  • Linear prediction improves dropout-only training by 6.04, while the controls support modality-conditioned latent prediction beyond generic consistency objectives.Per-token regression predicts missing-sensor latent tokens at matching spatial indices; the table mean averages all 15 nonempty subsets, including RADN.Source: B PMM Objective Controls and Teacher… · p. 18, Table 12 · p. 18

B.1 Full-context targets are an average prior

  • Across 1,428 MCubeS image/subset pairs, full-input predictions improve per-image mIoU by 4.44 on average and are better in 81.72% of pairs.The set comprises 102 images and 14 proper subsets; full-modality input does not guarantee better predictions when a sensor is degraded.Source: §B.1 · p. 18
  • Reliability-aware target weighting under sensor corruption remains future work.Teacher and student share a frozen initialization and architecture; only student adaptation parameters receive gradients, while teacher adaptation parameters follow an EMA update with coefficient 0.999.Source: §B.1 · p. 18

C Degradation and Spatial-Shift Evaluation

The evaluation examines predefined DELIVER degradation conditions and MCubeS latent prediction under controlled spatial shifts. The shift experiment concerns sensitivity to translation, not general reconstruction or end-to-end robustness.

C Degradation and Spatial-Shift Evaluation

  • FuseAdapt leads eight predefined DELIVER degradation conditions and the mean, while Any2Seg leads Night and UE.Table 13 evaluates all four present modalities across weather and sensor-degradation subsets; MB/OE/UE/LJ/EL denote motion blur, over-exposure, under-exposure, LiDAR jitter, and event low resolution.Source: C Degradation and Spatial-Shift… · p. 19, Table 13 · p. 19
  • Query prediction’s 16-pixel latent-cosine drop is 32.2 times smaller than linear prediction’s, although linear prediction has higher absolute clean cosine.The experiment shifts the three visible MCubeS sensors relative to the clean full-context EMA target and averages channel-wise cosine across tokens, images, modalities, and seeds.Source: C Degradation and Spatial-Shift… · p. 19, Table 14 · p. 19
  • The spatial-shift experiment measures latent-prediction sensitivity to translations, not general reconstruction superiority or end-to-end robustness to arbitrary sensor misalignment.The stated boundary includes arbitrary native token lengths, fields of view, and temporal offsets.Source: C Degradation and Spatial-Shift… · p. 19

D End-to-End Inference Efficiency

Under the stated H100 profiling setup, FuseAdapt has fewer unique deployed parameters than CMNeXt and StitchFusion, but the speed and memory comparisons vary by method. The reported numbers are profile-specific, and some comparator graphs are reconstructed or proxy implementations.

D End-to-End Inference Efficiency

  • The study profiled all methods on the same idle H100 at 1024 × 1024, batch size 1, FP32, with 20 warm-up iterations and 100 measured runs.Source: D End-to-End Inference Efficiency · p. 19
  • Any2Seg and MAGIC are faster, while CMNeXt and StitchFusion use less peak memory than FuseAdapt.These comparisons are under the common profile; reconstructed graphs and the OmniSegmentor proxy are not official implementation benchmarks.Source: D End-to-End Inference Efficiency · p. 19
  • FuseAdapt’s three-input latency is close to CMNeXt and below StitchFusion, with fewer unique deployed parameters than both in this profile.The table covers one-, two-, and three-modality inputs and reports latency, throughput, peak memory, and deployed parameters.Source: D End-to-End Inference Efficiency · p. 19, Table 15 · p. 20

E Methodological Justification

The methodological analyses motivate modality-specific attention adaptation, distinguish fixed sensor branches from dynamic routing, and specify how PMM and class-balanced OHEM are trained. Together, they describe the design choices and training procedures used in these sections.

E.2 Comparison with Mixture-of-Experts (MoE)

  • Each low-rank branch has a fixed source-sensor identity, while receiver-conditioned mixing varies by token and input; absent sensors are masked.A sparse MoE-style LoRA control obtains 54.94 versus 57.62 mIoU on MCubeS; all-path and dense-feature controls more directly test which activations should cross streams.Source: §E.2 · p. 21

E.3 Details of PMM.

  • PMM samples visible-modality masks after warm-up, predicts a missing target only when a modality is absent, and compares it with a detached full-input teacher target.The teacher is updated by EMA; full-input samples have zero missing-target PMM loss, and the teacher and predictor are discarded at inference.Source: §E.3 · pp. 21–22, §E.4 · p. 23

E.1 Rationale for Using Modality-Specific LoRA in Self-Attention Adaptation.

  • Across DELIVER and MCubeS layers, Q- and V-projection LoRA updates have low pairwise cosine similarity, supporting distinct modality-specific correction directions.The reported matrices cover shallow, mid, and deep layers; low similarity alone does not establish that routing those updates is harmful.Source: §E.1 · p. 21, Figure 6 · p. 22

E.4 Class-balanced OHEM Cross-Entropy Loss.

  • The class-balanced OHEM loss uses per-batch inverse-frequency class weights, clipped at 10, with absent-class fallback weight 1.It selects pixels above loss threshold τ = 0.7 and uses the top Nmin hardest pixels when fewer than Nmin pass the threshold.Source: §E.4 · p. 22

F Implementation Details

FuseAdapt uses frozen DINOv2 backbones with rank- and dataset-specific adaptation settings, trained with AdamW on benchmark-specific schedules and resolutions.

F Implementation Details

  • DELIVER and MCubeS use frozen DINOv2-B with rank 8, while MUSES uses frozen DINOv2-S with rank 4.The predictor token count matches the target feature sequence length.Source: F Implementation Details · p. 23
  • Training runs for 500 epochs on MCubeS and 200 epochs on DELIVER and MUSES, following full-modality warm-ups of 200 and 15 epochs, respectively.The initial learning rates are 3.5e−4 for MCubeS and 4e−4 for DELIVER and MUSES, with a 10-epoch warm-up at 0.1×.Source: F Implementation Details · p. 23
  • The runs use AdamW, batch size 1 per GPU for DELIVER and 4 for MCubeS and MUSES, at 1024×1024 or 512×512 resolution by dataset.Training uses A5000 GPUs and standard resizing, flipping, color-jitter, blur, and cropping augmentations.Source: F Implementation Details · p. 23

G More Ablation Studies

The ablations report gains from routing and denser LoRA insertion, with rank 8 best among tested MCubeS settings. Additional tests describe backbone transfer and the adaptation modules’ relative resource overhead.

Rank r

Number of Layers Equipped with FuseAdapt LoRAs

Routing Fusion in FFNs

  • Routing is modality-specific at each FFN stage, using noisy gating trained end-to-end without additional loss terms.All available branches participate without top-k selection or a load-balancing objective; mixing weights remain token-dependent and absent sources are masked.Source: G More Ablation Studies · p. 24
  • Routing fusion improves MCubeS performance from 56.09 to 57.62 mIoU; a shared router also gains competitively at 57.24 mIoU.Simple averaging yields only marginal gains, while learned routing adaptively combines modality-dependent FFN experts.Source: Table 18 · p. 24, G More Ablation Studies · p. 24

Scalability of FuseAdapt

  • For DINOv2-B at 512×512 and rank 8, each added modality costs 1.11M parameters, 16.90 MB memory, and 4.8 GFLOPs.The 4.8 GFLOPs are 0.55% of the 873.96 GFLOPs total model FLOPs.Source: G More Ablation Studies · pp. 24–25
  • For DINOv2-S at 1024×1024 and rank 4, each added modality introduces 0.29M parameters, 4.50 MB memory, and 18.12 GFLOPs.The 18.12 GFLOPs are 2.01% of the 902.76 GFLOPs total model FLOPs.Source: G More Ablation Studies · p. 25
  • Adaptation-module overhead grows linearly with modality count, but these figures exclude the total cost of processing each additional sensor through the frozen encoder.End-to-end deployment measurements are reported separately in Appendix D.Source: G More Ablation Studies · p. 25

Related papers

Loading 2610.21779v1…