PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Multimodal Fusion

Zihui Xue, Radu Marculescu

arXiv:2204.00102v2cs.CVcs.AIcs.MM

TL;DR

Static multimodal fusion applies identical computation despite differing input demands and modality redundancy. DynMM uses data-dependent gating and resource-aware training to adapt modality and fusion paths, achieving efficiency across tasks while retaining or improving performance. The authors present this as a direction for dynamic multimodal network design, while identifying broader architectures and task coverage as future work.

  • Problem

    Existing multimodal fusion approaches process inputs with identical computation, despite diverse computational demands and modality redundancy.

  • Method

    DynMM uses gating networks for modality-level and fusion-level decisions, together with a resource-aware loss that encourages computational efficiency.

  • Results

    Across three multimodal tasks, DynMM demonstrates efficacy, including 0.7% mIoU improvement with 21.1% lower MAdds in semantic segmentation.

  • Takeaways & Limitations

    Dynamic multimodal fusion can balance reduced computation with representation power, performance, and robustness across multimodal tasks.

  • Takeaways & Limitations

    The authors identify better dynamic architectures, sequential decision-making tasks, and broader multimodal task and modality coverage as future work.

Abstract

from arXiv · show

Deep multimodal learning has achieved great progress in recent years. However, current fusion approaches are static in nature, i.e., they process and fuse multimodal inputs with identical computation, without accounting for diverse computational demands of different multimodal data. In this work, we propose dynamic multimodal fusion (DynMM), a new approach that adaptively fuses multimodal data and generates data-dependent forward paths during inference. To this end, we propose a gating function to provide modality-level or fusion-level decisions on-the-fly based on multimodal features and a resource-aware loss function that encourages computational efficiency. Results on various multimodal tasks demonstrate the efficiency and wide applicability of our approach. For instance, DynMM can reduce the computation costs by 46.5% with only a negligible accuracy loss (CMU-MOSEI sentiment analysis) and improve segmentation performance with over 21% savings in computation (NYU Depth V2 semantic segmentation) when compared with static fusion approaches. We believe our approach opens a new direction towards dynamic multimodal network design, with applications to a wide range of multimodal tasks.

1. Introduction

Existing multimodal fusion methods process every input with a fixed architecture, despite differences in modality usefulness and computational redundancy. DynMM adapts modality and fusion decisions per instance to save computation on easy inputs while retaining richer processing for hard or noisy inputs.

  • 1. Introduction: DynMM reduces computation for easy inputs by skipping unnecessary modality paths or later fusion modules while preserving full multimodal processing for hard inputs.In the illustrated easy example, only the text path is activated.
  • 1. Introduction: Static and NAS-based fusion architectures process all instances identically, limiting efficiency and representation power when multimodal inputs differ in difficulty.Easy inputs can contain redundant modalities, whereas hard inputs may require complementary cues from all modalities.
  • 1. Introduction: DynMM can skip noisy modalities for particular instances, which the authors report improves predictions under modality perturbations and may enhance multimodal robustness.The introduction connects path skipping with both noise reduction and better predictions under perturbed inputs.
  • 1. Introduction: DynMM adaptively selects modalities and fusion operations per input, using early exits for easy cases and all fusion modules for hard cases.A gating network makes modality-level and fusion-level decisions during inference.
  • 1. Introduction: +0.7% mIoU with over 21% fewer MAdds is reported for DynMM on RGB-D semantic segmentation.The cited result concerns the depth encoder in the semantic-segmentation setting.

2. Related Work

Prior work includes static, hand-crafted, NAS-based, and early dynamic multimodal architectures. DynMM targets a more general formulation that emphasizes computational efficiency across multimodal tasks rather than task-specific dynamic designs.

  • 2. Related Work: Dynamic neural networks adapt structure to input characteristics through mechanisms such as early exits, skipped branches, or sample-wise width changes.Prior dynamic-network work spans image recognition, detection, segmentation, and machine translation.
  • 2. Related Work: Existing multimodal fusion methods use hand-crafted designs or NAS to combine features, but they do not generally adapt the architecture to each input.These approaches seek effective fixed fusion architectures rather than instance-dependent computation.
  • 2. Related Work: Early multimodal dynamic methods address segmentation, video recognition, visual-inertial odometry, and medical classification, but their goals and application scopes differ.Some methods emphasize feature exchange or informativeness, while computational efficiency is overlooked in one medical-classification line of work.
  • 2. Related Work: DynMM is positioned as a systematic and general formulation of dynamic multimodal fusion beyond video-specific action-recognition methods.The paper describes this as a first step toward a formulation suited to various multimodal tasks.

3. Method

DynMM generates data-dependent multimodal forward paths through modality-level or fusion-level decisions, using hard gating and resource-aware training to balance efficiency with representation power.

  • 3. Method: DynMM introduces modality-level and fusion-level decisions that generate data-dependent forward paths during inference.The method also addresses resource-aware training and optimization of its non-differentiable gating network.
  • 3.1. Modality-level Decision: Modality-level DynMM uses a gating network to select one expert specialized for a subset of modalities, avoiding computation in unselected branches.The gate produces a one-hot decision, so only one expert executes for each instance.
  • 3.2. Fusion-level Decision: Fusion-level decisions provide finer-grained control than modality-level skipping, which can otherwise degrade performance on challenging tasks such as semantic segmentation.Progressive fusion controls how and when auxiliary modalities assist the main prediction process.
  • 3.2. Fusion-level Decision: Fusion-level DynMM selects among candidate fusion operations in stacked cells, allowing computation to stop using an auxiliary modality when earlier fusion suffices.The architecture can skip later fusion operations and corresponding feature extraction for the unused modality, while retaining complex fusion for harder inputs.
  • 3.3. Training Objective: DynMM adds a resource-aware loss that penalizes selected expert or operation costs alongside task loss.The computation costs are predetermined, and λ controls the relative importance of task performance and resource usage.
  • 3.4. Optimization: Gumbel-softmax reparameterization enables end-to-end training despite the gating network’s discrete branch-selection decisions.The forward decision remains one-hot, while a real-valued soft vector is used during optimization.

4. Experiments

Experiments across movie genre classification, sentiment analysis, and RGB-D semantic segmentation show that DynMM adapts computation to multimodal inputs while preserving or improving task performance. Its resource-controlled variants trade inference cost against representation power across modality-level and fusion-level designs.

  • Experimental Setup: Across three tasks and modality configurations, experiments apply modality-level DynMM to classification and sentiment analysis and fusion-level DynMM to semantic segmentation.The tasks span image-text, video-audio-text, and RGB-depth inputs.
  • Movie Genre Classification: DynMM-a reduces MM-IMDB computation substantially while outperforming the unimodal baseline by 1.6% in macro F1, whereas soft-gated DynMM-d achieves the best micro and macro F1 scores.The variants expose a trade-off between computation and representation power against static unimodal and multimodal networks.
  • Movie Genre Classification: Increasing λ shifts branch selection toward the low-cost textual expert, while λ = 0 activates all experts and gives the best performance at higher computation.The resource regularization strength therefore controls whether DynMM favors inference efficiency or prediction performance.
  • Sentiment Analysis: A 46.5% computation reduction with only a 0.47% accuracy decrease demonstrates efficient sentiment prediction relative to the best static network.Allowing more computation yields variants that improve efficiency and accuracy, with DynMM-c achieving the best accuracy and lowest MAE.
  • Semantic Segmentation: A 0.7% mIoU improvement with 21.1% fewer MAdds shows fusion-level DynMM can outperform static fusion while reducing computation.On RGB-D semantic segmentation, DynMM-b achieves both gains simultaneously.

5. Conclusion

DynMM adaptively fuses multimodal inputs during inference, addressing redundancy while balancing computation, performance, and robustness. The authors identify broader architectures, sequential tasks, and additional modalities and tasks as future directions.

  • DynMM adaptively fuses multimodal inputs during inference, and experiments on three different multimodal tasks demonstrate its efficacy.
  • Dynamic multimodal fusion reduces computation while improving performance and robustness, motivating further research on multimodal dynamic networks.
  • Future work includes better architectures for multimodal redundancy, sequential decision-making tasks, and evaluation across additional multimodal tasks and modalities.

A. Implementation Details

The experiments use task-specific unimodal and multimodal architectures, matching gating networks to each task. Computation cost is measured with MACs and controlled through a regularization parameter.

  • MM-IMDB compares a text-only MLP network with a multimodal late-fusion network using text and image encoders.
  • CMU-MOSEI uses transformer-based text and multimodal networks, while NYU Depth V2 uses ResNet-50 image and depth encoders with the ESANet decoder.
  • Gating networks match the task architectures, using MLP, transformer, and convolution gates for MM-IMDB, CMU-MOSEI, and NYU Depth V2, respectively.
  • The resource loss uses each expert's MACs as computation cost, with λ weighting computation against task loss; reported variants use different λ values.

B. Visualization Results

DynMM's gating network selects sample-dependent fusion paths and adjusts how deeply depth features are combined with RGB features. Easy samples use earlier fusion to save computation, while challenging samples invoke more fusion.

  • DynMM adaptively executes different forward paths, selecting the degree and timing of depth–RGB fusion for each multimodal input.
  • Easy RGB-D samples fuse once after the first block and skip depth blocks 2–4, whereas challenging samples fuse at every layer.
  • The sample-wise fusion decisions provide a balance between computational efficiency and performance.

C. Analysis of Regularization Strength

The regularization parameter λ controls the trade-off between inference efficiency and accuracy in CMU-MOSEI. Increasing λ routes more samples to the cheaper expert, while smaller λ favors performance.

  • λ provides a flexible range of efficiency–accuracy trade-offs in DynMM on CMU-MOSEI sentiment analysis.
  • When λ is small, DynMM prioritizes performance and usually selects expert network 2.
  • As λ increases, more test samples are routed to expert network 1, which requires fewer computations.

D. Ablation Study

The ablation study evaluates DynMM training strategies and illustrates how gating decisions route NYU Depth V2 inputs through different computation paths. Joint optimization achieves the overall best performance, while the figures visualize routing behavior and resource-dependent branch selection.

  • Training-strategy ablation: Joint optimization achieves the overall best performance among the evaluated DynMM training strategies.
  • Routing visualization: Figure 8 visualizes RGB-D examples where gating decisions select early fusion or repeated middle-layer fusion, changing which computations DynMM skips.The upper example skips depth extraction layers under early fusion.
  • Resource regularization: Figure 9 compares DynMM with static unimodal and multimodal baselines and reports how branch selection ratios vary with resource regularization strength λ.
  • Training-strategy ablation: Without pre-training, DynMM routes all test samples to one forward path and reaches only 49.2% mIoU.The resulting biased optimization leaves forward paths unevenly optimized.
  • Training-strategy ablation: End-to-end joint optimization improves mIoU by 0.8% because feature extraction layers provide more informative inputs to the gating network.
Loading 2204.00102v2…