PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Multimodal Fusion by Channel Exchanging

Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, Junzhou Huang

arXiv:2011.05005v2cs.CVcs.LG

TL;DR

Existing multimodal fusion methods inadequately balance inter-modal integration with intra-modal processing. CEN dynamically exchanges channels using BN-based importance estimates while sharing convolutional filters and retaining private BN layers. Across RGB-D segmentation and multi-domain image translation, experiments report superior performance over aggregation- and alignment-based methods.

  • Problem

    Existing aggregation- and alignment-based fusion methods inadequately balance integrating common information across modalities with preserving modality-specific processing.

  • Method

    CEN is a parameter-free framework that adaptively exchanges low-importance channels between modality sub-networks using BN scaling factors, with shared convolutional filters and independent BN layers.

  • Results

    CEN outperforms aggregation- and alignment-based fusion methods across RGB-D semantic segmentation and multi-domain image translation evaluations.

  • Takeaways & Limitations

    CEN provides a self-adaptive and compact multimodal-fusion framework that combines channel exchange with preserved modality-specific processing.

  • Takeaways & Limitations

    The paper identifies insufficient interpretability as a drawback of applying deep neural networks to multimodal fusion.

Abstract

from arXiv · show

Deep multimodal fusion by using multiple sources of data for classification or regression has exhibited a clear advantage over the unimodal counterpart on various applications. Yet, current methods including aggregation-based and alignment-based fusion are still inadequate in balancing the trade-off between inter-modal fusion and intra-modal processing, incurring a bottleneck of performance improvement. To this end, this paper proposes Channel-Exchanging-Network (CEN), a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of our CEN compared to current state-of-the-art methods. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.

1 Introduction

Deep multimodal fusion must integrate modality-common information while preserving modality-specific processing. CEN addresses this trade-off through adaptive channel exchange, shared convolutional filters, private BN layers, and evaluations across two multimodal tasks.

  • Motivation: Existing aggregation- and alignment-based fusion methods struggle to balance inter-modal fusion with intra-modal processing, often requiring costly hierarchical combinations.Aggregation can weaken intra-modal propagation, whereas alignment may exchange messages ineffectively.
  • CEN: CEN dynamically exchanges channels between modality-specific sub-networks using BN scaling factors as self-adaptive channel-importance measures.Channels with close-to-zero scaling factors are replaced by the mean of channels from other modalities.
  • CEN: CEN shares convolutional filters while retaining private BN layers, supporting modality-common representations, modality-specific importance estimates, and a compact architecture.The authors state that this makes the multimodal architecture almost as small as a unimodal network.
  • Evaluation: CEN is evaluated on RGB-D semantic segmentation and multi-domain image translation against aggregation- and alignment-based fusion methods.The study also includes ablations isolating the contributions of the proposed components.

2 Related Work

Prior deep multimodal fusion methods primarily aggregate or align modality representations, with modulation as another related direction. CEN instead exchanges channels to address the tension between common and modality-specific information.

  • Aggregation-based fusion: Aggregation-based methods combine modality sub-networks through operations such as averaging, concatenation, or self-attention.Recent aggregation methods may preserve modality sub-networks to mitigate weakened intra-modal processing.
  • Alignment-based fusion: Alignment-based methods regulate multimodal embeddings toward similarity, but whole-distribution alignment can overlook modality-specific patterns.Maximum-Mean-Discrepancy is commonly used to measure distribution discrepancy.
  • Modulation: Modulation methods provide another multimodal-learning direction, including cross-modal message passing through conditional BN modulation.CBN modulates one modality's BN using information from another modality.
  • Channel exchanging: CEN differs from these approaches by using channel exchanging as an adaptive mechanism for combining modality-common information while preserving modality-specific patterns.BN scaling factors, previously used for channel-importance estimation in pruning, guide where channels are exchanged in CEN.

3 Channel Exchanging Networks

CEN combines channel exchanging with shared convolutional filters and modality-specific BN layers to fuse common information while preserving modality-specific propagation. Its exchange rule uses BN scaling factors to identify potentially redundant channels and replace them with channels from other modalities.

  • Overall framework: CEN introduces channel exchanging as a parameter-free fusion operation between multimodal sub-networks.The sub-network fuses multimodal information through channel exchanging while retaining separate sub-network propagation.
  • Overall framework: The framework penalizes selected BN scaling factors with an ℓ1 constraint to discover channels that can be replaced.Each sub-network has BN scaling factors γm,l, and sparsity is imposed on the selected portion ˆγm,l.
  • BN-based importance: The BN scaling factor measures a channel's influence because gradients with respect to its input approach zero as the factor approaches zero.Under the stated ℓ1 regularization, the paper analyzes zero as an attractive state, making such channels candidates for replacement.
  • Channel exchanging: A channel with a scaling factor below threshold θ is replaced by the mean of corresponding channels from other modalities before the next layer.Gradients are detached from the replaced channel and back-propagated through the new channels.
  • Channel exchanging: Channels are partitioned into M equal sub-parts, with exchanging restricted to a different sub-part for each modality.This directed design preserves modality-specific propagation in the remaining M−1 sub-parts and avoids exchanging channels whose scaling factors are not sparsity-constrained.
  • Sub-network sharing: All sub-network parameters except BN layers are shared, allowing private channel-importance estimates while common convolutional filters model modality-common statistics.The shared-filter design is intended for homogeneous modalities such as RGB-D or different image domains; the paper notes that heterogeneous modalities may lose expressive power under parameter sharing.

4 Experiments

CEN is evaluated on RGB-D semantic segmentation and multimodal image translation, with ablations and comparisons against aggregation- and alignment-based fusion methods. Results support channel exchanging, shared convolutions with private normalization, and selective cross-modal propagation.

  • Experimental Setup: CEN is tested on semantic segmentation and image-to-image translation using encoder-decoder frameworks with fusion restricted to modality-specific encoders.Semantic segmentation uses NYUDv2 and SUN RGB-D; translation uses Taskonomy-derived multimodal inputs.
  • Ablation Studies: 51.1 Mean IoU is achieved after directed channel exchanging under ℓ1 regulation, compared with 47.6 before exchanging on NYUDv2.RGB performance rises from 46.0 to 49.7, while Depth rises from 38.1 to 45.1.
  • Ablation Studies: 35.8 vs 38.4 Depth performance shows that sharing convolutional parameters helps, whereas additionally sharing BN layers causes a clear drop.The ablation distinguishes shared convolutional filters from modality-private BN layers.
  • Analysis: Feature-map visualizations associate non-zero RGB channels mainly with texture and non-zero Depth channels mainly with boundaries.The complementary patterns motivate exchanging channels between modalities.
  • Semantic Segmentation: CEN outperforms aggregation- and alignment-based baselines by an average improvement of more than 2% under both comparison settings.Self-attention with all-stage fusion reaches 49.1 versus CEN’s 51.1, while CEN uses parameter-free fusion.
  • Semantic Segmentation: CEN achieves new records across all metrics on both semantic-segmentation datasets and remains better than RDFNet with the same backbone.The comparison also reports a clear advantage for multimodal fusion over unimodal results.
  • Image-to-Image Translation: Lower FID/KID than competing methods is reported for two image-translation cases, and increasing modalities from one to four consistently improves translation performance.Additional translation cases also report lower MAE and MSE than baselines.

5 Conclusion

The paper concludes that CEN provides a distinct channel-exchanging framework for deep multimodal fusion. Its adaptive, BN-guided mechanism is intended to improve fusion while retaining intra-modal processing and compactness.

  • 5 Conclusion: CEN is a self-adaptive and compact deep multimodal fusion framework that uses individual BN scaling factors to guide channel exchanging.The conclusion frames the method as addressing both inter-modal fusion and intra-modal processing.

Broader Impact

The broader-impact passage describes potential benefits for robotics and autonomous vehicles while acknowledging interpretability as a drawback of deep neural multimodal fusion.

  • Broader Impact: Fusing complementary modalities may improve performance and robustness for autonomous vehicles and indoor manipulation robots under varying light and weather.The passage also notes that a global fusion criterion may simplify deployment relative to manually designed hierarchical strategies.
  • Broader Impact: Insufficient interpretability is identified as a drawback of applying deep neural networks to multimodal fusion.

A Proofs

The analysis argues that BN scaling factors approaching zero make channels redundant and that exchanging such channels can reduce training loss under stated conditions.

  • Theorem 1: Theorem 1 establishes that γ = 0 is an attractive local minimum for BN scaling factors under ℓ1 regularization.The attraction probability is expressed through a Gaussian cumulative distribution term involving λ and the loss gradient.
  • Corollary 1: If optimization yields γ = 0, channel exchanging cannot increase the training loss when the exchanged networks are sufficiently expressive and convolutional parameters are unshared.The result compares the optimal losses with and without channel exchanging.
  • Proof construction: The proof constructs an exchanged network whose affected feature contribution becomes a constant feature map, preserving the relevant computation.The construction uses shared downstream weights and biases for the corresponding channel term.
  • Empirical verification: Experiments support the conjecture that channels reaching zero scaling remain nearly redundant later in training.The observation uses NYUDv2 with RefineNet (ResNet101) and tracks 256 channels in a 3 × 3 convolutional layer.
  • Theorem 1: When γ approaches zero, the channel loses influence on the final prediction and becomes redundant during subsequent training.The gradient of the loss with respect to the channel input approaches zero.
  • Discussion: Under unshared parameters, full-channel exchanging scores 49.1 versus 48.5 for half-channel exchanging, while full-channel exchanging is worse with parameter sharing.The comparison is reported in the discussion of Corollary 1 and Table 7.

B Implementation Details

The experiments use convolutional backbones with normalization-based sparsity constraints and compare CEN against concatenation, alignment, and self-attention fusion at multiple depths.

  • Architectures: ResNet101 and ResNet152 support semantic segmentation experiments, while U-Net-256 supports image-to-image translation.BN scaling factors guide exchanging after selected 3 × 3 convolutional layers in ResNet.
  • Baselines: The principal fusion baselines are concatenation, alignment, and self-attention.Concatenation uses channel stacking followed by a 1 × 1 convolution; alignment and self-attention follow reimplemented reference settings.
  • Fusion placement: Baseline fusion is evaluated at early, middle, late, and all-stage locations in the ResNet experiments.These locations follow the second, third, fourth, or all four ResNet stages.
  • Metrics: Image-to-image translation evaluation uses FID and KID, with lower values indicating greater similarity between generated and real images.FID compares fitted Gaussian statistics, while KID uses a polynomial-kernel squared MMD with an unbiased estimator.

C Additional Results

Additional translation experiments cover three multimodal input combinations, and CEN achieves lower MAE and MSE than the baseline fusion methods.

  • Additional translation cases: The additional cases are RGB+Shade→Normal, RGB+Normal→Shade, and RGB+Edge→Depth.Baseline settings match Table 4, using early, middle, late, and all-layer fusion.
  • Results: CEN yields lower MAE and MSE than baseline methods across these additional image-translation cases.Both metrics use ×10^-1 scaling, and lower values indicate better performance.

D Results Visualization

The visualization studies compare CEN with concatenation, alignment, and self-attention using task-specific fusion placements, with prediction differences detailed in the captions.

  • Semantic segmentation: Semantic-segmentation visualizations compare concatenation, alignment, and self-attention baselines.Concatenation and self-attention use all-stage fusion, whereas alignment uses middle fusion after the second ResNet stage.
  • Image translation: Image-translation visualizations use all-layer fusion for concatenation and self-attention, and middle fusion at the fourth encoder layer for alignment.The settings are selected to achieve high performance for each baseline.
  • Visualization comparison: The figure captions provide detailed prediction differences among the compared methods.The visual comparisons are presented for both semantic segmentation and image translation.

E Ablation Studies

The ablations examine channel exchange, parameter sharing, normalization choices, and sensitivity to sparsity and threshold settings across segmentation and image translation. Results support shared convolutions with modality-specific normalization and show the fusion method’s qualitative advantages over alternatives.

  • Parameter sharing: Shared convolutional layers with individual INs support training 2–4 modalities in one network and can outperform individually trained networks.Sharing INs as well causes an obvious performance drop.
  • Channel-exchange settings: The ablations vary sparsity constraints and examine scaling-factor proportions across the seventh encoder layer and all U-Net layers.Sensitivity experiments vary λ from 0.1 × 10^-3 to 30.0 × 10^-3 or θ from 10^-5 to 10^-1 under the stated fixed settings.
  • Parameter sharing: Sharing convolutions with unshared BNs improves CEN performance on NYUDv2 compared with unshared convolution parameters.The comparison uses RefineNet (ResNet101) with single-scale evaluation.
  • Qualitative comparisons: Segmentation visualizations use challenging indoor scenes, lighting conditions, and Cityscapes examples; concatenation is reported as more sensitive to depth noise.The indoor results use RefineNet (ResNet101), while Cityscapes results use PSPNet (ResNet101), both with single-scale evaluation.
  • Normalization design: All four IN components are essential to keep unshared, while unshared scaling factors and offsets appear more important.The evaluated components are γ, β, running mean µ, and variance σ2; metrics are FID/KID (×10^-2).
  • Qualitative comparisons: In image translation, multimodal fusion combines complementary modality strengths, whereas concatenation, alignment, and self-attention remain weak or uncertain in disagreement regions.The comparisons cover Texture+Shade→RGB, RGB+Edge→Depth, and RGB+Shade↔Normal translation.
Loading 2011.05005v2…