PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Token Fusion for Vision Transformers

Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, Yunhe Wang

arXiv:2204.08721v2cs.CV

TL;DR

Multimodal vision transformers need fusion strategies that address scarce prior work, complicated inter-modal alignments, and potential disruption of single-modal architectures. TokenFusion dynamically replaces uninformative tokens with projected inter-modal features and preserves alignment through residual positional embeddings, surpassing state-of-the-art methods across three vision tasks.

  • Problem

    Multimodal vision transformers face scarce prior work, complicated inter-modal alignments, and possible disruption of established single-modal architectures.

  • Method

    TokenFusion dynamically prunes uninformative tokens, substitutes them with projected features from other modalities, and retains alignment using residual positional embeddings.

  • Results

    TokenFusion surpasses state-of-the-art methods across multimodal image translation, RGB-depth semantic segmentation, and image-point-cloud 3D object detection.

  • Takeaways & Limitations

    TokenFusion provides a generally applicable fusion method for homogeneous and heterogeneous vision modalities while preserving much of the original single-modal design.

Abstract

from arXiv · show

Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images. Intuitively, feeding multiple modalities of data to vision transformers could improve the performance, yet the inner-modal attentive weights may also be diluted, which could thus undermine the final performance. In this paper, we propose a multimodal token fusion method (TokenFusion), tailored for transformer-based vision tasks. To effectively fuse multiple modalities, TokenFusion dynamically detects uninformative tokens and substitutes these tokens with projected and aggregated inter-modal features. Residual positional alignment is also adopted to enable explicit utilization of the inter-modal alignments after fusion. The design of TokenFusion allows the transformer to learn correlations among multimodal features, while the single-modal transformer architecture remains largely intact. Extensive experiments are conducted on a variety of homogeneous and heterogeneous modalities and demonstrate that TokenFusion surpasses state-of-the-art methods in three typical vision tasks: multimodal image-to-image translation, RGB-depth semantic segmentation, and 3D object detection with point cloud and images. Our code is available at https://github.com/yikaiw/TokenFusion.

1. Introduction

Vision transformers have proven effective for single-modal tasks, but multimodal fusion must address scarce prior work, complex alignments, and possible disruption of established single-modal designs. TokenFusion adaptively reuses uninformative tokens for cross-modal fusion while preserving much of the original architecture, achieving strong results across three vision tasks.

  • Motivation: Multimodal vision-transformer fusion remains comparatively scarce, with key challenges in deciding how and where features from different modalities should interact.Existing vision-language approaches concatenate tokens before transformer layers but generally do not explicitly use inter-modal alignments.
  • Motivation: Alignment-aware fusion can exploit relations such as projections from world-space points to camera-plane pixels, but added projections may alter pretrained architecture and attention flow.This creates design choices over layers, tokens, channels, architecture, and optimization settings.
  • Contribution: TokenFusion prunes each single-modal transformer and substitutes pruned units with projected alignment features from other modalities.The method aims to preserve relative attention relations among important units while enabling multimodal interaction and inheritance of single-modal pretrained parameters.
  • Results: TokenFusion obtains state-of-the-art performance across multimodal image translation, RGB-depth segmentation, and image-point-cloud 3D detection.The evaluation covers up to four public datasets and seven modalities.
  • Results: 64.9% and 70.8% mAP@0.25 are reported for 3D object detection on SUN RGB-D and ScanNetV2, respectively.These results are presented as specific examples of TokenFusion's performance across the evaluated tasks.

2. Related Work

Prior work establishes transformers as powerful vision models and explores multimodal fusion, but combining off-the-shelf single-modal transformers without disrupting their designs remains challenging. Existing approaches span temporal alignment, CNN-transformer combinations, and transformer-based multimodal fusion.

  • Transformers in computer vision: Vision transformers have been applied successfully to classification, segmentation, detection, image generation, and other computer-vision tasks.The related work positions transformers as broadly useful beyond their original natural-language applications.
  • Fusion for vision transformers: Deep multimodal fusion can leverage multiple input sources, yet combining off-the-shelf single transformers without affecting their elaborate single-modal designs is difficult.This challenge motivates methods that preserve established modality-specific architectures while adding cross-modal interaction.
  • Fusion for vision transformers: Prior multimodal transformer studies address spatial-temporal alignment, infrared-visible fusion, LiDAR-image fusion, coarse-to-fine image processing, and hyperspectral imagery.The surveyed approaches combine transformers with multiple frames, CNN backbones, or modality-specific inputs.

3. Methodology

The methodology frames multimodal transformer fusion around alignment-aware and alignment-agnostic alternatives, then introduces TokenFusion to dynamically replace uninformative tokens with projected features while preserving positional alignment. The design supports homogeneous and heterogeneous modalities with modality-specific projection and transformer choices.

  • Overview: The methodology first reviews naive multimodal fusion methods, then develops TokenFusion for homogeneous and heterogeneous modalities.The framework is evaluated through separate fusion designs for the two modality types.
  • Basic Fusion for Vision Transformers: A transformer layer applies multi-head self-attention, layer normalization, and a multi-layer perceptron to modality-specific token features.The model contains L layers, and input tokens are initially obtained by linear projection of each modality.
  • Basic Fusion for Vision Transformers: Alignment-agnostic fusion concatenates modality tokens without explicitly using known inter-modal correspondences, whereas alignment-aware fusion selects tokens sharing pixels or 3D coordinates.Token projection maps a token from one modality to another using an identity function for homogeneous modalities or a shallow MLP for heterogeneous modalities.
  • Multimodal Token Fusion: TokenFusion dynamically scores token importance, prunes uninformative tokens, and substitutes them with token projections from other modalities before each transformer layer.An l1-norm pruning loss encourages low importance scores, while the task-specific losses remain part of the overall optimization objective.
  • Residual Positional Alignment: Residual Positional Alignment retains a substituted token's original positional embedding while adding the projected feature from another modality.Positional-embedding gradients are retained only at the first layer and frozen afterward.
  • Homogeneous Modalities: Homogeneous modalities use shared self-attention, MLP parameters, positional embeddings, and bidirectional token projection with token-wise pruning.For more than two modalities, tokens are randomly pre-allocated into fixed equal-sized groups associated with other modalities.

4. Experiments

TokenFusion is evaluated across homogeneous and heterogeneous multimodal vision tasks, including image translation, RGB-depth segmentation, and joint 3D/2D detection. Across these settings, it is compared with transformer and state-of-the-art baselines, with strong reported results and qualitative benefits.

  • Experimental Scope: Experiments cover seven modalities and four application scenarios using comprehensive comparisons with state-of-the-art methods.The evaluated tasks include multimodal image-to-image translation, RGB-depth semantic segmentation, and 3D object detection with images and point clouds.
  • Multimodal Image-to-Image Translation: 43.92/0.94 FID/KID scores are achieved by TokenFusion (S) for Shade+Texture→RGB, with a 29.8% relative FID decrease versus CEN.All predictions use 256×256 resolution, and lower FID/KID values indicate better performance.
  • Multimodal Image-to-Image Translation: TokenFusion predictions show richer colors, details, and natural patterns than the compared image-translation methods.Token fusion visualizations also indicate modality-specific behavior: texture preserves detailed boundaries while seeking facial tokens from shade.
  • RGB-Depth Semantic Segmentation: TokenFusion-equipped transformer models surpass state-of-the-art CNN models for RGB-depth semantic segmentation on NYUDv2 and SUN RGB-D.The experiments use relatively light B1 and B2 backbone settings; the authors expect larger backbones such as B5 to perform better.
  • Vision and Point Cloud 3D Object Detection: Images help TokenFusion locate 3D objects with sparse or missing point data and when points from two objects substantially overlap.The reported visualization compares models without and with TokenFusion, which mainly benefits 3D detection results.

5. Ablation Study

Ablations show that l1-based token selection is useful mainly because it identifies tokens for fusion, while residual positional alignment matters when fusion must preserve cross-modal correspondences.

  • l1-norm alone has little performance effect but is essential for revealing tokens for fusion.
  • TokenFusion with l1-norm substantially outperforms random token-fusion baselines.
  • RPA alone does not noticeably change performance, but becomes important when combined with token fusion, especially for 3D detection.
  • The ablations cover RGB-depth segmentation on NYUDv2 and image-point 3D detection on SUN RGB-D.

6. Conclusion

The paper concludes that TokenFusion adaptively fuses homogeneous and heterogeneous vision-transformer modalities by reusing uninformative tokens and exploiting inter-modal alignment. Across varied multimodal tasks, it reports superior performance and generality.

  • TokenFusion adaptively fuses homogeneous or heterogeneous modalities by reusing uninformative tokens for multimodal interaction.
  • Residual positional alignment and inter-modal projection explicitly incorporate alignment relations between modalities.
  • TokenFusion surpasses state-of-the-art methods across varied tasks, supporting its reported superiority and generality for multimodal fusion.

A. Additional Results

Additional experiments examine scaling to more modalities, network sharing, channel-wise fusion, visualized token behavior, and practical inference speed. They report improved performance with more inputs and combined pruning, while shared components reduce model size and added inference costs remain mild.

  • Multiple input modalities: Performance consistently improves as Taskonomy image-to-image translation uses more input modalities, from one to four.For more than two modalities, the group allocation strategy is used.
  • Network sharing: Shared MSA and MLP with individual Layer Normalization reduces storage while outperforming individually parameterized MSA and MLP on two tasks.Further sharing Layer Normalization causes performance drops, especially for image-to-image translation.
  • Channel-wise fusion: Channel-wise fusion prunes uninformative channels using Layer Normalization scaling factors and complements TokenFusion's input-conditional token pruning.
  • Channel-wise fusion: Combining token-wise and channel-wise fusion adds 0.5 mIoU on NYUDv2 over TokenFusion alone.The experiments use λ1 = λ2 = 10^-3 for RGB-depth segmentation.
  • Additional visualizations: Additional visualizations depict fused tokens under l1 sparsity constraints on Taskonomy validation examples at 256 × 256 resolution.The examples cover Texture+Shade→RGB and Shade+RGB→Normal.
  • Inference speed: Added inference time is mild in ScanNetV2 3D detection, partly because the added YOLOS-Ti has only three multi-heads.The speed test uses a single V100 with 256G RAM and varying numbers of input frames.

B. More Details of Image Translation

The image-translation implementation uses transformer-based generator and discriminator networks at 256 × 256 resolution. Evaluation combines pixel-error metrics with distributional similarity metrics derived from Inception representations.

  • The image-translation implementation uses separate transformers for the generator and discriminator, with 256 × 256 inputs and predictions.The discriminator has five stages with two layers each, while embedding dimensions and head counts increase across stages.
  • Evaluation uses MSE, MAE, FID, and KID to assess image-translation performance.
  • Lower FID and KID indicate greater visual similarity between generated and real images.FID compares fitted Gaussian statistics, while KID uses a polynomial-kernel squared MMD with an unbiased estimator.
Loading 2204.08721v2…