Meta-Transformer: A Unified Framework for Multimodal Learning

Yiyuan Zhang; Kaixiong Gong; Kaipeng Zhang; Hongsheng Li; Yu Qiao; Wanli Ouyang; Xiangyu Yue

Source-linked AI summary

Meta-Transformer: A Unified Framework for Multimodal Learning

Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, Xiangyu Yue

arXiv:2307.10802v1Jul 20, 2023cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Designing a unified network across modalities remains difficult because their data patterns differ substantially. Meta-Transformer maps inputs into a shared token space and uses a frozen shared encoder with task-specific heads across 12 modalities, achieving strong performance across diverse benchmarks. The framework indicates a promising direction for unified multimodal learning with transformers.

Problem
Unified multimodal learning lacks a common network for diverse modalities because modality gaps make their data patterns difficult to represent together.
Method
Meta-Transformer uses modality-specific tokenization into a shared token space, a frozen modality-shared encoder, and task-specific heads for downstream tasks.
Results
Meta-Transformer achieves outstanding or consistently superior performance across benchmarks spanning 12 modalities and diverse multimodal learning tasks.
Takeaways & Limitations
The results support the potential of a unified transformer framework for multimodal perception across fundamental, practical, and data-mining tasks.
Takeaways & Limitations
The token-embedding computation requires O(n2 × D), creating high memory and computation costs that make scaling difficult.

Abstract

from arXiv · show

Multimodal learning aims to build models that can process and relate information from multiple modalities. Despite years of development in this field, it still remains challenging to design a unified network for processing various modalities ($\textit{e.g.}$ natural language, 2D images, 3D point clouds, audio, video, time series, tabular data) due to the inherent gaps among them. In this work, we propose a framework, named Meta-Transformer, that leverages a $\textbf{frozen}$ encoder to perform multimodal perception without any paired multimodal training data. In Meta-Transformer, the raw input data from various modalities are mapped into a shared token space, allowing a subsequent encoder with frozen parameters to extract high-level semantic features of the input data. Composed of three main components: a unified data tokenizer, a modality-shared encoder, and task-specific heads for downstream tasks, Meta-Transformer is the first framework to perform unified learning across 12 modalities with unpaired data. Experiments on different benchmarks reveal that Meta-Transformer can handle a wide range of tasks including fundamental perception (text, image, point cloud, audio, video), practical application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph, tabular, and time-series). Meta-Transformer indicates a promising future for developing unified multimodal intelligence with transformers. Code will be available at https://github.com/invictus717/MetaTransformer

1 Introduction

Unified multimodal learning remains difficult because modalities have substantially different data patterns. Meta-Transformer addresses this gap with a shared-parameter framework spanning 12 modalities and reports strong benchmark performance.

Motivation: Modality gaps make it difficult to design one network that processes diverse data formats and transfers knowledge across modalities.Images, point clouds, audio, and video exhibit distinct redundancy, sparsity, temporal, and spatial patterns.
Motivation: Transformers have improved perception across language, 2D vision, 3D vision, and audio, motivating unified multimodal architectures.Prior transformer-based systems demonstrate versatility across several modality-specific tasks.
Framework: Meta-Transformer simultaneously encodes 12 modalities using the same parameters through modality-specialist tokenization, a modality-shared encoder, and task-specific heads.Inputs are transformed into token sequences in a common manifold before shared representation extraction.
Experiments: Meta-Transformer is evaluated on benchmarks covering 12 modalities and demonstrates consistently superior outcomes over state-of-the-art methodologies in different multimodal learning tasks.The experiments use images from LAION-2B exclusively for pretraining.

2 Related Work

Earlier neural architectures achieved modality-specific success, while transformer-based multimodal methods expanded cross-modal modeling. However, unified modality-agnostic learning remains constrained by modality differences and limited extensibility beyond vision and language.

Single-Modality Perception: MLPs, recurrent networks, and CNNs have been applied successfully to text, images, point clouds, audio, and sequence modeling.These architectures were often adapted to the properties of individual modalities.
Transformer: Transformers now support text, image, point-cloud, and audio tasks, including understanding, generation, classification, detection, segmentation, and recognition.Transformer adoption has broadened across multiple perception domains.
Transformed-based Multimodal Perception: There is no common architecture for modality-agnostic learning despite the complementarity of information from different modalities.A shared parameter space is needed to bridge their complicated representations.
Transformed-based Multimodal Perception: Existing multimodal methods commonly use cross-attention or partial parameter sharing to align representations across modalities.Examples include MCAN, BEiT-v3, and MoMo, covering cross-modal alignment, mask-and-reconstruction, and shared-encoder strategies.
Transformed-based Multimodal Perception: Most multimodal research focuses on vision and language, limiting direct applicability to point clouds, audio, and other modalities while often requiring expensive training.The paper identifies broader extensibility and cross-modal knowledge transfer as continuing challenges.

3 Meta-Transformer

Meta-Transformer unifies multimodal processing by mapping diverse inputs into a shared token space, encoding them with a common transformer, and applying task-specific heads. Its tokenizer adapts to modality-specific structures while the encoder and downstream pipeline support multiple data types.

The framework combines a data-to-sequence tokenizer, a modality-agnostic encoder, and task-specific heads for downstream prediction.The tokenizer projects modalities into a shared embedding space before unified encoding.
Data-to-Sequence Tokenization: Meta-tokenization transforms data from different modalities into token embeddings within a shared manifold space.The scheme is adapted to practical characteristics of each modality.
Data-to-Sequence Tokenization: Text is converted into WordPiece subword embeddings, while images are reshaped into flattened patches and projected to the common dimension D.Image token counts are determined by the image resolution and patch size.
Data-to-Sequence Tokenization: Point clouds are tokenized using farthest-point sampling, K-nearest-neighbor grouping, and aggregation of local structural representations.The procedure uses grouped local geometry to represent 3D data.
Data-to-Sequence Tokenization: Audio spectrograms are split into overlapping time-frequency patches and flattened into token sequences.The process uses S × S convolution and produces Ns patches.
Unified Encoder: A unified transformer encoder with frozen parameters processes token sequences from different modalities, using position embeddings and a CLS summary representation.The encoder stacks multi-head self-attention and MLP blocks, while task-specific heads mainly use MLPs.

4 Experiments

Experiments evaluate Meta-Transformer across 12 modalities and diverse perception, forecasting, tabular, and graph tasks. Results show competitive or superior performance in many settings, often with few trainable parameters, while graph understanding remains limited.

Experimental scope: Experiments cover 12 modalities using benchmarks for text, images, infrared, X-Ray, hyperspectral data, point clouds, audio, video, time series, tabular data, and graphs.The experimental design includes modality-specific datasets and tasks spanning classification, detection, segmentation, recognition, forecasting, and graph understanding.
Text understanding: On GLUE, frozen Meta-Transformer-B16F reaches 54.6% sentiment, 81.1% paraphrase, 66.0% duplication, 63.4% inference, and 56.3% answering scores.Finetuning improves these scores to 81.3%, 81.8%, 78.0%, 70.0%, and 60.3%, respectively.
Image understanding: On ImageNet classification, Meta-Transformer-L14T achieves 88.1% accuracy, exceeding SwinV2-L/24‡ at 87.6% and InternImage-XL‡ at 88.0%.The model also remains competitive in object detection and semantic segmentation, although SwinV2-L/24‡ performs better on both tasks.
Point cloud understanding: Point-cloud experiments report 72.3% mIoU and 83.5% mAcc on S3DIS, plus 87.0% instance mIoU and 85.2% category mIoU on ShapeNetPart.These results use 2.3M parameters for the S3DIS and ShapeNetPart evaluations.
Other modalities: Meta-Transformer is competitive across specialized modalities, including 97.0% audio accuracy, 94.1% X-Ray accuracy, and time-series performance exceeding several existing methods with 19K or 2M trainable parameters.Video recognition is a weaker case: 46.6% accuracy is obtained with 1.1 million trainable parameters, below video-specialized methods.
Limitations: Graph understanding exposes a current limitation: Meta-Transformer-B16F records 0.8034 train MAE and 0.8863 validation MAE on PCQM4M-LSC, versus Graphormer’s 0.0582 and 0.1234.The authors describe this as limited ability for structural data learning and identify it as future work.

5 Limitation

Meta-Transformer has computational, methodological, and application limitations that constrain scalability, temporal and structural modeling, and the currently demonstrated scope of multimodal perception.

Complexity: O(n^2 × D) computation, high memory cost, and heavy computation burden make Meta-Transformer difficult to scale up.The limitation arises when processing token embeddings [E1, · · · , En].
Methodology: Meta-Transformer lacks temporal and structural awareness compared with Axial Attention mechanisms in TimeSformer and Graphormer.This may affect tasks where temporal or structural modeling is critical, including video understanding, visual tracking, and social network prediction.
Application: Meta-Transformer’s demonstrated advantages primarily concern multimodal perception, while its ability for cross-modal generation remains unknown.The authors identify cross-modal generation as future work.

6 Conclusion

The paper situates plain transformers within the progression from earlier neural architectures for multimedia representation learning and explores their broader potential.

6 Conclusion: The paper traces AI development from MLPs through recurrent and convolutional networks before exploring plain transformers for unified multimodal representation learning.Earlier architectures had achieved representation-extraction success across text, images, point clouds, and audio.

A Summary

The appendix extends Meta-Transformer evaluation across additional modalities and multimodal tasks, describing tokenizer adaptations and benchmark settings for video, infrared, hyperspectral, and X-ray data.

A Summary: The appendix evaluates Meta-Transformer on video, infrared, X-ray, and hyperspectral images beyond the modalities covered in the main paper.It reports additional experimental results in the appendix.
A Summary: The appendix also studies multimodal tasks involving inputs from more than one modality to perform predictions.These experiments further examine Meta-Transformer’s performance and merits in multimodal settings.
Video: For video recognition, the tokenizer replaces 2D embeddings with a 3D embedding layer to encode spatial-temporal information from frames.A modality-shared encoder and task-specific heads then extract video features for UCF101 action recognition.
Video: UCF101 contains 13,320 video clips spanning 101 action categories, with frames recorded at 320×240 resolution and 25 FPS.The categories include body motion, human interactions, musical instruments, and sports.
Infrared: For infrared images, the framework encodes temperature values alongside visual features while retaining the RGB-image tokenizer.The adaptation targets thermal information in infrared recognition.
Infrared: The appendix uses the RegDB dataset to evaluate infrared recognition under variations in pose, expression, illumination, and occlusion.RegDB is intended to represent unconstrained and realistic scenarios.
Hyperspectral: For hyperspectral images, spectral bands are represented in token embeddings, requiring a new linear projection layer instead of the existing 2D convolution layer.The modification addresses high-dimensional spectral information.
Hyperspectral: The Indian Pine hyperspectral dataset consists of 145 × 145 pixels with 145 spectral bands captured in Indiana.It is used for remote sensing and hyperspectral image analysis.

B.4 X-Ray Image Recognition

The paper extends Meta-Transformer’s perception framework to medical X-ray analysis and also examines a challenging audio-visual segmentation task.

B.4 X-Ray Image Recognition: Meta-Transformer uses the RGB-image tokenizer to encode raw medical images for Chest X-Ray analysis.The Chest X-Ray dataset contains 7,000 chest X-ray images annotated for thoracic-condition analysis.
B.4 X-Ray Image Recognition: Because text, image, point cloud, and audio are all studied, the paper evaluates multimodal learning on audio-visual segmentation rather than comprehensive standard multimodal experiments.The task focuses on aligning an intelligent listener with fundamental visual tasks.

C.1 Audio-Visual Segmentation

This section presents Meta-Transformer for audio-visual segmentation on AVSS, combining visual and audio representations without dedicated multimodal fusion modules. It reports stronger segmentation metrics, fewer parameters, faster convergence, and improved efficiency than competing methods.

Task and Dataset: Audio-visual segmentation identifies and delineates objects associated with different audio sources in a referring image.The task analyzes synchronized audio and visual signals and supports applications including video conferencing, surveillance, multimedia analysis, and augmented reality.
Method: Meta-Transformer directly concatenates visual and audio embeddings after Data-to-Sequence processing instead of introducing multimodal fusion modules.
Results: 31.33% mIou and 0.387 F-score are the highest reported values for Meta-Transformer on AVSS.It also uses 86.5 million parameters, compared with approximately 80M to 180M for other methods.
Advantages: The unified architecture processes audio and images with one encoder, reducing reliance on modality-specific encoders and computation.
Advantages: Four training epochs are sufficient for Meta-Transformer to reach 31.33% mIou, which the paper associates with deeper alignment between modalities.
Advantages: Meta-Transformer improves performance by 10% over methods with a similar parameter scale while requiring only one-third as many parameters.

D Experimental Details

The implementation builds on several open-source projects and directs readers to the source code for further details.

Implementation: The codebase uses open-source projects including MMClassification, MMDetection, MMSegmentation, OpenPoints, Time-Series-Library, and Graphomer.
Implementation: Additional implementation details are available in the source code.

E.1 Modality-Free Perception

This section describes Meta-Transformer’s modality-free perception across vision, language, point clouds, video, remote sensing, medical imaging, and infrared data. The shared encoder projects diverse modalities into a common representation space and supports cross-modal applications.

Modality-Free Perception: A shared encoder represents natural language, 2D images, 3D point clouds, and audio spectrograms in a shared representation space.The paper states that this reduces the modality gap and the burden of cross-modal alignment.
Modalities: The experiments cover RGB images, text, point clouds, video understanding, hyperspectral images, infrared images, and X-ray images.
Applications: Video understanding can integrate text, audio, and image information for action recognition, event detection, and video summarization.The paper also identifies video surveillance, indexing, and content-based retrieval as application areas.
Applications: Hyperspectral perception extracts high-level semantic features for classification, target detection, and land-cover mapping.The paper connects these capabilities to environmental monitoring, agriculture, urban planning, and disaster management.
Applications: X-ray analysis can combine X-ray images, clinical data, and other modalities for disease detection, anomaly identification, and treatment planning.
Applications: Infrared processing supports object detection, tracking, and recognition in low-light conditions, including cooperation between infrared and RGB cameras.

E.3 Conclusion

The conclusion attributes Meta-Transformer’s unification ability to modality-invariant patterns and flexible, length-variable token embeddings. It identifies unseen-modality generalization and unified multimodal decoding as future directions, while noting uncertainty about generative tasks.

Conclusion: Neural network architectures can learn modality-invariant patterns, which the paper identifies as the basis of Meta-Transformer’s unified multimodal learning.
Conclusion: Length-variable token embeddings provide flexible but unified forms of multimodal semantics.
Future Directions: Future work includes algorithms that generalize to unseen modalities and a unified decoder that can generate modality-specific outputs.
Limitations: The effectiveness of the proposed architectures for generative tasks remains uncertain.The paper specifically identifies modality-invariant generative models as an unresolved research question.