Source-linked AI summary
Multimodal Learning with Transformers: A Survey
Peng Xu, Xiatian Zhu, David A. Clifton
TL;DR
Transformer-based multimodal learning lacks a unified survey connecting its rapidly growing methods across modalities, applications, and challenges. This paper provides a geometrically grounded, self-attention-centered taxonomy and review of multimodal Transformer research. It synthesizes current designs and applications while identifying bottlenecks and open research directions.
Problem
Rapid growth of multimodal Transformer methods across disciplines creates a need for a structured review of their global landscape, shared challenges, and open problems.
Method
The paper surveys multimodal Transformer designs and applications using a geometrically topological perspective and a two-tier taxonomy based on applications and challenges.
Results
The survey synthesizes multimodal pretraining and task-specific applications, self-attention-based cross-modal interactions, shared challenges and designs, and potential research directions.
Takeaways & Limitations
The review provides a structured reference for understanding Transformer-based multimodal learning across related research and application domains.
Takeaways & Limitations
Multimodal pretraining models oriented toward discriminative tasks do not directly support generative tasks, often requiring a separate decoder.
Abstract
from arXiv · showhide
Transformer is a promising neural network learner, and has achieved great success in various machine learning tasks. Thanks to the recent prevalence of multimodal applications and big data, Transformer-based multimodal learning has become a hot topic in AI research. This paper presents a comprehensive survey of Transformer techniques oriented at multimodal data. The main contents of this survey include: (1) a background of multimodal learning, Transformer ecosystem, and the multimodal big data era, (2) a theoretical review of Vanilla Transformer, Vision Transformer, and multimodal Transformers, from a geometrically topological perspective, (3) a review of multimodal Transformer applications, via two important paradigms, i.e., for multimodal pretraining and for specific multimodal tasks, (4) a summary of the common challenges and designs shared by the multimodal Transformer models and applications, and (5) a discussion of open problems and potential research directions for the community.
1 INTRODUCTION
This survey organizes Transformer-based multimodal learning around modality-agnostic self-attention, offering a geometric-topological interpretation and a two-tier taxonomy spanning applications and challenges. It reviews Transformer designs, multimodal applications, shared challenges, bottlenecks, and future directions.
- Motivation: Transformers support multimodal learning with fewer modality-specific architectural assumptions because tokenized sequences can represent one or multiple modalities.Self-attention input patterns can control per-modal specificity and inter-modal correlation.
- Scope: The survey covers a broad range of modalities and tasks, including images, video, audio, language, point clouds, documents, and 3D scenes.Its scope includes modality-specific Transformer designs across many multimodal learning settings.
- Contribution: The authors describe the work as the first comprehensive review of Transformer-based multimodal machine learning and discuss current bottlenecks and research directions.The survey is intended as a reference for researchers and practitioners across related disciplines.
- Theoretical perspective: The survey interprets self-attention geometrically as modeling arbitrary tokens from arbitrary modalities as nodes in a fully connected graph.This perspective is used to explain the modality-agnostic traits of Transformer architectures.
- Theoretical perspective: The paper systematically reviews Vanilla Transformer, Vision Transformer, and multimodal Transformers from a geometrically topological perspective.It discusses Transformer components in multimodal contexts as mathematically as possible.
- Taxonomy and applications: Its two-tier taxonomy organizes multimodal Transformer research by application and challenge, enabling formula-driven comparison across domains.Application coverage includes multimodal pretraining and specific multimodal tasks, while the challenge perspective summarizes shared designs and difficulties.
2 BACKGROUND
Multimodal learning has expanded with the growth of multimodal applications, datasets, and Transformer-based models. This survey situates that development across modalities, scenarios, dataset trends, and multimodal pretraining.
- Multimodal learning: Multimodal learning supports applications spanning perception, communication, interaction, healthcare, and autonomous systems.Examples include camera- and LiDAR-equipped navigation robots, lip reading, sign-language translation, and vision-and-language navigation.
- Transformer ecosystem: Transformers emerged from successful sequence modeling in NLP and subsequently expanded into visual and multimodal learning.ViT applies Transformer encoding directly to image patches, while VideoBERT extended Transformers to multimodal tasks.
- Transformer ecosystem: CLIP reframed classification as retrieval during multimodal pretraining, enabling pretrained models to perform zero-shot recognition.Subsequent work extends the CLIP idea to semantic segmentation and other multimodal models.
- Multimodal big data: Multimodal datasets are becoming larger, more diverse in modality, broader in scenario coverage, and more difficult in task formulation.Examples include million-scale datasets, ten-modality collections, autonomous-driving scenarios, and metaphor-understanding or hateful-meme tasks.
- Multimodal big data: Instructional videos provide pretraining settings that align task instructions with observed actions, while multimodal big data supports zero-shot learning in VLP Transformers.The survey identifies instruction-video alignment as a pretext task and links large data with zero-shot capability.
3 TRANSFORMERS
The survey reviews Vanilla, Vision, and multimodal Transformers mathematically, emphasizing tokenized inputs and self-attention. It interprets self-attention geometrically as constructing fully connected graphs over tokens.
- Theoretical review: The survey uses mathematical formulations to review tokenized inputs, self-attention, multi-head attention, and Transformer layers across three Transformer types.The reviewed types are Vanilla Transformer, Vision Transformer, and multimodal Transformers.
- Geometrical perspective: Self-attention allows Transformer inputs from any modality to be modeled as fully connected graphs in topological geometry space.This perspective extracts the mathematical essence of multimodal Transformer practices from self-attention designs.
3.1 Vanilla Transformer
Vanilla Transformer processes tokenized sequences through attention-based blocks, positional information, masks, and feed-forward networks. Its tokenization and self-attention provide modality-flexible, globally connected representations, while position encoding remains an open problem.
- Architecture: Vanilla Transformer uses stacked encoder-decoder blocks containing multi-head self-attention and position-wise feed-forward sub-layers.Normalization and residual connections support gradient backpropagation, while the FFN applies successive linear transformations with nonlinear activation.
- Input tokenization: Tokenized inputs let Transformers process modalities through irregular sparse structures rather than modality-specific sequential or grid constraints.Each token can be treated as a graph node, enabling a common input representation across modalities.
- Input tokenization: Tokenization flexibly combines modality attributes and supports customized tokens for tasks such as masked language modeling and classification.Sketch tokens can integrate coordinates, ordering, and pen state, while position embeddings can be summed or concatenated with token embeddings.
- Position embedding: Position embedding supplies temporal or spatial information, but its interpretation and necessity remain open questions across modalities.For point clouds and sketch strokes, coordinates may already provide positional information, making additional position embeddings optional.
- Self-attention: Self-attention projects embeddings into queries, keys, and values, allowing every input element to attend to all others.This gives the Transformer encoder a fully connected graph interpretation and global non-local perception.
- Masked self-attention: Masked self-attention prevents positions from attending to subsequent positions and can inject domain or prior knowledge through designed masks.Masks may be hard or soft and are used in both encoder and decoder settings.
- Multi-head self-attention: Multi-head self-attention ensembles several attention sub-layers so the model attends jointly across multiple representation sub-spaces.The concatenated head outputs are fused through a projection matrix.
3.2 Vision Transformer
Vision Transformer converts an image into a sequence of embedded patches and processes that sequence with a standard Transformer encoder. This provides an image-specific input pipeline based on fixed-size patch tokenization.
- Image tokenization: Vision Transformer splits an image into fixed-size patches, linearly embeds them, adds position embeddings, and encodes the resulting sequence.The input image X ∈ R^H×W×C is reshaped into N flattened 2D patches with resolution P × P.
- Image tokenization: The patch sequence is formed by reshaping the image into flattened 2D patch vectors before standard Transformer encoding.The projection matrix provides the linear mapping into the Transformer embedding space.
3.3 Multimodal Transformers
Multimodal Transformers represent heterogeneous inputs as token sequences and use self-attention variants to model cross-modal interactions. The survey organizes these designs by input processing, attention patterns, and hierarchical or stream-based architectures, with different trade-offs in context coverage and computational complexity.
- Multimodal Input: Transformers treat token embeddings from arbitrary modalities as graph nodes, enabling modality-agnostic multimodal processing.Self-attention models the input as a fully connected graph and can therefore process diverse modalities through tokenization and embedding.
- Multimodal Input: Multimodal inputs are prepared by tokenization and embedding selection, with flexible granularity such as regions of interest or CNN features for images.Tokenization choices range from coarse-grained to fine-grained representations.
- Self-Attention Variants: Early summation combines modality embeddings position-wise with weights α and β, avoiding increased computational complexity but requiring manually set weightings.The fused sequence is then processed by Transformer layers.
- Self-Attention Variants: Early concatenation lets all multimodal token positions attend across the combined sequence, encoding global context while increasing computational complexity through longer sequences.This design is also termed all-attention or Co-Transformer.
- Self-Attention Variants: Hierarchical attention combines independent modality streams into one stream or processes a shared stream before separating modalities, with the latter preserving uni-modal representation independence.Multi-stream-to-one-stream attention is described as late interaction or fusion and as a special case of early concatenation.
- Self-Attention Variants: Cross-attention exchanges query embeddings between two streams so each modality attends conditionally to the other, while cross-attention followed by concatenation restores global-context modelling.Cross-attention avoids higher computational complexity but may lose global cross-modal context when applied separately to each modality.
4 APPLICATION SCENARIOS
The survey organizes multimodal Transformer applications around multimodal pretraining and specific multimodal tasks, emphasizing cross-modal interaction, transferable pretext objectives, and persistent bottlenecks.
- Multimodal Transformer applications are reviewed through multimodal pretraining and specific multimodal task paradigms.
- Multimodal pretraining: Large-scale multimodal pretraining has produced strong downstream performance and zero-shot generalization across diverse modality combinations.
- Multimodal pretraining: Existing multimodal pretraining models are mainly single-stream, multi-stream, or hybrid-stream, while speech can serve as text-derived cross-modal supervision.
- Multimodal pretraining: Pretraining pipelines comprise tokenization, Transformer representation, and objective supervision, with cross-modal interactions occurring at different components or levels.
- Challenges: Well-aligned multimodal corpora remain costly and limited, motivating weakly aligned data that can support competitive performance and zero-shot learning.
- Multimodal pretraining: Pretext objectives include masking, describing, matching, and ordering, and can be combined or applied directly to downstream tasks.
- Challenges: BERT-style discriminative pretraining does not directly support generation, and common multimodal pretraining models can underperform on fine-grained instance-level tasks.
5 CHALLENGES AND DESIGNS
The survey frames multimodal Transformer design around seven challenges and reviews architectural, training, and efficiency strategies for addressing them.
- The seven surveyed challenges are fusion, alignment, transferability, efficiency, robustness, universalness, and interpretability.
- Fusion: Multimodal fusion occurs at input, intermediate-representation, or prediction levels, with one-stream models using attention and masking for modality integration.
- Alignment: Contrastive learning commonly maps paired modalities into a shared representation space, supporting zero-shot transfer but making fine-grained alignment more computationally costly.
- Transferability: Transferability is constrained by distribution gaps, dataset bias, cross-task workflows, missing modalities, and cross-lingual differences.
- Robustness: Adversarial training, augmentation, and fine-grained losses are used to improve multimodal Transformer robustness.
- Efficiency: Self-attention creates quadratic time and memory complexity with sequence length, while large models require substantial data and high-dimensional multimodal representations intensify computation.
- Efficiency: Efficiency strategies include distillation, model simplification, token dropping, asymmetric capacities, and improved use of training samples.
- Universalness: Universal multimodal Transformers seek compatibility across modalities, alignment conditions, tasks, and few-shot or zero-shot settings, but current solutions remain preliminary.
6 DISCUSSION AND OUTLOOK
The outlook identifies task-agnostic universal models, efficient semantic alignment, noise-aware training, and a clearer account of Transformer strengths as open research directions.
- Designing one universal model that excels across unimodal and multimodal tasks remains difficult because task-specific architectures can be more efficient.
- Fine-grained multimodal learning needs latent semantic alignment, but detector-based semantic parts are complex, error-prone, and computationally costly.
- Expanding scale produces noisy and heterogeneous data, while longer multimodal inputs make Transformer quadratic complexity more acute.
- Reported Transformer strengths include implicit knowledge encoding, multiple modelling sub-spaces from multi-head attention, parallel sequence computation, permutation invariance, and flexible tokenization.
7 CONCLUSION
The survey reviews Transformer designs and training for multimodal learning, summarizes challenges and solutions, and discusses open problems and future research directions.
- The survey provides an overview intended for new researchers, practitioners, and experts in multimodal learning and Transformer design.
APPENDIX
The appendix establishes notation conventions and catalogs special/customized tokens used as semantic placeholders in uni-modal and multimodal Transformers.
- Table 4 defines the notation and abbreviations used throughout the survey.
- Special/customized tokens serve as semantic placeholders in token sequences across uni-modal and multimodal Transformers.
- Table 5 summarizes common special/customized tokens used in the survey.