Source-linked AI summary
MM-LLMs: Recent Advances in MultiModal Large Language Models
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, Dong Yu
TL;DR
MM-LLMs address the computational cost of training multimodal models from scratch and the challenge of connecting separately pretrained modality models. This survey synthesizes their architectures and training pipelines, classifies 126 SOTA models, reviews benchmarks and training recipes, and identifies future directions while tracking developments online. It reports trends toward broader modality conversion, refined alignment, higher-quality data, and more efficient architectures.
Problem
Training multimodal models from scratch is computationally expensive, while separately pretrained modality models must be effectively connected for collaborative inference.
Method
The survey formulates five-component architectures and two-stage training pipelines, then synthesizes 126 SOTA MM-LLMs, benchmarks, training recipes, and future directions.
Results
The survey identifies field trends toward specific-modality and any-to-any generation, progressively refined training, diversified modalities, higher-quality data, and more efficient architectures.
Takeaways & Limitations
The survey provides a structured reference for understanding MM-LLM formulations and supports continued research through benchmark analysis, training recipes, and a real-time tracking website.
Takeaways & Limitations
Page limits prevent detailed coverage of all technical details, and the survey may not fully capture recent advances in this dynamic field.
Abstract
from arXiv · showhide
In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategies. The resulting models not only preserve the inherent reasoning and decision-making capabilities of LLMs but also empower a diverse range of MM tasks. In this paper, we provide a comprehensive survey aimed at facilitating further research of MM-LLMs. Initially, we outline general design formulations for model architecture and training pipeline. Subsequently, we introduce a taxonomy encompassing 126 MM-LLMs, each characterized by its specific formulations. Furthermore, we review the performance of selected MM-LLMs on mainstream benchmarks and summarize key training recipes to enhance the potency of MM-LLMs. Finally, we explore promising directions for MM-LLMs while concurrently maintaining a real-time tracking website for the latest developments in the field. We hope that this survey contributes to the ongoing advancement of the MM-LLMs domain.
1 Introduction
MM-LLMs reduce the cost of multimodal research by augmenting pretrained unimodal foundation models, especially LLMs, while connecting modalities for collaborative inference. This survey organizes the field's architectures, training pipeline, 126 SOTA models, benchmark performance, training recipes, and future directions.
- Traditional multimodal models incur substantial computational costs when trained from scratch as model and dataset scales expand.
- MM-LLMs capitalize on pretrained unimodal foundation models, especially LLMs, to mitigate computational expenses and enhance multimodal pre-training.
- MM-LLMs use LLMs for language generation, zero-shot transfer, and in-context learning while relying on other modality models for high-quality representations.
- Their core challenge is effectively connecting separately pretrained LLMs and other modality models to enable collaborative inference.
- The survey presents general architecture and training formulations, a taxonomy of 126 MM-LLMs, benchmark reviews, training recipes, future directions, and a real-time tracking website.
2 Model Architecture
MM-LLM architectures connect modality-specific encoders and generators to an LLM through input and output projectors. Design choices span lightweight linear or MLP projectors, more complex attention-based modules, parameter-efficient tuning, and pretrained modality generators.
- The general architecture comprises five components: Modality Encoder, Input Projector, LLM Backbone, Output Projector, and Modality Generator.
- Modality Encoder: Modality encoders transform image, video, audio, 3D, and other inputs into features, with ImageBind offering a unified encoder for six modalities.
- Input Projector: Input Projectors align non-text features with the text feature space and feed the resulting prompts into the LLM alongside textual features.
- Input Projector: Input Projectors range from Linear Projectors and MLPs to Cross-attention, Q-Former, P-Former, and MQ-Former implementations.
- LLM Backbone: The LLM Backbone performs multimodal understanding, reasoning, and decision-making, producing textual outputs and signal tokens that guide multimodal generation.
- Output Projector: Output Projectors map LLM signal-token representations into features understood by modality generators, while PEFT methods can add fewer than 0.1% of total LLM parameters.
- Modality Generator: Modality Generators commonly use pretrained latent diffusion models, conditioned on projected features to synthesize images, videos, or audio.
3 Training Pipeline
MM-LLM training has two principal stages: multimodal pre-training aligns modalities using X-Text data, and multimodal instruction tuning adapts models to instructions and human intent. The survey distinguishes SFT and RLHF within instruction tuning and catalogs relevant datasets.
- The MM-LLM training pipeline consists of two principal stages: MM PT and MM IT.
- MM PT: During MM PT, Input and Output Projectors are trained on X-Text datasets to align modalities through predefined objectives.
- MM PT: X-Text datasets include image-text, video-text, and audio-text data, with image-text data represented as pairs or interleaved corpora.
- MM IT: MM IT fine-tunes pretrained MM-LLMs on instruction-formatted datasets so models can generalize to unseen tasks and improve zero-shot performance.
- MM IT: MM IT includes supervised fine-tuning and reinforcement learning from human feedback to align models with human intents and improve interaction capabilities.
4 SOTA MM-LLMs
The survey classifies 126 SOTA MM-LLMs by functional and design perspectives, comparing their architectures and training scales. It identifies trends toward broader modalities, improved alignment, higher-quality data, and simpler architectures.
- 126 SOTA MM-LLMs are classified from both functional and design perspectives.
- 43 mainstream MM-LLMs are compared by architecture and MM PT/MM IT dataset scales.Table 1 summarizes input-to-output modalities, modality encoders, and training dataset sizes.
- MM-LLM development progresses from understanding toward modality-specific generation and any-to-any modality conversion.The survey gives the progression MiniGPT-4 → MiniGPT-5 → NExT-GPT as an example.
- Training pipelines advance from MM PT to supervised fine-tuning and RLHF to improve human-intent alignment and conversational interaction.
- Recent systems diversify modalities, improve training data quality, and replace complex Q- and P-Former projectors with simpler linear projectors.The survey cites X-LLM, LLaVA-1.5, and VILA as examples of these trends.
5 Benchmarks and Performance
The survey compares major MM-LLMs across 18 vision-language benchmarks, focusing on OKVQA, IconVQA, VQAv2, and GQA. It also reviews training recipes involving resolution and token-sequence management.
- Major MM-LLMs are compared across 18 VL benchmarks, with detailed analysis focused on OKVQA, IconVQA, VQAv2, and GQA.Table 2 marks the highest and second-highest results using red and blue, respectively.
- MiniGPT-v2 and MiniGPT-v2-chat perform best on OKVQA and IconVQA, indicating strong reasoning, perception, and cognitive-reasoning capabilities.OKVQA tests commonsense, world, and visual knowledge; IconVQA tests diagram comprehension and holistic reasoning.
- VILA-13B performs best on VQAv2, demonstrating multimodal comprehension and resistance to language biases in acquired knowledge.
- LLaVA-1.5 and VILA-7B perform best on GQA, which uses scene graphs and compositional questions requiring structured logical reasoning.
- Higher image resolution supplies more visual detail but creates longer token sequences and additional training and inference costs.Reported resolutions include 336 × 336, 448 × 448, and methods supporting up to 1300 × 800 or 2560 × 2560.
6 Future Directions
The survey identifies future directions spanning broader modalities and benchmarks, safer and more capable models, lightweight deployment, embodied intelligence, and continual learning. It also highlights hallucination, bias, and autonomy challenges.
- Future MM-LLMs should support additional modalities, diverse LLM types and sizes, and higher-quality, more varied MM instruction-tuning datasets.Examples of additional modalities include web pages, heat maps, and figures and tables.
- More challenging, larger-scale benchmarks should include additional modalities and avoid overlap with pretraining or instruction-tuning tasks.Existing efforts include multipanel VQA, cross-style visual evaluation, and studies of optical character recognition.
- Lightweight implementations are needed for resource-constrained platforms such as low-power mobile and IoT devices.MobileVLM downscales LLaMA and uses a downsample projector with fewer than 20 million parameters.
- Embodied MM-LLMs connect perception, planning, and control, but further work is needed to improve robot autonomy.
- Continual learning is needed because massive training costs make frequent retraining impractical while new skills and knowledge remain necessary.
- Hallucinations can arise from biases, annotation errors, and semantic drift associated with paragraph separators, motivating mitigation methods.
- MM-LLMs can reproduce training-data biases, causing stereotypes and representational harm to vulnerable populations.Suggested responses include bias benchmarks and more fine-grained alignment methods.
7 Conclusion
The paper surveys recent MM-LLM advances through architectural and training-pipeline formulations, SOTA model taxonomy, benchmark capabilities, and future directions. It aims to provide insights supporting continued research in the field.
- The survey organizes MM-LLM architectures into five components and presents general design formulations and training pipelines.It also introduces SOTA MM-LLMs distinguished by their specific formulations.
- The survey reviews MM-LLM capabilities across diverse benchmarks and considers future developments in the rapidly evolving field.
Social Impact
MM-LLMs may improve accessibility, education, and media experiences, but their adoption also raises privacy, bias, and employment concerns requiring responsible deployment.
- MM-LLMs may improve accessibility through better voice recognition and visual aids, supporting more equal access to information.
- In education, MM-LLMs may enable more interactive learning experiences tailored to diverse learning styles.
- In media, MM-LLMs may create more engaging content and enrich consumer experiences.
- Widespread adoption raises privacy and data-security concerns, alongside risks of biased outputs and job displacement.
Limitations
The survey synthesizes the rapidly evolving MM-LLM landscape but acknowledges incomplete coverage and limited technical detail, supported by a real-time tracking website.
- The survey may omit some aspects because the MM-LLM field is dynamic and recent advances may not be fully captured.
- Page limits prevent detailed treatment of all technical details, so mainstream MM-LLM contributions receive concise overviews.
- A crowdsourced website provides ongoing updates intended to address the survey’s coverage constraints over time.
A Related Surveys
Earlier surveys often focused on traditional or narrowly scoped MM research, whereas this survey broadens coverage, formalizes architecture, and summarizes development and training practices.
- Prior survey limitations: Earlier MM pre-training surveys commonly incurred substantial computational costs and omitted LLM-based instruction following, ICL, CoT, and interaction capabilities.
- Prior survey scope: Recent surveys separately emphasize early vision-language understanding, visual instruction tuning, modal alignment, or autonomous-driving applications.
- Survey coverage: The survey covers around 120 or more MM-LLMs, including understanding and generative models across vision-language, audio, and 3D point-cloud modalities.
- Architecture: Its general architecture accommodates any-to-any modality transformations and details component roles and implementation choices.
- Training synthesis: The survey summarizes MM-LLM development trends and training recipes intended to enhance effectiveness.
- Research infrastructure: An open-source website supports crowdsourced updates and collaboration among MM-LLM researchers.
- Modality encoders: Mainstream modality encoders reviewed include NFNet-F6 and ViT for visual processing, CLIP ViT and Eva-CLIP ViT for vision-language learning, C-Former for audio, and HuBERT and BEATs for speech or audio representations.
C Mainstream PEFT Methods
Parameter-efficient fine-tuning adapts frozen or largely preserved LLMs with a small number of trainable parameters, while representative MM-LLMs connect pretrained modality models to LLMs across diverse tasks.
- PEFT overview: PEFT keeps the pretrained LLM frozen while adjusting a small number of additional trainable parameters.
- Prefix-tuning: Prefix-tuning adds learnable prefix tokens to the keys and values of an attention module.The prefixes are represented by Pk and Pv, which are concatenated with the attention inputs.
- Adapter: Adapters use a residual block with down-projection A, nonlinear activation σ(·), and up-projection B, inserted into pretrained LLM layers.Their module output is expressed as h′ = h + σ(xA)B.
- LoRA: LoRA models parameter changes in a low-rank space by decomposing ∆W into AB, with r ≪ min(c, d).QLoRA quantizes pretrained weights to 4 bits before applying LoRA-based PEFT.
- Representative MM-LLMs: Representative MM-LLMs span general-purpose, image-text, video, audio, and multi-modality systems using bridges such as Q-Former, linear alignment, and cross-modal pretraining.PandaGPT handles text, image/video, audio, thermal, depth, and inertial measurement units, while Video-LLaMA aligns vision and audio with language.
G Training Dataset
The survey summarizes datasets used for multimodal pretraining and instruction tuning, covering modality-specific data quantities and input-to-output modality combinations.
- Dataset overview: The survey presents MM PT and MM IT dataset statistics in Tables 3 and 4, respectively.
- MM PT datasets: Table 3 reports statistics for multimodal pretraining datasets, including quantities of Image, Video, Audio, Text, and X-Text pairs.Here, X denotes Image, Video, or Audio, and X-Text denotes corresponding modality-text pairs.
- MM IT datasets: Table 4 reports statistics for multimodal instruction-tuning datasets using input-to-output modality combinations.Its notation includes Text, Image, Video, Audio, Bounding box, Point Cloud, Table, and Web page modalities.