The Revolution of Multimodal Large Language Models: A Survey

Davide Caffagni; Federico Cocchi; Luca Barsellotti; Nicholas Moratelli; Sara Sarto; Lorenzo Baraldi; Lorenzo Baraldi; Marcella Cornia; Rita Cucchiara

Source-linked AI summary

The Revolution of Multimodal Large Language Models: A Survey

Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara

arXiv:2402.12451v2Feb 19, 2024cs.CVcs.AIcs.CLcs.MM

TL;DR

MLLM research needs a consolidated account of how visual and textual modalities are integrated across rapidly evolving models. This survey analyzes architectures, alignment and training strategies, tasks, datasets, benchmarks, performance, and computational requirements, while identifying challenges and future directions.

Problem
Rapidly expanding MLLM research requires a comprehensive account of visual models, their components, tasks, benchmarks, performance, and computational requirements.
Method
The survey synthesizes visual MLLMs by examining their architectures, training methodologies, tasks, datasets, evaluation benchmarks, performance, and computational requirements.
Results
The survey provides a comprehensive overview of recent visual-based MLLMs, including visual understanding, visual grounding, image generation and editing, and domain-specific applications.
Takeaways & Limitations
The review identifies open challenges and promising future research directions for further empowering MLLMs.
Takeaways & Limitations
The review may miss minor works, incompletely cover non-visual MLLMs, and omit technical details because of space constraints.

Abstract

from arXiv · show

Connecting text and visual modalities plays an essential role in generative intelligence. For this reason, inspired by the success of large language models, significant research efforts are being devoted to the development of Multimodal Large Language Models (MLLMs). These models can seamlessly integrate visual and textual modalities, while providing a dialogue-based interface and instruction-following capabilities. In this paper, we provide a comprehensive review of recent visual-based MLLMs, analyzing their architectural choices, multimodal alignment strategies, and training techniques. We also conduct a detailed analysis of these models across a wide range of tasks, including visual grounding, image generation and editing, visual understanding, and domain-specific applications. Additionally, we compile and describe training datasets and evaluation benchmarks, conducting comparisons among existing models in terms of performance and computational requirements. Overall, this survey offers a comprehensive overview of the current state of the art, laying the groundwork for future MLLMs.

1 Introduction

Multimodal Large Language Models extend language models to visual inputs and outputs through connected vision-language architectures. This survey reviews their architecture, training, tasks, datasets, benchmarks, performance, computational requirements, challenges, and future directions.

MLLMs merge vision and language architectures through vision-to-language adapters and training approaches designed for modality alignment and instruction following.
The survey focuses on visual MLLMs, covering their architecture, training methodologies, and designed tasks.
It examines vision encoders, adapter modules, training data, training processes, and the range of tasks addressed by MLLMs.
The survey concludes by discussing persistent challenges and promising directions for future MLLM research.
The review addresses visual grounding, image generation and editing, and compares reported performance and hardware requirements using benchmarks and quantitative results.

2 Empowering LLMs with Multimodal Capabilities

MLLMs combine language models, visual encoders, and adapters, with training strategies that align visual features to language and support multimodal instruction following. The survey compares these architectural and training choices, including their capabilities and computational trade-offs.

Preliminaries: MLLMs contain an LLM backbone, one or more visual encoders, and vision-to-language adapter modules.
Pre-Training of Model Components: Visual encoders commonly use CLIP-based architectures, while language models primarily use Transformers and sometimes Mamba architectures.
Visual Encoder: Stronger image encoders improve performance, motivating ensembles of frozen backbones and visual backbones scaled to billions of parameters.
Visual Encoder: Frozen visual encoders make large models feasible but can limit visual-language alignment and produce lengthy feature sequences with high computation.
Vision-to-Language Adapters: Adapters range from linear or MLP projections to Q-Former and added cross-attention layers that connect visual and textual representations.
Multimodal Training: MLLM training uses single-stage or two-stage autoregressive next-token prediction, with visual instruction tuning as a prominent multimodal paradigm.

3 Tackling Visual Tasks with MLLMs

MLLMs have expanded from visual understanding into fine-grained grounding, visual generation, video, multimodal-input, and domain-specific applications. These capabilities rely on task-specific representations, adapters, datasets, and integrations with diffusion models.

Standard MLLMs support visual question answering, captioning, and multi-turn conversation, while newer systems address visual grounding and image generation.
Visual Grounding: Visual grounding covers referring and grounding, including region captioning, referring expression generation, referring expression comprehension, and related localization tasks.
Visual Grounding: Region-as-text methods represent bounding boxes or polygons as coordinate sequences or special location tokens in generated text.
Visual Grounding: Embedding-based and text-to-grounding approaches instead use region encoders, segmentation or detection decoders, open-vocabulary categories, and models such as Grounding-DINO and SAM.
Image Generation and Editing: Visual generation commonly integrates MLLMs with Stable Diffusion, whose denoising U-Net is conditioned on textual or visual embeddings through cross-attention.
Image Generation and Editing: GILL maps frozen LLM output embeddings to frozen diffusion-model conditioning embeddings, whereas other methods fine-tune the LLM, diffusion components, or both through reconstruction, CLIP, or score-distillation objectives.
Video Understanding: Video MLLMs extract frame-level features with CLIP-based backbones and combine them using pooling, Q-Former, linear projections, or video-specific adapters.
Any-Modality Models and Domain-Specific MLLMs: Any-modality models align multimodal features with Transformer blocks such as Q-Former and Perceiver, while domain-specific MLLMs target applications including document analysis and embodied AI.

4 Conclusion and Future Directions

The survey synthesizes how MLLMs acquire multimodal capabilities and the tasks they address, then identifies unresolved challenges for future research. It highlights under-explored multimodal retrieval augmentation and the need to address hallucinations and harmful or biased generation.

The survey reviews how LLMs are equipped with multimodal capabilities and the main tasks addressed by MLLMs.
Multimodal Retrieval-Augmented Generation: Multimodal retrieval-augmented generation remains under-explored, although emerging visual-question-answering datasets require external retrieved knowledge.
Correction of Hallucinations: MLLMs exhibit high hallucination rates, especially when generating longer captions, and correcting their underlying causes remains an open challenge.
Prevent Harmful and Biased Generation: Preventing harmful and biased generation is identified as an important safety and fairness challenge for models trained on web-crawled data.

Limitations

The survey’s scope is centered on visual-based MLLMs, and its coverage and technical detail are constrained by omissions and space limits. High-resolution and arbitrary-aspect-ratio processing also introduces adaptation and computational concerns.

The review may omit minor works and does not completely cover MLLMs focused on modalities other than vision.
Space constraints limited explanations to the most relevant novelty points, so readers should consult original papers for technical and implementation details.
Most MLLMs process images at low resolution and fixed square aspect ratios, while higher-resolution backbones can improve fine-grained understanding and reduce hallucinations.
Positional-Encoding Interpolation: Interpolating positional encodings from low-resolution visual backbones can cause adaptation issues, partly requiring a high-resolution training stage.
Sub-Images Slicing: Sub-image slicing processes fixed-size patches independently alongside a downsized whole image, concatenating their features to form a global representation.
Others: Alternative methods include directly feeding 30 × 30 pixel patches to the LLM or fusing CNN high-resolution features with ViT low-resolution features.

B Additional Training Data

Training data for specialized MLLM capabilities is matched to the desired output: grounding datasets provide region-level supervision, while generation and editing datasets pair images with captions or instructions.

Visual Grounding: Visual grounding uses task-specific instruction-formatted data, including CoinIt’s 14 converted benchmarks spanning coarse single-image to region-level multi-image tasks.
Visual Grounding: Additional instruction tuning can preserve conversational capabilities, while custom datasets combine regions and captions or use detectors to ground noun chunks and reconstruct referring expressions.
Image Generation and Editing: Image generation requires image-caption datasets such as LAION-400M, COYO-700M, and COCO, while interleaved text-image generation uses MMC4, OBELICS, and VIST.
Image Generation and Editing: Image editing typically uses specialized datasets such as InstructPix2Pix and MagicBrush.

C Evaluation

MLLM evaluation spans standard visual comprehension and recognition benchmarks alongside newer tests of multimodal conversation capabilities.

Table 4 compares common MLLMs on standard VQA and image-captioning datasets.It also includes benchmarks specifically designed for MLLM evaluation.
The survey reports datasets and benchmarks used to assess MLLMs across visual understanding and conversational abilities.

C.1 Standard Benchmarks

Standard benchmarks evaluate image-question answering, text reading, visual reasoning, and image captioning using task-specific datasets and metrics.

VQA benchmarks measure answer accuracy from images, with VQAv2 discouraging reliance on language priors through contrasting image examples.GQA emphasizes scene understanding and compositionality, while OKVQA requires answers not completely available in the image.
107,439 questions make IconQA a benchmark for visual reasoning over abstract icons and textbook-style problems.Its question types include multiple-image-choice, multiple-text-choice, and fill-in-the-blank.
TextVQA and OCR-VQA test reading text in images, with TextVQA containing 5,734 test examples and OCR-VQA comprising 207,572 book-cover images.OCR-VQA includes more than 1M question-answer pairs.
Image captioning evaluations commonly use CIDEr to compare generated descriptions with ground-truth captions.COCO, Flickr30k, and nocaps provide complementary image-captioning settings, including novel-object captioning in nocaps.
145k captions aligned with 28k images make TextCaps an OCR-intensive image-captioning benchmark.The task requires recognizing text in images while describing the entire visual content.

C.2 MLLM-Specific Benchmarks

MLLM-specific benchmarks extend evaluation toward hallucination, multimodal communication, broad capability coverage, mathematical reasoning, and domain-specific knowledge.

MLLM-specific benchmarks demand strong perception, cognition, and often deep domain-specific knowledge beyond standard datasets.Several works use proprietary models such as ChatGPT or GPT-4 to automatically judge answers.
8,910 binary queries make POPE a benchmark for object hallucination across random, popular, and adversarial subsets.
MME assesses multimodal communication through 14 comprehension and manipulation tasks involving quantification, spatial determination, and color identification.
MMBench uses approximately 3,000 multiple-choice questions across 20 domains organized by perception, reasoning, localization, and attribute inference.
SEED-Bench covers 12 dimensions with 19k human-written multiple-choice questions spanning scene understanding, OCR, and action recognition.Other benchmarks target dialogue, multi-capability evaluation, mathematical reasoning, and university-level multimodal knowledge.

C.3 Visual Grounding Evaluation

Visual grounding evaluation covers referring, localization, segmentation, grounded captioning, and grounded conversation, using task-specific metrics and increasingly realistic datasets.

Visual grounding includes region captioning, referring expression generation, region-level question answering, REC, RES, and grounded captioning.
REC uses Acc@0.5, RES uses cIoU, and region captioning commonly uses METEOR and CIDEr.Some newer benchmarks evaluate grounded conversations involving multiple rounds.
RefCOCO, RefCOCO+, and RefCOCOg provide referring expressions linked to objects in COCO images for comprehension evaluation.RefCOCOg contains 85,474 expressions for 54,822 objects in 26,711 images.
Visual Genome and Visual7W connect image regions with language for region-level captioning, question answering, and object-level grounding.Visual Genome contains 5.4M region descriptions and 1.7M visual question-answer pairs, while Visual7W contains 328k question-answer pairs.
Grounding datasets increasingly use automated pipelines to associate noun phrases, attributes, relationships, or natural-language descriptions with regions or masks.GRIT contains 91M images, 115M text spans, and 137M associated bounding boxes; MUSE contains 246k question-answer pairs averaging 3.7 targets per answer.

C.4 Image Generation and Editing Evaluation

Image generation and editing are evaluated with complementary benchmarks and metrics that assess realism, image similarity, and text-image alignment.

Metrics: FID measures the congruence between generated-image and real-image distributions, with lower scores indicating better visual quality and realism.The survey presents FID as the reference metric for generated-image evaluation.
Benchmarks: DreamBench is used to report subject-driven image generation results.
Metrics: CLIP-I and DINO compare generated and ground-truth images using cosine similarity from their respective visual backbones.
Metrics: CLIP-T measures image-text alignment by computing cosine similarity between input captions and generated images encoded with CLIP.

D Computational Requirements

The survey compares the hardware and training time required by common MLLMs, revealing a wide computational range from a few hundred to several hundred thousand GPU hours.

Hardware: Table 11 summarizes the hardware required to train common MLLMs, including the type and number of GPUs or TPUs.
Hardware: Most MLLMs distribute training across 8 A100 GPUs.
Hardware: Flamingo and PaLI are exceptions to the predominant use of A100 or A6000 GPUs because they are trained on large numbers of TPUs.
Training time: Flamingo, PaLI, and IDEFICS require a few hundred thousand GPU training hours, whereas LLaVA requires only a few hundred.Figure 2 converts one TPU hour to approximately 1.5 GPU hours.

E Additional Details on Other Modalities and Applications

The survey extends beyond image-based MLLMs to video, 3D, audio, multimodal generation, interactive tool use, and domain-specific applications, cataloguing their architectures, training choices, and capabilities.

Video Understanding: Video-based MLLM summaries report the starting LLM, visual encoder, tasks, capabilities, and whether the LLM is frozen or fine-tuned.Fine-tuning may be full or use parameter-efficient strategies.
3D Understanding: 3D MLLMs address 3D VQA, conversation, and dense captioning by projecting 3D data into the LLM representation space.3D-LLM renders scenes from multiple views and uses EVA-CLIP features connected to a fine-tuned BLIP-2 model.
Any-Modality Models: Any-modality models connect modality-specific encoders to a shared LLM space through dedicated projections or interfaces such as Q-Former and C-Former.These approaches cover image, video, and speech inputs.
Any-Modality Models: Some systems generate multiple output modalities by using signal or special tokens to activate and condition modality-specific decoders.Examples include image, video, audio, surface-normal, depth, and segmentation-mask generation.
Interactive and Compositional Systems: Interactive and compositional systems use an LLM to select visual tools for complex tasks, with some open-source models fine-tuned through PEFT.LLaVA-Plus directly uses the query image during tool selection and invocation through an instruction-following tool-use dataset.
Domain-Specific MLLMs: Domain-specific MLLM summaries identify each model’s starting LLM and main characteristics, while related systems support mouse-based pointing interactions on images or videos.