A Survey on Multimodal Large Language Models

1 INTRODUCTION

• MLLMs address the complementarity between text-only LLMs with strong reasoning abilities and vision models that perceive images but commonly lag in reasoning.Scaling enables instruction following [5], [6], In-Context Learning, and Chain of Thought [8] in LLMs. Earlier multimodal approaches include discriminative CLIP, which aligns visual-textual representations, and generative OFA, which unifies tasks through sequence-to-sequence modeling.Source: §1 · p. 1
• MLLMs differ from traditional multimodal models through billion-scale LLM backbones and training paradigms such as multimodal instruction tuning [19], [20].Their demonstrated capabilities include generating website code from images [21], interpreting memes’ deeper meanings [22], and OCR-free mathematical reasoning [23].Source: §1 · p. 1
• Research extends MLLMs toward finer-grained interaction, broader input and output modalities, additional languages, specialized domains, and real-world agents.Examples include box-based [28] and click-based [29] prompts, multimodal outputs through NExT-GPT [32], and Chinese-language support [33], [34]. Applications encompass medical images [35],, [37], document parsing [38], [39], [40], embodied agents [41], [42], and GUI agents [43], [44], [45].Source: §1 · p. 1
• The survey synthesizes MLLM concepts, methods, and progress, emphasizing vision and language while also covering video and audio.The authors describe it as the first MLLM survey to their knowledge and maintain an accompanying GitHub collection updated in real time.Source: §1 · p. 2
• The survey reviews architectures, training strategies and data, and evaluation before examining capability extensions and mitigation of multimodal hallucination.It concludes with a summary and potential research directions.Source: §1 · p. 2
• The survey examines M-ICL for inference-time few-shot performance, M-CoT for complex reasoning, and LLM-based systems for composite reasoning tasks and user queries.These techniques are presented as approaches suited to distinct scenarios.Source: §1 · p. 2

2 ARCHITECTURE

• Typical MLLMs combine a pretrained modality encoder, a pretrained LLM, and a modality interface, optionally adding a generator for non-text outputs.The encoder processes incoming signals, while the interface aligns modalities for the LLM. Pretrained LLMs contribute world knowledge, generalization, and reasoning capabilities.Source: §2 · p. 2, §2.2 · p. 3

2.1 Modality encoder

• Pre-aligned encoders compress raw inputs into compact representations that are easier to align with LLMs, as exemplified by CLIP.Encoder-free Fuyu-8b instead projects image patches directly, supporting flexible resolutions. Beyond images, Pengi uses CLAP for audio, while ImageBind-LLM [30] uses ImageBind for image, text, audio, depth, thermal, and IMU inputs.Source: §2.1 · pp. 2–3
• Input resolution matters more than encoder parameter size or training-data composition [52], and higher resolutions improve performance [34], [50], [51], [52].Direct scaling uses higher-resolution inputs, sometimes requiring encoder tuning or replacement; CogAgent [44] instead connects high- and low-resolution encoders through cross-attention. Patch-division methods such as Monkey [51] and SPHINX [53] combine image patches with a downsampled image to capture local and global features.Source: §2.1 · p. 3

2.2 Pre-trained LLM

• 7B-to-13B LLM scaling improves performance across benchmarks in Liu et al. [50],.A 34B LLM also exhibits zero-shot Chinese capability despite training with only English multimodal data [50],. Conversely, MobileVLM [63], [64] uses MobileLLaMA 1.4B/2.7B to enable efficient inference on mobile processors.Source: §2.2 · p. 3
• MM1 [52] and MoE-LLaVA [68] find that mixture-of-experts LLMs outperform dense counterparts on almost all evaluated benchmarks.Selective parameter activation enables sparse models to increase total parameter size without increasing computational cost.Source: §2.2 · p. 3

2.3 Modality interface

• Learnable connectors either concatenate encoder-derived tokens with text tokens or fuse visual and textual features inside the LLM.Token-level interfaces include BLIP-2’s [59] query-based compression and LLaVA’s linear MLP projections [20], [50]. Feature-level approaches include Flamingo’s inserted cross-attention layers and CogVLM’s [75] per-layer visual expert modules.Source: §2.3 · p. 4
• Token-level fusion outperforms feature-level fusion on VQA [73], while MM1 [52] finds visual-token count and input resolution more important than adapter type.Zeng et al. [73] suggest that cross-attention models might require more complicated hyperparameter searches to achieve comparable performance.Source: §2.3 · p. 4
• Expert models offer a training-free interface by translating multimodal inputs into language, but this conversion loses information and is less flexible than learnable connectors.VideoChat-Text [25] combines pretrained vision models with speech recognition to describe videos, but textual conversion distorts spatial-temporal relationships.Source: §2.3 · p. 4

• A full-fledged MLLM undergoes three training stages: pre-training, instruction-tuning, and alignment tuning.Source: §3 · p. 4
• Each training stage requires different types of data and fulfills different objectives.Source: §3 · p. 4
• The section examines training objectives, data collection, and data characteristics for each training stage.Source: §3 · p. 4

3.1.1 Training Detail

• Pre-training aligns modalities and builds multimodal world knowledge using large-scale caption pairs.Vision–text alignment typically uses autoregressive caption prediction with cross-entropy loss. Methods commonly freeze pretrained encoders and LLMs while training an interface [20], [35], [72], although others unfreeze additional modules for alignment [34], [81], [82].Source: §3.1.1 · p. 4

3.1.2 Data

• Coarse-grained captions are abundant but often short and noisy because they originate from web-image alt-text; CLIP similarity can filter mismatched pairs.LAION additionally filters image sizes and text lengths, deduplicates by URL, and uses CLIP embeddings to remove potentially illegal content.Source: §3.1.2 · p. 5
• 3.3M image-caption pairs comprise CC-3M, while CC-12M contains 12.4M pairs after relaxing and simplifying its collection pipeline.CC-3M filters images, text, and image–text consistency through a multistage cleaning pipeline. SBU Captions contains 1M Flickr image-text pairs, retaining descriptions based on length, predefined terms, and spatial-relation words.Source: §3.1.2 · p. 5
• 5.85B image-text pairs constitute multilingual LAION-5B, including a 2B English subset.LAION-COCO contains 600M images from that English subset, with synthetic captions generated by BLIP and selected using CLIP. COYO-700M contains 747M CommonCrawl image-text pairs, filtered for image and text quality, public-dataset overlap, and duplication.Source: §3.1.2 · p. 5
• Fine-grained captions generated by strong MLLMs [83], [91], [92] offer longer, more accurate descriptions but incur higher costs and smaller data volumes.These descriptions enable finer-grained image–text alignment. 100K GPT-4V-generated examples train the ShareGPT4V [83] captioner, which then scales the data volume to 1.2M.Source: §3.1.2 · pp. 5–6

3.2.1 Introduction

• Instruction tuning teaches models to understand user instructions and generalize to unseen tasks, improving zero-shot performance rather than fitting individual tasks.Unlike task-specific supervised fine-tuning or specialized prompting, instruction tuning emphasizes generalization across tasks.Source: §3.2.1 · p. 6

3.2.2 Training Detail

• Multimodal instruction samples contain an optional task description, an image with optional text, and a target response; templates also support multi-round conversations [20], [37], [71], [98].Inputs can therefore accommodate both image-only captioning and image–question answering.Source: §3.2.2 · p. 6, Table 5 · p. 6
• MLLM instruction tuning typically retains the LLM autoregressive objective, predicting each response token from the instruction, multimodal input, and preceding tokens [20], [37], [71], [101].Training examples are represented as (I, M, R): instruction, multimodal input, and ground-truth response.Source: §3.2.2 · p. 6

3.2.3 Data Collection

• Instruction data collection uses adaptation, self-instruction, and data mixture; adding language-only conversations can improve conversational and instruction-following abilities [81], [98], [101], [103].Adaptation uses manually designed or GPT-expanded task instructions. LaVIN [101] samples language-only and multimodal data per minibatch; MultiInstruct [102] explores shuffled mixtures and text-first sequential tuning.Source: §3.2.3 · pp. 6–7
• Concise answers in existing VQA and caption datasets can limit MLLM response length; explicit length instructions or answer expansion offer remedies.ChatBridge [104] explicitly labels short-answer and single-sentence examples. M3IT [105] expands answers by prompting ChatGPT with the original question, answer, and image context such as captions and OCR.Source: §3.2.3 · pp. 6–7
• Self-instruction uses a few handcrafted demonstrations to guide LLM-generated instruction samples [106], addressing real-world needs such as multi-round conversations.LLaVA [20] supplies captions and bounding boxes to text-only GPT-4 to construct LLaVA-Instruct-150k. LVIS-Instruct4V [91] and ALLaVA [92] instead use GPT-4V for higher-quality generation.Source: §3.2.3 · p. 7

3.2.4 Data Quality

• Smaller, cleaner datasets can outperform larger, noisier ones [73], while higher-quality instruction-tuning data can improve performance despite reduced quantity [108].Wei et al. [108] propose quality metrics and automatic filtering of inferior vision-language data. Lynx [73] also finds that diverse prompts improve performance and generalization.Source: §3.2.4 · p. 7
• Visual reasoning training improves performance more than captioning or QA, while instruction complexity may matter more than task diversity or fine-grained spatial annotations [109].These findings concern the tasks and instruction characteristics represented in training data.Source: §3.2.4 · p. 7

3.3.1 Introduction

• Alignment tuning targets human preferences, including fewer hallucinations, primarily through RLHF and DPO.Collecting response-preference feedback is generally expensive, so alignment typically uses less data than preceding training stages.Source: §3.3.1 · p. 7, §3.3.3 · p. 8

3.3.2 Training Detail

• RLHF [110], combines supervised fine-tuning, reward modeling from human preference pairs, and PPO-based policy optimization.Supervised fine-tuning can be skipped when starting from an instruction-tuned model. Policy and reference models typically start from the supervised model, while a per-token KL penalty limits policy deviation.Source: §3.3.2 · p. 8
• DPO learns human preferences through binary classification loss without an explicit reward model, simplifying training to preference-data collection and preference learning.Unlike PPO-based RLHF, DPO does not require learning a separate reward model.Source: §3.3.2 · p. 8
• LLaVA-RLHF [112] applies human preference tuning to LLaVA [20] to reduce hallucinations, while RLHF-V [114] and Silkie [115] use DPO.RLHF-V [114] applies dense DPO using segment-level preference pairs obtained by correcting hallucinations. Silkie [115] collects preferences by prompting GPT-4V and distills that supervision into an instruction-tuned model through DPO.Source: §3.3.2 · p. 8

3.3.3 Data

• 10K preference pairs form LLaVA-RLHF [112]; RLHF-V [114] contains 5.7K fine-grained human feedback data, and VLFeedback [115] contains more than 380K comparison pairs.LLaVA-RLHF measures honesty and helpfulness to reduce hallucinations, while RLHF-V collects segment-level hallucination corrections. VLFeedback uses GPT-4V scores for helpfulness, faithfulness, and ethical concerns.Source: §3.3.3 · p. 8

4.1 Closed-set

• Closed-set evaluation uses predefined answer options and benchmark metrics, with zero-shot testing on held-out datasets or tasks and finetuned evaluation often targeting domain-specific tasks.InstructBLIP [60] reports accuracy on ScienceQA and CIDEr on NoCaps and Flickr30K. Zero-shot evaluation tunes on held-in datasets before testing on unseen datasets or tasks.Source: §4.1 · pp. 8–9
• Dedicated benchmarks broaden evaluation beyond selected datasets, covering general capabilities, video understanding, and hallucination.MME [123] includes 14 perception and cognition tasks with manually designed instruction-answer pairs to avoid leakage; MMBench [124] uses ChatGPT to match open responses to predefined choices. Video-ChatGPT [130] and Video-Bench [131] provide video-specific assessment, while POPE [132] assesses hallucination.Source: §4.1 · p. 9

4.2 Open-set

• Open-set chatbot responses are harder to judge because their content is unrestricted; manual scoring uses handcrafted questions to assess specific capability dimensions.mPLUG-Owl [81] evaluates natural-image, diagram, and flowchart understanding. GPT4Tools [107] separately assesses finetuned and zero-shot responses in terms of thought, action, arguments, and the whole response.Source: §4.2 · p. 9
• GPT scoring reduces reliance on labor-intensive manual assessment, with LLaVA [20] using text-only GPT-4 to compare multimodal dialogue responses for helpfulness and accuracy.30 COCO validation images each receive short, detailed, and complex reasoning questions generated through GPT-4 self-instruction. GPT-4 then compares model answers with its own answers.Source: §4.2 · p. 9
• Text-only GPT-4 judges cannot inspect images directly [35], whereas Woodpecker [77] uses GPT-4V to assess answers against images.Reliance on captions or bounding box coordinates makes treating text-only GPT-4 as a performance upper bound questionable [35]. Direct image access is expected to improve evaluation accuracy with GPT-4V [77].Source: §4.2 · p. 9
• Case studies supplement scoring, with Fu et al. [137] finding comparable visual reasoning abilities between Gemini-Pro and GPT-4V despite different response styles.Yang et al. [135] qualitatively examine GPT-4V across basic skills and complex knowledge-and-reasoning tasks, including joke understanding and indoor navigation. Wen et al. [136] focus on automatic-driving scenarios.Source: §4.2 · p. 9

Granularity Support

• Finer-grained interaction extends MLLM inputs from images to regions and pixels, while grounded outputs progress from bounding boxes to masks.Shikra [28] supports region references expressed as natural-language bounding boxes and box-grounded responses; Ferret [141] uses hybrid representations for point, box, and sketch prompts. Osprey [29] uses a segmentation model [9] for single-click entity or part selection, while LISA [142] enables mask-level understanding, reasoning, and pixel-level grounding.Source: §5 · p. 9

Modality Support

• Modality extensions add 3D point-cloud inputs [41], [143], [144], and mixed text, image, audio, and video inputs and outputs through NExT-GPT [32].NExT-GPT [32] uses an encoder-decoder architecture with the LLM as the understanding and reasoning pivot, attaching diffusion models, to support multimodal generation.Source: §5 · pp. 9–10

Language Support

• 22.7% of Qwen-VL [34] pre-training data is Chinese, included to preserve the Chinese–English capabilities of its underlying bilingual LLM, Qwen [58].Multilingual extensions address predominantly unilingual models amid scarce high-quality non-English training corpora. VisCPM [33] uses English-pivot multistage training, a pretrained bilingual LLM, and translated instruction-tuning samples to transfer multimodal capabilities to Chinese.Source: §5 · p. 10

Scenario/Task Extension

• Scenario-specific extensions support resource-limited mobile deployment through MobileVLM [63] and GUI interaction through CogAgent [44], AppAgent [43], and Mobile-Agent [45].MobileVLM [63] uses smaller LLMs and quantization to accelerate computation on mobile devices. The GUI assistants plan and guide task execution step by step according to user requests.Source: §5 · p. 10
• Domain-specific extensions develop OCR-free document understanding through mPLUG-DocOwl [38], document-task training through TextMonkey [39], and medical image assistance through LLaVA-Med [158].mPLUG-DocOwl [38] tunes on varied document-level data, while TextMonkey [39] adds position-related tasks to reduce hallucinations and ground responses visually. LLaVA-Med [158] injects medical knowledge into LLaVA [20] for medical image understanding and question answering.Source: §5 · p. 10

6.1 Preliminaries

• Multimodal hallucination includes nonexistent objects, incorrect object attributes, and false relationships between objects [159].These errors make responses inconsistent with image content [77]. Attribute and relationship hallucinations depend on object existence.Source: §6.1 · p. 10, §6 · p. 10

6.2 Evaluation Methods

• CHAIR measures caption object hallucinations, POPE [132] tests binary object-existence choices, and MME [123] additionally evaluates count, position, and color.CHAIR measures proportions of sentences containing hallucinated objects or hallucinated objects among all mentioned objects. POPE converts responses through yes/no keywords and includes harder settings informed by data statistics to assess robustness.Source: §6.2 · p. 10
• Evaluation extends beyond matching through HaELM [161], Woodpecker [77], FaithScore [162], and AMBER [163].HaELM uses text-only LLM judges against reference captions, whereas Woodpecker uses GPT-4V to assess responses directly against images, avoiding limited image context and reference dependence. FaithScore evaluates descriptive sub-sentences separately; AMBER is an LLM-free benchmark covering discriminative and generative tasks across all three hallucination types.Source: §6.2 · p. 10

6.3 Mitigation Methods

• Pre-correction uses negative instruction tuning in LRV-Instruction and human-preference reinforcement learning in LLaVA-RLHF [112] to reduce hallucinated responses.LRV-Instruction combines positive instructions with negative instructions at different semantic levels. LLaVA-RLHF fine-tunes models using collected human-preference pairs.Source: §6.3 · p. 11
• In-process correction includes HallE-Switch’s [159] imagination control, VCD’s contrastive decoding, and HACL’s [166] contrastive representation learning.HallE-Switch hypothesizes that ungrounded objects originate in LLM knowledge and introduces a continuous inference-time control with corresponding training; VCD attributes hallucinations to corpus bias and language priors, countering them through amplify-then-contrast decoding after observing stronger prior reliance with image noise. HACL pulls paired cross-modal representations together while separating hallucinated from non-hallucinated text representations.Source: §6.3 · p. 11
• Post-correction repairs generated responses using Woodpecker’s [77] training-free expert-model pipeline or LURE’s [167] trained revisor.Woodpecker supplements image context and corrects hallucinations step by step, with inspectable intermediate results and image-grounded objects. LURE masks uncertain objects in descriptions and regenerates responses.Source: §6.3 · p. 11

7.1 Multimodal In-Context Learning

• M-ICL adds multimodal demonstrations at inference time, learning by analogy without training; demonstration number and ordering are adjustable, but models are arrangement-sensitive [168], [171].Instruction tuning can enhance ICL ability [19]. Table 9 illustrates two multimodal demonstrations followed by a query, using a template adapted from [98].Source: §7.1 · p. 11, Table 9 · p. 11

7.1.1 Improvement on ICL capabilities

• MIMIC-IT [172] improves few-shot captioning through multimodal-context instruction tuning, while Emu [173] and Sheng et al. [174] extend outputs to text and images.Emu [173] extends Flamingo with additional generation modalities and training data, using Stable Diffusion as a vision decoder for extra visual supervision. Sheng et al. [174] instead use unified quantization and a shared embedding layer rather than a specialized image encoder.Source: §7.1.1 · pp. 11–12
• Link-context learning [175], MMICL [176], and Jeong [177] strengthen contextual learning through contrastive image–description training, standardized interleaved contexts, and irrelevant-context filtering, respectively.Link-context learning [175] uses positive and negative pairs to strengthen causal image–label links, while MMICL [176] targets reasoning over multiple related images. Jeong [177] finds that even a small fraction of incoherent images or text can induce context-inconsistent responses.Source: §7.1.1 · p. 12

7.1.2 Applications

• M-ICL supports visual reasoning [22],,,, [180] and external-tool use [169],, [181], using task-level demonstrations versus executable step sequences.Visual reasoning demonstrations convey task intent and output format for similar questions. Tool-use demonstrations provide finer-grained sequential steps, connecting this application to CoT.Source: §7.1.2 · p. 12

7.2 Multimodal Chain of Thought

• M-CoT extends intermediate-step reasoning to multimodal tasks [184], [185], [186], [187], prompting models to produce the reasoning process alongside the final answer [8].CoT has demonstrated effectiveness on complex reasoning tasks [8], [182], [183].Source: §7.2 · p. 12

7.2.1 Learning Paradigms

• M-CoT is acquired through finetuning or more computationally efficient training-free few-shot and zero-shot learning, with progressively decreasing sample requirements.ScienceQA provides lectures and explanations for finetuning; Multimodal-CoT [185] generates a rationale before its answer, while CoT-PT [187] combines prompt tuning with step-specific visual bias. Few-shot learning uses handcrafted demonstrations, whereas zero-shot instructions elicit reasoning without examples [184], [186]; task and tool descriptions can also guide task decomposition [22], [188].Source: §7.2.1 · p. 12

7.2.2 Chain Configuration

• M-CoT chains vary by structure—single-chain or tree-shaped—and length control, with adaptive stopping or predefined limits.Single-chain reasoning follows question–rationale–answer steps, [185]; DDCoT instead solves subquestions using an LLM or visual experts, then aggregates their rationales. Adaptive chains let LLMs decide when to halt, whereas predefined chains stop at a specified length.Source: §7.2.2 · p. 12

7.2.3 Generation Patterns

• Infilling-based M-CoT fills gaps between preceding and following steps, whereas predicting-based M-CoT extends reasoning from instructions and prior history; both require correct, consistent steps.Infilling is used in [184], [186], while predicting-based generation appears in [22],, [169],, [185], [188].Source: §7.2.3 · p. 12

7.3.1 Introduction

• LLM-aided visual reasoning systems exhibit zero/few-shot generalization to unseen concepts, complex-task reasoning, and user-friendly fine control.MM-REACT [22] can explain why a meme is funny; fine control can use clicks and natural-language queries [79].Source: §7.3.1 · p. 13

7.3.2 Training Paradigms

• Training-free systems freeze pretrained models, using few-shot examples to guide tool-execution plans or zero-shot prompting to exploit linguistic knowledge and reasoning abilities.PointCLIP V2 [197] prompts GPT-3 for 3D-semantic descriptions, while CAT [79] refines captions according to user queries.Source: §7.3.2 · p. 13
• Finetuning improves tool-use planning [107] or localization capabilities [142], [205].GPT4Tools [107] collects a tool-related instruction dataset and uses it for instruction tuning.Source: §7.3.2 · p. 13

7.3.3 Functions

• Controllers use chain-of-thought reasoning to decompose complex tasks and assign subtasks to tools or modules, often completing tasks in a single round.VisProg prompts GPT-3 to generate a visual program whose lines invoke modules with specified input arguments; handcrafted examples guide planning [169],, [181].Source: §7.3.3 · p. 13
• Decision makers typically operate across multiple rounds, summarizing context and history, judging information sufficiency, and organizing user-friendly answers.This process can be iterative [195].Source: §7.3.3 · p. 13
• Semantics refiners exploit linguistic and semantic knowledge to integrate information into fluent sentences [202] or generate task-specific text [79], [197],.Source: §7.3.3 · p. 13

Long-context processing

• Limited long-context multimodal processing restricts MLLMs’ ability to handle long videos and long documents interleaving images and text.These applications require models to process more multimodal tokens.Source: §8 · p. 13

Complex instruction following

• MLLMs need better complex instruction following, a capability that currently motivates reliance on closed-source GPT-4V for high-quality question-answer data generation.Other models generally fail to match GPT-4V’s advanced instruction-following capabilities for this purpose.Source: §8 · p. 13

M-ICL and M-CoT

• M-ICL and M-CoT research remains rudimentary, with weak model capabilities motivating investigation of underlying mechanisms and potential improvements.Source: §8 · p. 13

Embodied agents

• Developing MLLM-based embodied agents that interact with the real world requires perception, reasoning, planning, and execution capabilities.Source: §8 · p. 14

Safety

• MLLMs can produce biased or undesirable responses under crafted attacks [177], [207],, making improved safety an important research direction.Source: §8 · p. 14

9 CONCLUSION

• The survey offers a broad view of MLLM research directions, including the basic recipe and related extensions.Source: §9 · p. 14
• The survey highlights current research gaps that remain to be filled.Source: §9 · p. 14
• The survey points to promising research directions, aiming to clarify current MLLM progress and inspire further work.Source: §9 · p. 14