PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Agentic Multimodal Large Language Models

Huanjin Yao, Ruifei Zhang, Jiaxing Huang, Jingyi Zhang, Yibo Wang, Bo Fang, Ruolin Zhu, Yongcheng Jing, Shunyu Liu, Guanbin Li, Dacheng Tao

arXiv:2510.10991v1cs.CVcs.AIcs.CL

TL;DR

Static query–response MLLMs and workflow-bound agents are limited for dynamic tasks that call for planning, tool use, and environment interaction. This survey organizes agentic MLLM capabilities and research resources, concluding with a taxonomy and future directions while noting challenges such as scarce agentic training data.

  • Problem

    Traditional MLLMs use static query–response interactions, while complex dynamic tasks require internal intelligence, external tool invocation, and environment interaction.

    Source: §1 · p. 1

  • Method

    The paper systematically surveys agentic MLLM foundations, capability areas, technical approaches, training and evaluation resources, applications, and research directions.

    Source: §1 · p. 2

  • Results

    The survey organizes agentic MLLMs into internal intelligence, external tool invocation, and environment interaction, and reviews related methods, resources, applications, and directions.

    Source: §1 · p. 2, §8 · p. 20

  • Takeaways & Limitations

    The review provides an organized overview of current agentic MLLM work and outlines pathways for further research.

    Source: §1 · p. 2

    Agentic training datasets remain scarce, much generated data is inaccessible, and multimodal data synthesis remains underexplored.

    Source: §7.4 · p. 19

Abstract

from arXiv · show

With the recent emergence of revolutionary autonomous agentic systems, research community is witnessing a significant shift from traditional static, passive, and domain-specific AI agents toward more dynamic, proactive, and generalizable agentic AI. Motivated by the growing interest in agentic AI and its potential trajectory toward AGI, we present a comprehensive survey on Agentic Multimodal Large Language Models (Agentic MLLMs). In this survey, we explore the emerging paradigm of agentic MLLMs, delineating their conceptual foundations and distinguishing characteristics from conventional MLLM-based agents. We establish a conceptual framework that organizes agentic MLLMs along three fundamental dimensions: (i) Agentic internal intelligence functions as the system's commander, enabling accurate long-horizon planning through reasoning, reflection, and memory; (ii) Agentic external tool invocation, whereby models proactively use various external tools to extend their problem-solving capabilities beyond their intrinsic knowledge; and (iii) Agentic environment interaction further situates models within virtual or physical environments, allowing them to take actions, adapt strategies, and sustain goal-directed behavior in dynamic real-world scenarios. To further accelerate research in this area for the community, we compile open-source training frameworks, training and evaluation datasets for developing agentic MLLMs. Finally, we review the downstream applications of agentic MLLMs and outline future research directions for this rapidly evolving field. To continuously track developments in this rapidly evolving field, we will also actively update a public repository at https://github.com/HJYao00/Awesome-Agentic-MLLMs.

1 INTRODUCTION

Traditional MLLMs and workflow-bound agents are limited in handling dynamic, complex tasks, while agentic MLLMs combine adaptive planning, proactive action, and broader generalization. This survey organizes their foundations, capabilities, resources, applications, and research directions.

1 INTRODUCTION

  • Static query–response MLLMs and fixed-workflow agents are inadequate for dynamic tasks requiring internal intelligence, external tools, and environment interaction.Existing agents also tend toward passive execution and domain-specific applications.Source: §1 · p. 1
  • Agentic MLLMs adapt workflows, proactively plan and execute actions, and operate across diverse tasks rather than relying on static, domain-specific procedures.Their capabilities include reasoning, reflection, memory, tool use, and environment interaction.Source: §1 · pp. 1–2, Figure 2 · p. 3
  • The survey addresses the lack of a comprehensive review by organizing agentic MLLMs around internal intelligence, external tool invocation, and environment interaction.It also covers foundations, technical approaches, training and evaluation resources, and future research directions.Source: §1 · p. 2
  • The paper presents a systematic review, links methodological and resource advances, and identifies research challenges and potential directions for agentic MLLMs.The authors describe it as the first survey in this field.Source: §1 · p. 2

2 DISCUSSION OF MLLM AGENT AND AGENTIC MLLM

The section contrasts conventional MLLM agents’ fixed, developer-designed workflows with agentic MLLMs’ autonomous, adaptive decision-making. It frames the distinction through dynamic workflows, proactive action execution, and generalization across domains.

2 DISCUSSION OF MLLM AGENT AND AGENTIC MLLM

  • The section distinguishes agentic MLLMs from conventional MLLM agents by their dynamic workflows, proactive action execution, and cross-domain generalization.These distinctions are illustrated in Figure 2.Source: §2 · p. 3

2.1 Overview of MLLM Agents

  • Conventional MLLM agents decompose tasks into subtasks handled through developer-designed prompts, but their fixed workflows constrain adaptability and generalizability.Intermediate outputs pass sequentially between role-specific instances.Source: §2.1 · p. 3

2.2 Overview of Agentic MLLMs

  • Agentic MLLMs treat task-solving as autonomous decision-making, selecting actions in response to contextual features and evolving environmental states.This contrasts with workflows in which developers preassign roles and instructions.Source: §2.2 · p. 3, §2.1 · p. 3

2.2.1 Dynamic Workflow

  • Agentic MLLMs adaptively select strategies as the state evolves, rather than following a static, developer-designed pipeline.The state transition is represented as st+1 = δ(st, at), where st is the current state, at the chosen action, and δ the transition function.Source: §2.2.1 · p. 3

2.2.2 Proactive Action Execution

  • Agentic MLLMs proactively choose each action from the current state, shifting from instruction-following to deciding what to do next.The chosen action is determined under the current state according to policy π.Source: §2.2.2 · p. 4

2.2.3 Generalization Across Domains

  • Agentic MLLMs adapt workflows across environments and domains, with generalization framed as maximizing expected cumulative reward over task and environment distributions.The formulation includes a discount factor that controls the relative importance of long-term versus short-term rewards.Source: §2.2.3 · p. 4
  • The section models agentic MLLMs as adaptive policies in an action-oriented Markov decision process that continually updates internal states and makes context-sensitive decisions.This formulation is presented as supporting autonomous planning, acting, and generalization across tasks and domains.Source: §2.2.3 · p. 4

3 FOUNDATIONS OF AGENTIC MLLMS

The foundations section covers agentic MLLM architectures and action spaces, then describes training stages and complementary process- and outcome-based evaluation.

3.1 Agentic Foundational MLLMs

  • Dense MLLMs activate all parameters per token, while MoE models route tokens sparsely to top-k experts for efficient specialization and varied agentic behaviors.MoE architectures are increasingly used to support adaptive reasoning and dynamic tool invocation.Source: §3.1 · pp. 4–5

3.2 Agentic Action Space

  • Agentic action spaces use either distinct special tokens or a unified action token with a JSON-like tool specification for selecting actions.At each state, models reason over candidate actions and select one to optimize task completion.Source: §3.2 · p. 5

3.3 Agentic Contiunal Pre-training

  • Agentic CPT integrates new knowledge while supporting planning and tool use without forgetting, using large-scale synthetic corpora and maximum-likelihood training.The survey says CPT can reduce optimization conflicts in later alignment stages.Source: §3.3 · p. 5

3.4 Agentic Supervised Fine-tuning

  • Agentic SFT provides an initial policy prior from high-quality agentic trajectories, aligning models with action patterns before reinforcement learning.Trajectory data may be synthesized through reverse engineering, graph-based synthesis, or formalized task modeling.Source: §3.4 · p. 5

3.5 Agentic Reinforcement Learning

  • Agentic RL refines planning and policies through exploration and reward feedback; PPO clips policy updates, while GRPO estimates its baseline from grouped rollouts without a separate value function.PPO commonly adds a reference-model KL penalty to encourage linguistic coherence and mitigate reward hacking; GRPO normalizes group rewards into relative advantages.Source: §3.5 · pp. 5–6

3.6 Agentic Evaluation

  • Agentic MLLM evaluation separates process quality, including reasoning and tool use, from outcome quality, assessing accurate and helpful solutions across downstream tasks.Together, the dimensions assess intermediate-process quality and final-outcome effectiveness.Source: §3.6 · pp. 6–7

4 AGENTIC MLLM

The survey organizes agentic MLLMs around internal intelligence, external tool invocation, and environmental interaction. It presents internal intelligence as coordinating plans, tools as extending problem-solving beyond model knowledge, and environment feedback as supporting adaptation.

4 AGENTIC MLLM

  • Internal intelligence combines long-chain reasoning, reflection, and memory to construct plans and coordinate actions step by step.The survey describes it as the cognitive core that orchestrates subsequent actions.Source: §4 · p. 7
  • Agentic MLLMs use search, code execution, and visual processing tools to acquire information and extend problem-solving beyond intrinsic knowledge.Source: §4 · p. 7
  • Interaction with virtual and physical environments provides perception and feedback, enabling dynamic adaptation in real-world deployments.Source: §4 · p. 7

4.1 Agentic Internal Intelligence

The survey reviews agentic internal intelligence through reasoning, reflection, and memory, describing approaches and tradeoffs for each capability.

4.1.1 Agentic Reasoning

  • Prompt-based reasoning ranges from explicit step-by-step instructions to Best-of-N selection and tree search, but fixed model knowledge and limited inference-time search constrain open-ended tasks.Tree search expands reasoning paths beyond linear chains; prompt-based methods remain limited in scalability and robustness on challenging tasks.Source: §4.1.1 · p. 7
  • SFT reasoning trains on annotated long-chain traces, including directly generated, structured, or tree-organized paths, but depends on high-quality data and can bind models to fixed patterns.These constraints remain challenges for generalizable reasoning.Source: §4.1.1 · p. 7
  • RL-based reasoning uses exploration and feedback to optimize reasoning trajectories, supporting more flexible and adaptive reasoning.Source: §4.1.1 · p. 7
  • Outcome-based rewards are simple to implement and scale but ignore intermediate reasoning, whereas process-based rewards offer finer supervision at higher design and computational cost and remain vulnerable to reward-model hacking.Process rewards can assess intermediate steps through rules or specialized reward models.Source: §4.1.1 · p. 8

4.1.2 Agentic Reflection

  • Reflection can emerge through RL or be explicitly induced after complete responses or between reasoning steps, where drafts are critiqued and revised.The survey describes reflection as supporting response verification and refinement.Source: §4.1.2 · pp. 8–9

4.1.3 Agentic Memory

  • Contextual memory reuses history within the context window, with token compression and window extension addressing its fixed-length constraint.Compression can use learned tokens or pooling, while window-extension approaches directly increase context length.Source: §4.1.3 · p. 9
  • External memory ranges from fixed heuristic pipelines to systems that dynamically store, update, and retrieve information; most agentic memory work remains text-centric.The survey identifies multimodal agentic memory management as a notable research gap.Source: §4.1.3 · pp. 9–10

4.2 Agentic External Tool Invocation

The survey reviews how agentic MLLMs use external tools within reasoning, focusing on information search, code execution, and visual processing. It traces a shift toward models that decide when and how to invoke tools, including image operations that support multimodal reasoning.

4.2 Agentic External Tool Invocation

  • Agentic MLLMs integrate external tools into step-by-step reasoning and autonomously determine when and which tools to use, beyond prompt-triggered tool use.The survey organizes this work around information search, code execution, and visual processing.Source: §4.2 · p. 10

4.2.1 Agentic Search for Information Retrieval

  • Web search and retrieval-augmented generation extend MLLMs beyond fixed parametric knowledge by giving them access to external sources for emerging or rapidly changing information.The survey describes agentic search as using reinforcement learning to decide when and what to search for, with search embedded in reasoning and multi-turn interaction.Source: §4.2.1 · pp. 10–11

4.2.2 Agentic Coding for Complex Computations

  • Agentic coding addresses limits in program synthesis, computation, and symbolic reasoning by having models plan, generate, and refine code through iterative reasoning and tool use.Methods use outcome-based or process-oriented rewards, tool-integrated reasoning, and reinforcement learning to support code use while managing tool-call efficiency.Source: §4.2.2 · p. 11

4.2.3 Agentic Visual Processing for Thinking with Image

  • Agentic visual processing develops through three approaches: thinking with cropped images, manipulated images, and generated images.These approaches use image operations or generation within reasoning; VPRL generates image sequences as visual cues for visual planning.Source: §4.2.3 · p. 12
  • ReVPT’s visual toolkit, cold-start SFT, and RL achieve state-of-the-art results on spatial reasoning and image-understanding benchmarks.The toolkit includes depth estimation, zooming, object detection, and edge detection.Source: §4.2.3 · p. 12

4.3 Agentic Environment Interaction

Agentic environment interaction places MLLMs in virtual or physical settings where they perceive changes, act, use feedback, and adapt toward long-term goals. The survey reviews GUI agents learned from demonstrations or direct interaction, alongside embodied perception, planning, navigation, and manipulation.

4.3 Agentic Environment Interaction

  • Agentic environment interaction is a continuous loop of action, environmental perception, and feedback that lets models adapt strategies and pursue long-term goals.The survey covers both virtual and physical environments.Source: §4.3 · p. 12

4.3.1 Agentic Virtual Environment Interaction

  • GUI agents learn either from collected demonstration trajectories or through direct interaction in dynamic GUI environments.Offline demonstration learning provides a foundation for GUI automation, but its models can lack robustness to unexpected events and execution errors; online interaction is presented as a response to this limitation.Source: §4.3.1 · p. 13
  • 136 training examples let UI-R1 achieve substantial accuracy gains on in-domain and out-of-domain mobile GUI tasks; other approaches use policy optimization, deliberation, and error recovery.Online methods include automated task generation and reward estimation, iterative tuning, and multi-turn learning from task-completion rewards.Source: §4.3.1 · p. 13

4.3.2 Agentic Physical Environment Interaction

  • Embodied-agent research spans active perception, long-horizon planning, navigation, and manipulation, connecting environmental sensing to physical action.The surveyed methods include uncertainty-aware evidence gathering, interactive policy optimization, dual-system navigation, and approaches to scalable preference generation or reduced human annotation.Source: §4.3.2 · pp. 13–14

5 TRAINING & EVALUATION

The survey organizes open resources for developing and assessing agentic MLLMs across training frameworks, training datasets, and evaluation datasets. It reviews resources for internal intelligence, external tool use, and environment interaction.

5 TRAINING & EVALUATION

  • Training frameworks, training datasets, and evaluation datasets provide the algorithmic infrastructure, agentic cross-modal alignment, and capability measurement needed to develop and assess agentic MLLMs.The survey presents these as three core components for agentic MLLM research.Source: §5 · p. 14

5.1 Training Framework

  • The surveyed open-source frameworks support agentic continual pre-training, supervised fine-tuning, and reinforcement learning.The resources include general CPT/SFT and RL libraries as well as agentic RL frameworks for tool-calling, environments, and agent training.Source: §5.1 · pp. 14–15

5.2 Training dataset

  • Training datasets cover internal reasoning, reflection, and memory; external search, code, and visual processing; and interaction in virtual and physical environments.Examples include multimodal reasoning and reflection corpora, tool-use datasets, GUI data, and embodied navigation and manipulation data.Source: §5.2 · pp. 15–16

5.3.1 Benchmark Internal Intelligence

  • Evaluation datasets assess reasoning and reflection across general, STEM, chart, and document problems, as well as memory over long multimodal contexts and interactions.The memory benchmarks include MileBench, MMLongBench-Doc, LongVideoBench, and LVBench.Source: §5.3.1 · pp. 16–17

5.3.2 Benchmark External Tool Invocation. •

  • Search benchmarks test multi-hop, knowledge-intensive, up-to-date retrieval and evidence integration; code benchmarks assess code generation and code-integrated reasoning.Visual-processing benchmarks evaluate high-resolution image understanding that may require cropping and zooming to reveal clues.Source: §5.3.2 · p. 17

5.3.3 Benchmark Environment Interaction

  • Virtual-interaction benchmarks evaluate GUI task execution, while physical-interaction benchmarks assess navigation and manipulation.The survey names ScreenSpot, AndroidWorld, and OSWorld for GUI interaction; LH-VLN and HA-VLN for navigation; and VLABench for manipulation.Source: §5.3.3 · p. 17

6 APPLICATION

The survey reviews agentic MLLMs across applications in knowledge work, physical and digital environments, healthcare, driving, and recommendation. It highlights how reasoning, tool use, and interaction support complex tasks across these settings.

6.1 Deep Research

  • Deep Research agents combine multi-step reasoning and tool use to automate information discovery and synthesis for knowledge-intensive work.The survey describes applications in finance, science, policy, and education.Source: §6.1 · p. 18

6.2 Embodied AI

  • Vision-language-action models connect multimodal reasoning with motion control, and robotic agents demonstrate open-world generalization.The survey notes potential commercial uses in logistics, smart manufacturing, and personalized services.Source: §6.2 · p. 18

6.3 Healthcare

  • Healthcare systems combine RL-activated reasoning and reflection with tools such as domain-aware retrieval and medical knowledge graphs for complex medical reasoning.Earlier supervised-finetuning approaches using curated medical QA data often show limited generalization.Source: §6.3 · p. 18

6.4 GUI Agents

  • GUI agents perceive screens, interpret natural-language commands, and execute actions across web, mobile, and desktop environments.Applications range from file management and web operations to cross-app workflows and personalized support.Source: §6.4 · p. 18

6.5 Autonomous Driving

  • Autonomous-driving approaches use chain-of-thought reasoning for motion trajectories or external perception tools; SFT combined with RL teaches tool use and improves policy robustness and generalization.Tools include object detection, depth estimation, and occupancy prediction.Source: §6.5 · p. 18

6.6 Recommender System

  • Agentic recommenders proactively explore preferences, simulate future behaviors, and adapt strategies, balancing immediate feedback with long-horizon personalization.They combine multimodal cues with reasoning, reflection, memory, tool use, and virtual interaction.Source: §6.6 · pp. 18–19

7 CHALLENGES AND FUTURE DIRECTIONS

The survey characterizes agentic MLLMs as an early-stage field with unresolved challenges. It identifies constraints in tool access, efficiency, memory, training and evaluation data, and safety, alongside directions for further work.

7.1 Richer Action Space of Agentic MLLM

  • Existing agentic MLLMs often have action spaces and tool access restricted to a single type, motivating broader access to tools and services.Suggested additions include data-analysis platforms, simulations, multimodal sensors, and interactive APIs.Source: §7.1 · p. 19

7.2 Efficient Agentic MLLMs

  • A single task may take up to thirty minutes, adding computational and reasoning overhead that challenges real-time use and large-scale deployment.The survey notes that research on speeding up tool invocation remains limited and calls for faster training and inference without compromising performance.Source: §7.2 · p. 19

7.3 Long-term Agentic Memory

  • Agentic memory research remains primarily language-focused, while constrained effective memory length limits coherent knowledge across longer time horizons.The survey calls for persistent memory that is scalable for multimodal streams and selective in what it retains and retrieves.Source: §7.3 · p. 19

7.4 Agentic Training and Evaluation Dataset

  • Training data remains scarce, and existing benchmarks leave gaps in multimodal trajectories, memory use, coordination across tool calls, and correct action execution.Some synthetic trajectory data is inaccessible to the research community.Source: §7.4 · p. 19

7.5 Safe Agentic MLLMs

  • Autonomous tool and environment actions can propagate harmful or incorrect information into unsafe decisions, with ambiguous or adversarial multimodal inputs further destabilizing behavior.The survey calls for rigorous benchmarking, adversarial stress testing, and normative frameworks to support reliability, controllability, and alignment with human intent.Source: §7.5 · pp. 19–20

8 CONCLUSION

The survey organizes agentic MLLM advances and distinguishes these models from traditional MLLM agents through dynamic workflows, proactive actions, and cross-domain generalization. It consolidates foundations, a three-part capability taxonomy, research resources, applications, and a repository tracking developments.

8 CONCLUSION

  • The survey distinguishes agentic MLLMs from traditional MLLM agents by dynamic workflows, proactive action, and generalization across domains.It also introduces foundational topics including action spaces, CPT, SFT, RL, and evaluation methodologies.Source: §8 · p. 20
  • Its taxonomy groups agentic capabilities into internal intelligence, external tool invocation, and environment interaction.These cover long-horizon decisions, proactive information acquisition and manipulation, and iterative adaptation through environmental feedback.Source: §8 · p. 20
  • The survey consolidates training frameworks, datasets, and benchmarks, reviews applications, and tracks developments through a real-time GitHub repository.The authors present these resources as a practical reference for future research.Source: §8 · p. 20

Related papers

Loading 2510.10991v1…