Source-linked AI summary
Large Multimodal Agents: A Survey
Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, Guanbin Li
TL;DR
LLM-powered agents are increasingly being extended to multimodal settings, where they must interpret diverse inputs and perform more intricate tasks. This survey synthesizes LMA components, taxonomies, collaboration, evaluation, applications, and future directions, emphasizing the need for systematic assessment and coordinated progress.
Problem
Method
The survey reviews LMA perception, planning, action, and memory; classifies studies into four types; examines multi-agent frameworks; and compiles evaluation methodologies, applications, and future directions.
Source: §1 · p. 2, §2 · p. 2, §7 · p. 10Results
The review provides a comprehensive evaluation framework and identifies current LMA progress, applications, unresolved challenges, and promising research directions.
Source: §7 · p. 10Takeaways & Limitations
Systematic evaluation, unified single-agent designs, and effective coordination among multiple multimodal agents are key directions for advancing LMAs.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs) have achieved superior performance in powering text-based AI agents, endowing them with decision-making and reasoning abilities akin to humans. Concurrently, there is an emerging research trend focused on extending these LLM-powered AI agents into the multimodal domain. This extension enables AI agents to interpret and respond to diverse multimodal user queries, thereby handling more intricate and nuanced tasks. In this paper, we conduct a systematic review of LLM-driven multimodal agents, which we refer to as large multimodal agents ( LMAs for short). First, we introduce the essential components involved in developing LMAs and categorize the current body of research into four distinct types. Subsequently, we review the collaborative frameworks integrating multiple LMAs , enhancing collective efficacy. One of the critical challenges in this field is the diverse evaluation methods used across existing studies, hindering effective comparison among different LMAs . Therefore, we compile these evaluation methodologies and establish a comprehensive framework to bridge the gaps. This framework aims to standardize evaluations, facilitating more meaningful comparisons. Concluding our review, we highlight the extensive applications of LMAs and propose possible future research directions. Our discussion aims to provide valuable insights and guidelines for future research in this rapidly evolving field. An up-to-date resource list is available at https://github.com/jun0wanan/awesome-large-multimodal-agents.
1 Introduction
Large multimodal agents extend LLM-powered agents beyond text to process multimodal information and perform complex, multi-step tasks. This survey organizes the field’s developments, frameworks, evaluations, applications, and future directions.
Background
- LLM-powered agents improve command interpretation, knowledge assimilation, reasoning, and learning, while memory supports interaction with environments and varied language tasks.Source: §1 · p. 1
Motivation
- LMAs extend LLM-powered agents with multimodal perception and reasoning for tasks involving text, images, videos, and interactive environments.Web searching illustrates the pipeline: entering requirements, navigating pages, processing multimodal content, and integrating extracted information into an answer.Source: §1 · pp. 1–2
Research gap
- The survey addresses the limited multimodal focus of prior agent surveys by summarizing and comparing existing LMA studies conducted in isolation.Source: §1 · p. 2
Scope and contributions
- The paper covers LMA components, a four-type taxonomy, collaborative frameworks, evaluation methodologies, real-world applications, and future research directions.Source: §1 · p. 2
2 The Core Components of LMAs
LMAs comprise perception, planning, action, and memory components that transform multimodal environmental information into decisions and executable behavior. The survey reviews their models, representations, planning strategies, tools, action types, and memory mechanisms.
Overview
- The four core elements organize LMA design, while Table 1 records task modalities, planner models and formats, actions, collaboration, and long-term memory.Source: §2 · p. 2, Table 1 · p. 3
Perception
- Perception extracts task-relevant information from diverse modalities to support more effective planning and execution.Simple image- or audio-to-text conversion can produce redundant information, motivating sub-task tools and vocabulary-based processing for complex modalities.Source: §2 · pp. 2–3
Planning
- Planners reason about tasks using models such as GPT-3.5, GPT-4, LLaMA, or LLaVA, and express plans as natural language, programs, or hybrids.Open-source multimodal models such as LLaVA can directly process multiple modalities, while inspection and reflection improve robustness in complex environments.Source: §2 · p. 3
- Planning strategies are dynamic when plans are decomposed from initial input and static when plans are repeatedly formed from current environmental information or feedback.Static planning can revert to the original state for re-planning after errors are detected.Source: §2 · p. 4
Action
- Actions translate plans into tool use, embodied actions, or virtual actions such as web interaction, using prompts or action data for open-source-model fine-tuning.Tools include visual foundation models, APIs, and Python; embodied actions involve robots or characters, while virtual actions include clicking, scrolling, and keyboard use.Source: §2 · pp. 4–5, Table 2 · p. 5
Memory
- Memory may be short- or long-term, with multimodal agents increasingly requiring long-term storage of information and successful experiences across modalities.Some systems convert modalities to text, whereas others store multimodal states as keys and successful plans as values, retrieving analogous experiences by encoded similarity.Source: §2 · pp. 5–6
3 The Taxonomy of LMAs
The survey classifies LMAs into four types according to planner training and how long-term memory is incorporated. The taxonomy progresses from prompt-based systems without long-term memory toward memory-enabled agents suited to more dynamic environments.
Type I: Closed-source LLMs as Planners w/o Long-term Memory
- Type I uses prompts to guide closed-source LLMs as planners without long-term memory, typically for simpler visual tasks such as image editing, grounding, and VQA.Source: §3 · p. 6
Type II: Finetuned LLMs as Planners w/o Long-term Memory
- Type II fine-tunes open-source language or multimodal models with multimodal instruction data so they can reason, plan, and execute tasks without long-term memory.These systems generally address traditional visual or multimodal tasks similar to Type I.Source: §3 · p. 6
Type III: Planners with Indirect Long-term Memory
- Type III equips LLM planners with indirect long-term memory accessed through tools, such as task-relevant memory banks for spatiotemporal video reasoning.Source: §3 · p. 6
Type IV: Planners with Native Long-term Memory
- Type IV lets LLMs interact directly with long-term memory, using retrieved multimodal experiences and self-checking to refine plans in open-world tasks.The cited Minecraft agent completed over 200 distinct tasks and stored successful plans with their initial multimodal states for later retrieval.Source: §3 · p. 6
4 Multi-agent Collaboration
Multi-agent LMA frameworks distribute perception, planning, execution, and memory responsibilities across cooperating agents. Their defining distinction is whether long-term memory is included, while both aim to coordinate agents toward shared goals.
Frameworks
- Multiple LMAs assume different roles and coordinate actions to distribute the burden of completing a task, with frameworks differing by long-term-memory capability.Source: §4 · p. 7
Role specialization
- Specialized agents can divide environmental sensing, plan checking, replanning, and action execution across a coordinated workflow.In one framework, a perceiver senses the environment while a Patroller checks feedback and prompts the planner to reorganize action sequences after failures.Source: §4 · p. 7
- MemoDroid separates offline exploration, online subtask selection, action-sequence deduction, and memory recall for automating mobile tasks.Source: §4 · p. 7
5 Evaluation
The survey argues that LMAs need evaluation methods beyond conventional task metrics, combining subjective human assessment with objective metrics and benchmarks. It organizes evaluation around capabilities such as versatility, usability, scalability, safety, multimodal task performance, and real-world operation.
5 Evaluation
- Limited evaluation methodologies and reliance on conventional metrics make LMA assessment difficult, motivating pragmatic criteria and benchmark datasets.Source: §5 · p. 7
5.1 Subjective Evaluation
- Subjective evaluation uses human assessment to measure versatility, user-friendliness, scalability, and value and safety.These dimensions cover tool and task diversity, satisfaction with task outcomes, adaptation to unseen tools, and alignment with human values.Source: §5.1 · pp. 7–8
5.2 Objective Evaluation
- Objective evaluation quantitatively and systematically assesses LMA capabilities and is currently the most widely used evaluation approach.Source: §5.2 · p. 8
- Traditional task metrics are insufficient for LLM-powered LMAs, encouraging specialized measures such as VisualWebArena’s visual understanding, state-transition, response, and action-alignment criteria.Source: §5.2 · p. 8
- Benchmarks diversify evaluation through games, curated questions, and real webpages that test reasoning, multimodal processing, navigation, tool use, and visual-text understanding.SmartPlay specifies capability levels, GAIA contains 466 questions, and VisualWebArena evaluates agents on real webpages.Source: §5.2 · p. 8
6 Application
LMAs support applications across interfaces, physical environments, games, driving, video, visual content, complex reasoning, and audio. Their multimodal adaptability extends agent use beyond language-only interaction into varied real-world and creative tasks.
6 Application
- LMAs’ adaptability to diverse modalities makes them useful for decision-making and response generation in real-world, multisensory environments.Source: §6 · p. 8, Figure 4 · p. 9
GUI Automation
- GUI automation enables LMAs to navigate interfaces, execute repetitive tasks, and simplify workflows, potentially saving users time and supporting accessibility.GPT-4V-Act combines GPT-4V with web browsing for interaction, workflow automation, and automated UI testing.Source: §6 · p. 8
Robotics and Embodied AI
- Robotics and embodied AI integrate perception, reasoning, and action for physical interaction using sensory channels such as vision, audition, and touch.MP5 uses active perception and large language models to decompose and execute extensive Minecraft tasks.Source: §6 · p. 9
Game Developement
- Game AI creates more intelligent and realistic agents that support engaging and challenging experiences in interactive virtual environments.Source: §6 · p. 9
6 Application
- Multimodal agents support autonomous driving, video understanding, visual generation and editing, complex visual reasoning, and audio creation or manipulation.Examples include GPT-Driver for safe and comfortable trajectories, video analysis of objects and actions, LLaVA-Interactive for edited-image dialogue, multimodal question answering, and audio tools.Source: §6 · pp. 9–10
7 Conclusion and Future Research
The survey synthesizes LMA components, research categories, evaluation methods, benchmarks, and applications while identifying unresolved challenges. It highlights future work on unified and collaborative frameworks, standardized evaluations, and real-world human-computer interaction.
7 Conclusion and Future Research
- The survey reviews LMA components and research categories, compiles evaluation methodologies into a comprehensive framework, and surveys major application scenarios.Source: §7 · p. 10
On frameworks
- Future frameworks may unify single-agent planning, tools, and memory while improving coordination, communication, and task distribution among multiple LMAs.Source: §7 · p. 10
On evaluation
- Systematic evaluation should combine tasks of varying difficulty, clear nonredundant metrics, and datasets that more closely reflect real-world scenarios.Source: §7 · p. 10
On application
- Real-world applications, especially web browsing and human-computer interaction, remain promising because LMAs process multiple modalities for complex and nuanced tasks.Source: §7 · p. 10