PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications

Neel Mokaria, Rishie Raj, Dheeraj Baiju, Xiaoqian Shen, Shraman Pramanick, Kevin Qinghong Lin, Arda Senocak, Mike Zheng Shou, Philip Torr, Mohamed Elhoseiny, Yapeng Tian, Ruohan Gao, Salman Khan, Sayan Nag, Sanjoy Chowdhury, Dinesh Manocha

arXiv:2608.20379v1cs.AI

TL;DR

The survey examines how multimodality has been incorporated into agentic frameworks, addressing the limited systematic analysis of its architectural impact. It synthesizes fusion strategies, modules, applications, performance, and efficiency trade-offs, concluding that tighter perception–reasoning integration is central while deployment remains unresolved.

  • Problem

    Existing surveys insufficiently analyze how multimodality evolves and reshapes agentic architecture, despite its importance for multimodal perception, reasoning, planning, memory, and action.

  • Method

    The survey develops a modality-centric taxonomy and reviews fusion strategies, agent modules, application domains, performance, and efficiency-scalability trade-offs.

  • Results

    The evidence indicates that the largest gains arise from tighter perception–reasoning integration rather than model-size scaling alone.

  • Takeaways & Limitations

    Native multimodal models offer strong zero-shot capabilities, but token-heavy inference creates latency and cost that limit practical scalability.

  • Takeaways & Limitations

    No single architectural approach currently balances state-of-the-art performance with deployment feasibility.

Abstract

from arXiv · show

Advances in large language models (LLMs) have fueled a wave of research into agency: the ability to reason, plan, and act. This effort has produced agentic frameworks that orchestrate perception, memory, and decision-making around powerful LLM backbones. With the advent of large multimodal models (LMMs), these systems can process and integrate diverse modalities, including images, audio, and video, thereby improving their real-world applicability. Yet, while surveys of LLM-based agents exist, the role of multimodality in shaping agency has not been systematically examined in recent years. This survey fills the gap by analyzing the impact of multimodality across the core functional modules of the agentic framework: perception, reasoning, planning, memory, and action. Using this lens, we trace the evolution from text-centric agents to multimodal frameworks, examine how modalities are integrated through delegated, late-fusion, and early-fusion architectures, and assess the emergence of agentic behaviors enabled by grounded perception and multimodal reasoning. We organize existing work through a modality-centric taxonomy that links architectural design choices to agent capabilities. Moreover, we review multimodal agentic systems across various application domains, including Robotics, GUI & Web Navigation, Multimedia Content Generation & Editing, and Long-form Video Understanding & Retrieval. Beyond capabilities, we analyze performance across these settings and discuss efficiency-scalability trade-offs, including training and inference costs, latency, and deployment constraints. By focusing on the impact of multimodality in agentic design, we aim to identify key gaps and chart a roadmap toward robust and general-purpose intelligent systems.

1 Introduction

This survey addresses the lack of a unified account of how multimodality changes agentic architectures and capabilities. It traces multimodal integration across agent modules, perception-fusion strategies, application domains, and efficiency constraints.

  • Foundations: Multimodal frameworks mirror a cognitive architecture with perception for sensory grounding, an orchestrator for executive function and memory, and action interfaces for execution.
  • Research gap: Existing surveys largely focus on LLM-based agents or treat multimodality peripherally, leaving its architectural evolution insufficiently analyzed.
  • Contributions: The survey introduces a modality-based taxonomy spanning perception, reasoning, planning, memory, and action.
  • Contributions: It formalizes delegated, late-fusion, and early-fusion perception strategies and reports that multimodal fusion generally outperforms language-mediated processing.
  • Contributions: The analysis correlates architectural choices with quantitative performance across robotics, GUI and web navigation, multimedia generation, and long-form video understanding.
  • Contributions: The survey maps efficiency and scalability trade-offs involving resource use, task completion time, deployment costs, training, inference, and latency.

2 Foundations of Agentic Framework

Agentic frameworks coordinate reasoning, planning, memory, perception, and action in a cognitive loop, with multimodal perception evolving from tool delegation toward native processing. This progression improves grounding but creates trade-offs involving information loss, efficiency, and deployment.

  • Agentic framework components: The orchestrator coordinates reasoning, planning, and memory management as the framework’s central decision-making unit.
  • Reasoning and planning: LLM reasoning supports stepwise planning by decomposing high-level tasks into executable subtasks.
  • Memory: Memory systems store traces beyond the context window and retrieve relevant knowledge for subsequent planning actions.
  • Perception: Perception fusion progresses from delegated tool calls, through modality-specific embedding projections, to native processing of raw multimodal inputs.
  • Perception: Delegated perception offers modularity and specialization but loses information when multimodal inputs are translated into text.
  • Action: Action closes the perception-action loop through tools, APIs, webpage navigation, code execution, or physical embodiment, while updating memory with observations.
  • Modalities: Dense modalities such as images and video require native multimodal processing to preserve spatial and temporal details lost during text conversion.

3 Taxonomy

Multimodal agentic frameworks adapt perception, reasoning, memory, and interaction to preserve information across sensory inputs and executable actions. The taxonomy distinguishes delegated, modality-specific, and native multimodal processing, alongside increasingly grounded reasoning and bandwidth-aware memory.

  • Perception: Perception strategies range from external tool calls and modality-specific encoders to early-fusion architectures that process unified multimodal token sequences.
  • Reasoning: Reasoning progresses from text-only abstractions to visually grounded inference and cross-modal integration of asynchronous sensory inputs.
  • Memory: Modality-specific memory separates media from textual indices, requiring retrieval to identify relevant timestamps or frames before accessing raw visual data.
  • Memory and efficiency: Agents must manage interaction bandwidth because dense per-timestep information can rapidly fill model context.
  • Evaluation: Evaluation practices vary by domain: robotics relies on task success, video uses deterministic metrics, and GUI navigation combines programmatic checks with limited LLM-based evaluation.

4 Application

The application taxonomy organizes representative agentic frameworks by application area, environment, evaluation metrics, and representative systems. Evaluation predominantly favors deterministic or programmatic measures, with limited use of LLM-based judgments.

  • Taxonomy: Representative frameworks are classified by application category, primary metrics, and representative systems.
  • GUI and Web Navigation: GUI and web-navigation systems commonly use state validation, exact or substring matching, element accuracy, grounding accuracy, and step success rate.
  • Evaluation: Human evaluation measures human preference and task completion, while WebVoyager and WebArena also use restricted GPT-based trajectory or fuzzy-matching evaluation.
  • Evaluation: Table 1 summarizes evaluation metrics and indicates that deterministic measures such as task success rate and grounding accuracy are generally preferred over LLM-based judgments.

4.1 Text-only Applications

Text-only applications established language as the primary interface for browsing, collaboration, software development, and tool use. This design is efficient and controllable but limits access to visual and temporal details.

  • Question answering and self-correction: WebGPT and ReAct connected language models to external environments through browser actions, search, scrolling, citations, and interleaved reasoning.
  • Multi-agent collaboration and software development: Multi-agent software frameworks assign specialized roles to LLM instances so they can collaboratively generate complex software repositories.
  • Generalist tool and API calling: ToolFormer learns API invocation as natural-token prediction, while ToolkenGPT represents tools more compactly to scale across large toolsets.
  • Limitation: Text-only systems convert structured inputs such as webpages or GUI states into language, improving efficiency and control while limiting visual and temporal information.

4.2 Robotics and Physical Embodiment

Robotics research has progressed from language-guided affordance checks and modular perception toward grounded multimodal planning, hierarchical control, and end-to-end vision-language-action models. These systems increasingly connect sensory inputs directly to executable physical actions.

  • Grounding: Embodied agents require perceptual grounding to plan successfully, and evaluations include virtual text-based environments such as ALFWorld and WebShop.
  • Grounded planning with LLMs: SayCan evaluates actions using language probability and affordance probability so plans are both semantically valid and physically executable.
  • Grounded planning with LLMs: Dynamic code-based policy generation extends static affordance checks by handling spatial constraints and environmental feedback.
  • Multimodal reasoning with MLLMs: MLLM planners use direct visual reasoning and hierarchical architectures that separate high-level intent from precise, high-frequency manipulation.
  • Vision-language-action models: Vision-language-action models fuse perception, reasoning, and control, with systems such as PaLM-E, RT-2, Octo, and OpenVLA grounding plans or actions in physical states.
  • Vision-language-action models: Recent VLA systems use continuous flow-matching to generate smooth, high-frequency motor commands and extend sensory grounding beyond vision toward tactile sensing.
  • Multi-agent systems: Multi-agent robotics frameworks coordinate specialized robots through LLM-mediated dialogue, including verbal negotiation of assignments and spatial conflicts.

4.3 GUI and Web Navigation

GUI and web agents evolved from text-based tool augmentation toward late-fusion and native multimodal systems that preserve visual information. Reliable interaction remains constrained by spatial grounding, requiring specialized architectures or auxiliary pipelines.

  • Text-Based Tool-Augmented Agents: Early agents converted webpages or interfaces into simplified text, enabling standard LLMs to issue searches and commands.This approach included HTML, UI XML, retrieval, and vision-tool outputs converted into textual representations.
  • Text-Based Tool-Augmented Agents: HTML-to-text navigation is bottlenecked by context limits and loses spatial and layout information needed for robust GUI interaction.Aggressive simplification makes visual interface structure unavailable to text-only reasoning.
  • Late Fusion (Modality-Specific Encoding): Late-fusion agents project screen features from vision encoders into an LLM embedding space through adapters.Representative systems combine visual encoders with language models and are trained for webpage screenshots, HTML history, or GUI tasks.
  • Late Fusion (Modality-Specific Encoding): General-purpose vision encoders are insufficient for digital navigation; strong performance requires high-resolution processing and explicit pixel-level coordinate grounding.These requirements address the fine spatial precision of GUI actions.
  • Natively Multimodal Agents: Natively multimodal agents use models such as GPT-4V or Gemini as reasoning cores, emphasizing prompting, memory, tools, and visual grounding rather than additional model training.OpenCUA reports 45.0% on OSWorld-Verified with its 72B open-source model.
  • Natively Multimodal Agents: Native multimodality removes domain-specific training needs, but weak spatial grounding still motivates visual tagging overlays and explicit self-reflection loops.These auxiliary components are presented as necessary for reliable GUI interaction.

4.4 Multimedia Content Generation and Editing

Multimedia agents evolved from text-mediated neuro-symbolic orchestration toward native multimodal systems and role-specialized collaborative loops. These frameworks plan, compose tools, inspect visual outputs, and iteratively refine generation or editing.

  • Text-Based Tool-Augmented Agents: Early multimedia agents converted non-textual inputs into textual descriptions or executable code, leaving media manipulation to specialized external modules.The LLM handled semantic routing, task decomposition, and code generation rather than direct media operations.
  • Natively Multimodal Agents: Native multimodal agents directly perceive images, construct planning trees, execute tools, and self-correct using visual feedback.These systems avoid the text-conversion bottleneck while coordinating multi-step generation tasks.
  • Natively Multimodal Agents: Efficient, temporally consistent editing pairs native multimodal perception with tuning-free constraints such as A* search and optical-flow guidance.The constraints mitigate the computational costs of multi-step generation.
  • Multi-Agent Systems: Multi-agent creative systems assign specialized roles such as Creative Director, Art Critic, Editor, and Critic to iterative critique and revision loops.This division of labor supports generation, audiovisual synchronization, and video nonlinear editing workflows.
  • Multi-Agent Systems: CREA’s collaborative loops yield outputs that surpass single-agent baselines in semantic alignment and creative diversity.The cited comparison concerns the role-specialized creative framework described in the survey.

4.5 Long Form Video Understanding and Retrieval

Long-form video agents address massive token and attention costs by retrieving relevant frames, modularizing temporal reasoning, or using native multimodal context. Adaptive tool selection can outperform prior approaches on challenging video understanding benchmarks.

  • Motivation: Long-form video produces hundreds to thousands of visual tokens per minute, causing quadratic attention growth that overwhelms standard context windows.Context expansion and token compression are architectural responses, while agents treat video as a dynamic environment.
  • Iterative Retrieval Agents: Iterative retrieval agents selectively query relevant frames through plan–execute–inspect–learn loops instead of encoding entire videos in one pass.Intermediate outputs guide subsequent retrieval and reasoning decisions.
  • Iterative Retrieval Agents: Early retrieval frameworks relied on lossy vector searches or text descriptions, sacrificing fine-grained continuous visual details for computational feasibility.The limitation applies to proxies used to overcome video token bottlenecks.
  • Specialized Fine-tuned Reasoning Agents: VideoMind modularizes temporal reasoning with Planner, Grounder, and Verifier roles implemented through dynamically switched LoRA adapters.The roles decompose questions, locate timestamps, and check evidence.
  • Specialized Fine-tuned Reasoning Agents: Specialized agents pursue deep temporal grounding without massive compute scaling by modularizing reasoning or restructuring video representations.The survey identifies dynamic task-specific weights as one route to this architectural shift.
  • Natively Multimodal Agents: 74.2% accuracy on LVBench is reported for DVD, whose adaptive planning selects tools over a multi-granular video database and surpasses prior works.DVD replaces rigid workflows with autonomous iterative refinement.

5 Performance Analysis

Across application domains, tighter integration between perception, reasoning, and action improves performance by reducing information bottlenecks, while GUI agents still remain far below human performance.

  • Robotics and Physical Embodiment: Delegated perception establishes competence but loses information during text conversion and remains constrained by predefined skills.SayCan achieves 84% planning success and 74% execution success through affordance scoring, but does not generalize well beyond known skill primitives.
  • Robotics and Physical Embodiment: Early-fusion robotics models improve novel-object performance by unifying vision, language, and action representations.RT-2 reaches 62% success on novel-object tasks versus RT-1’s 32%, while OpenVLA surpasses larger alternatives across 29 evaluation tasks.
  • GUI and Web Navigation: Native multimodal GUI agents process screenshots directly, but their benchmark performance remains substantially below human performance.GPT-4-based agents achieve 14.41% on WebArena versus 78.24% for humans; GPT-4V with Set-of-Marks reaches 16.4% versus 88.7% on VisualWebArena.
  • Multimedia Content Generation and Editing: Visual feedback loops enable multimodal content-generation agents to detect and correct spatial and attribute-binding errors.GenArtist improves spatial relationship accuracy by over 20% versus DALL-E 3, with a further 7–10% gain from hierarchical planning.
  • Cross-Domain Findings: Across domains, tighter perception-reasoning integration improves capability, but GUI agents still lack the grounding, planning, and recovery needed for human-level interaction.The reported GUI gap remains 60–70 percentage points versus human performance on complex interactive tasks.

6 Efficiency

Multimodal agent efficiency depends on balancing sensory-processing demands, inference latency, throughput, and memory overhead. Selective processing, smaller backbones, quantization, and unified architectures reduce deployment costs, but API-based systems can remain slow and expensive.

  • Efficiency Dimensions: Multimodal agents incur overhead from high-resolution vision, temporal video modeling, and continuous audio processing.Efficiency is primarily assessed through inference latency, throughput, and memory overhead.
  • Robotics and Physical Embodiment: OpenVLA achieves approximately 6 Hz on a single consumer GPU through a smaller backbone and 4-bit quantization.RT-2 reaches only 1–3 Hz with models up to 55B parameters, whereas smaller variants reach approximately 5 Hz.
  • Robotics and Physical Embodiment: Native multimodal agents can incur severe API latency, with GPTArm task execution averaging 33–110 seconds.This latency forces a stop-and-think behavior that limits real-time closed-loop control.
  • Multimedia Content Generation and Editing: Tool-use agents reduce training demands by shifting intelligence into planning and selectively executing modular vision or audio tools.VISPROG and AudioGPT avoid end-to-end retraining by composing pretrained components for each query.
  • Long-Form Video Understanding and Retrieval: Unified multimodal models can improve latency and serving cost while preserving performance across modalities.GPT-4o reports approximately 232 ms audio response latency and roughly 50% lower serving cost than GPT-4 Turbo.
  • Long-Form Video Understanding and Retrieval: Selective attention improves long-video efficiency by processing sparse task-relevant content rather than dense frame sequences.VideoAgent processes 20× fewer frames, while the supplied comparison identifies delegated and late-fusion strategies as especially efficient.

7 Scalability

Multimodal-agent scalability reflects a trade-off between upfront training infrastructure and recurring operational costs. Modular retrieval, lightweight adapters, quantization, and large unified contexts provide different routes to scaling across tasks and modalities.

  • Scalability Dimensions: Scalability balances training compute against recurring per-token API costs, data efficiency, and deployment constraints.The survey evaluates scalability through task complexity, data volume, environmental diversity, and the ability to generalize to novel tasks.
  • LLM-based Tool-Use Agents: Tool-use agents avoid large end-to-end training but remain constrained by recurring API costs and the availability of robust low-level primitives.Their scaling profile favors modular reuse while shifting expense toward plan generation and synthesized code.
  • Fine-tuned Late-fusion Multimodal Agents: Fine-tuned late-fusion agents require substantial upfront resources but can offer more favorable inference economics after training.OpenVLA required 21,500 A100 hours for pre-training, while PaLM-E scales to 562 billion parameters.
  • Natively Multimodal Agents: Native multimodal systems relying on proprietary APIs are primarily limited by recurring visual-query costs, whereas end-to-end systems shift costs into training.AppAgent and Mobile-Agent incur costs for every screenshot processed, while EMMA requires substantial task-specific training compute.
  • Long-Form Video Understanding and Retrieval: Retrieval and modular role specialization scale long-video understanding by avoiding dense processing or multiple heavy networks.VideoAgent retrieves approximately 8.4 frames per query, while VideoMind uses role-specific LoRA adapters within one backbone.
  • Long-Form Video Understanding and Retrieval: Unified long-context models scale multimodal reasoning by processing text, video, and audio within a single architecture.Gemini 1.5 Pro reasons over millions of tokens and can process entire books or hours of video without chunking or external retrieval.

8 Latency

Multimodal agents use prompt, token, model, and architecture optimizations to reduce latency, but deployment still faces a speed–reliability trade-off and API or compute constraints.

  • Latency-reduction strategies: Prompt pruning, token reduction, and adaptive feedback frequencies reduce model calls, context, or intermediate evaluations to lower latency.GPTArm reduced inference latency by ∼50% on complex tasks, alongside a ∼9% drop in SR.
  • Deployment constraints: External API calls and large multimodal contexts hinder high-frequency deployment, while reduced visual verification exposes systems to anomalies and execution failures.Robotics and Web/GUI navigation impose especially strict latency requirements, and EMMA’s fused LiDAR/radar and camera streams can enlarge context windows.
  • GUI and Web Navigation: 21.3% lower inference latency is achieved by AutoDroid through UI-element merging and memory-based shortcuts from offline exploration.UI-element merging cuts prompt token count by almost 50%.
  • GUI and Web Navigation: CogAgent reduces computational cost by ∼25x through a cross-attention module that makes visual-feature processing linear.The approach uses a locally deployed VLM, avoiding network-induced latency and API cost bottlenecks.
  • GUI and Web Navigation: WebGLM reduces webpage-retrieval latency from ∼2 mins to 5 secs using parallel asynchronous crawling.

9 Reliability

Reliability depends on detecting unexpected states and containing cascading errors, but verification improves recovery at the cost of control speed, while distribution shift remains unresolved.

  • Reliability requirements: Multimodal agents must handle out-of-distribution data, unexpected intermediate outputs, and cascading errors in real-world conditions.Failures may include hallucinated bounding boxes, incorrect spatial captions, or outdated state representations.
  • Robotics and Physical Embodiment: Closed-loop verification re-queries perception after each physical action and replans from the verified state when observations diverge from expectations.
  • Robotics and Physical Embodiment: Dense visual verification improves recovery but reduces control frequency, whereas reducing verification frequency yields approximately a 9% drop in complex-task success rate.The analysis characterizes reliability and throughput as structurally opposed in current robotic frameworks.
  • GUI Grounding and Web Navigation: GUI agents use pre-action grounding checks or post-action screen-state comparisons to detect incorrect element selections and trigger corrections.SeeAct uses a two-pass verification strategy, while WebVoyager and Mobile-Agent compare resulting states against expected states.
  • GUI Grounding and Web Navigation: Live-web randomness, long contexts, navigation loops, and imprecise pixel grounding continue to constrain reliable GUI-agent deployment.Continuous self-reflection and state verification are needed because of interruptions, advertisements, and unpredictable loading states.

10 Limitations

The survey identifies persistent grounding, efficiency, memory, verification, security, and safety limitations across multimodal agentic systems and application domains.

  • Grounding Gap in Complex Environments: A ∼60-70% performance gap remains between agents and humans on high-precision GUI benchmarks such as WebArena and VisualWebArena.The survey attributes the gap to insufficient coordinate-level grounding and multi-step error recovery.
  • Performance-Efficiency Trade-off: Native multimodal models offer strong generalization and zero-shot capabilities but incur prohibitive inference costs and latency at scale.Fine-tuned domain-specific models are more efficient but require substantial upfront training compute and have limited generalization.
  • Long-Horizon Memory and Reasoning: Current memory architectures struggle with error accumulation, belief revision, and consistent state maintenance over extended interactions.These brittleness issues also affect retrieval-based long-form video understanding approaches processing task-relevant frames.
  • Verification and Benchmarking Leakage: Reliance on proprietary closed-source APIs prevents independent verification and systematic ablations, while public benchmark data may introduce leakage and performance inflation.
  • Adversarial Robustness: Richer sensory inputs expand the attack surface through adversarial visual prompt injections and cross-layer vulnerabilities in embodied environments.
  • Safety-Critical Failure Modes: Weak grounding and limited foresight can produce instruction drift, hallucinated actions, and invalid planning steps, while current self-correction remains brittle.

11 Future Research Directions

Future research targets representations optimized for planning and control, adaptive memory, hybrid model architectures, broader sensing, and native multimodal communication.

  • Unified Multimodal Representation and Reasoning: Unified multimodal representations should preserve spatial and temporal structure while dynamically reweighting modalities for downstream planning and control.
  • Scalable Memory Architectures for Lifelong Learning: Lifelong-learning memory architectures should manage multiple knowledge timescales, revise conflicting beliefs, trace evidence sources, and update knowledge coherently.This direction moves beyond passive storage and retrieval-augmented generation toward active reasoning about knowledge state.
  • Hybrid Architectures for Resolving Trade-offs: Hybrid architectures could combine lightweight models for routine tasks with larger models for edge cases to balance efficiency and capability.
  • Multimodal Integration Beyond Vision-Language: Multimodal agents should extend beyond vision-language fusion to tactile, force, and 3D spatial sensing for embodied intelligence.These modalities provide contact dynamics and volumetric knowledge unavailable through vision alone.
  • Multimodal Multi-Agent Communication: Native multimodal communication protocols could reduce latency, ambiguity, and information loss in multi-agent coordination beyond language-only interfaces.

12 Conclusion

Across application domains, multimodal agents gain most from tighter perception–reasoning integration, while deployment remains constrained by reliability, latency, and cost trade-offs.

  • Tighter integration of perception and reasoning produces the largest multimodal-agent gains across domains, rather than scaling model size alone.
  • Robotics shows that compact end-to-end systems can outperform substantially larger pipelines separating perception from action.
  • Web-navigation agents remain far from human-level reliability on tasks requiring precise control and error recovery.
  • Native large multimodal models provide strong zero-shot capabilities, but token-heavy inference creates latency and cost barriers to practical scalability.
  • Domain-specific and fine-tuned architectures offer a more sustainable efficiency trade-off by exchanging higher training costs for lower inference latency.
Loading 2608.20379v1…