A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications

Agentic foundations

• Contemporary agentic frameworks integrate perception, orchestration, memory, and action into a cognitive loop, overcoming earlier systems’ isolated reasoning, recognition, planning, and policy-learning components.The loop connects sensory grounding, executive planning, stored interaction experiences, and tool-mediated or embodied action.Source: §1 · p. 4, Figure 1 · p. 5

From LLMs to multimodality

• LLM-based agents enable open-world reasoning, planning, and tasks including web browsing, image editing, and robotic manipulation, but translating non-text inputs into language loses information.LLMs serve as foundational components, whereas end-to-end agents are complete frameworks built around them.Source: §1 · p. 4

Fusion strategies

• Multimodal perception evolved from delegated and late-fusion adapters to early-fusion end-to-end models, with native cross-attention enabling deeper multimodal integration.Late-fusion models encode modalities into an LLM’s word-embedding space, while early-fusion models integrate multimodal perception directly into the foundational model; the survey also identifies delegated perception as a distinct strategy.Source: §1 · pp. 4–6

Scope and definition

• The survey defines a multimodal agent as an LMM-backed framework that natively perceives, reasons, and plans across modalities before orchestrating tools or embodiment to act.Late-fusion agents are lightweight and faster at inference, whereas early-fusion agents better support fine-grained understanding and reasoning.Source: §1 · p. 5

Research gap

• Prior surveys emphasize LLM agents or domain-specific applications, leaving multimodality’s architectural evolution and links among fusion, memory, efficiency, safety, and evaluation insufficiently unified.This survey addresses the gap across the full stack from perception and reasoning through memory and action.Source: §1 · pp. 5–6

Survey contributions

• The survey contributes a modality-centered taxonomy, four-domain performance analysis, and fusion-strategy comparisons spanning resource use, completion time, deployment costs, and proprietary API trade-offs.The application domains are robotics and physical embodiment, GUI and web navigation, multimedia content generation, and long-form video understanding.Source: §1 · p. 6

2.1 Orchestrator

• The orchestrator coordinates reasoning, planning, and memory, evolving from brittle symbolic systems into flexible foundation-model-based control that supports adaptive, goal-directed behavior.LLMs contribute few-shot learning, multi-step reasoning, and instruction following, while reasoning, planning, and memory mirror executive functions in human cognition.Source: §2.1 · pp. 6–7

2.1.2 Planning

• Planning decomposes high-level tasks into subtasks, interleaves plans with retrieval or environmental feedback, and revises actions through executable code, tool calls, or physical affordances.Representative frameworks ground planning in interpreters, APIs, and real-world value models for embodied tasks.Source: §2.1.2 · p. 7

2.1.3 Memory

• Memory gives agents persistent state beyond the context window through stored traces and retrieval, spanning episodic reflections, external databases, long-term forgetting mechanisms, interactive control, and RAG.Hierarchical retrieval can access multimodal objects across heterogeneous vector, graph, and web databases.Source: §2.1.3 · pp. 7–8

2.2 Perception

• Perception converts multimodal inputs into grounded abstractions that support multimodal reasoning, planning, and memory, progressing from delegated tools through late fusion to native early-fusion processing.Delegated perception is modular but loses information in text translation; late fusion projects modality features into the LLM space, while native processing preserves fine-grained spatial, temporal, and prosodic details.Source: §2.2 · pp. 8–9, Figure 2 · p. 8

2.3 Action

• Action closes the perception-action loop by turning plans into tool or API calls, code execution, interface navigation, or physical manipulation while incorporating environmental feedback into memory.Modern action modules support both digital workflows and embodied robot control.Source: §2.3 · p. 9

2.4 Modalities

• Text remains a foundation for reasoning, code generation, tool orchestration, and interface abstraction, while images support visual reasoning, GUI navigation, and robot action generation.Frameworks range from text-only web and software agents to vision-tool compositions and vision-language-action models.Source: §2.4 · pp. 9–10
• Video and audio require modality-specific strategies: video systems address temporal dependencies and computational cost, whereas audio agents orchestrate editing, generation, retrieval, and synchronized multimodal production.Text and audio transcripts provide stable semantic abstractions, but dense images and video require native processing to preserve spatial and temporal details.Source: §2.4 · p. 10

3.1 Multimodal Perception Strategies

• Multimodal perception spans delegated tool calling, late fusion through projected encoder features, and early fusion with unified tokenization, reflecting increasing integration between sensory inputs and reasoning backbones.Delegated systems convert images or audio into textual descriptions; late-fusion systems inject encoded visual tokens into language models; early-fusion systems process text, pixels, and audio through shared attention, with newer models also unifying understanding and generation.Source: §3.1 · p. 11, §3.1.1 · p. 11, §3.1.2 · p. 11

3.2 Multimodal Reasoning and Planning

• Multimodal reasoning progresses from language-only thought traces and tool actions to visually grounded coordinates and cross-modal inference over asynchronous sensory inputs.Language-based methods support decomposition, branching, action interleaving, and reflective memory; grounded methods anchor decisions to pixels, regions, or temporal video dynamics; cross-modal systems bind heterogeneous modalities for inference and generation.Source: §3.2 · p. 12, §3.2.1 · p. 12, §3.2.2 · p. 12

3.3 Multimodal Memory Architecture

• Modality-specific memory separates media from textual indices for faster retrieval, whereas unified memory stores interleaved multimodal context and shared embeddings for content-based cross-modal recall.Modality-specific systems retrieve text timestamps or frame identifiers before accessing media, while unified systems support direct attention to multimodal tokens and hierarchical memory objects representing holistic past events.Source: §3.3 · p. 13, §3.3.1 · p. 13, §3.3.2 · pp. 13–14

3.3.3 Temporal Context Management / 3.3.4 Bandwidth Management

• Temporal context and interaction bandwidth are managed through compression, retrieval, recency limits, sparse sampling, symbolic event memory, and compact narration, but compression can lose multimodal information.VideoAgent retrieves an average of 8.4 frames per query, reducing processed frames 20x, while VLog reports a 10-20x speedup over baselines by reducing retrieval complexity to vocabulary size.Source: §3.3.3 · p. 14, §3.3.4 · p. 14

3.4 Multimodal Action Spaces

• Multimodal action spaces extend from language-mediated API and tool commands to visually grounded coordinates and embodied continuous motor control.Language-driven tools can accept or produce non-textual data and ToolLLM supports over 16,000 RESTful APIs; visual actions target interface elements or object pixels, while VLA systems generate robot actions and maintain state from multisensor inputs.Source: §3.4 · p. 14, §3.4.1 · pp. 14–15, §3.4.2 · p. 15

3.5 Evaluation Methodologies

• The taxonomy’s evaluation audit identifies five assessment categories and finds that LLM-as-a-Judge comprises only ∼5% of surveyed frameworks, exclusively in GUI and web navigation.The other categories are environment task completion, programmatic functional correctness, deterministic automated metrics, and human evaluation.Source: §3.5 · p. 15
• Robotics frameworks rely exclusively on task completion or success rates in simulated or physical environments, while long-form video systems use deterministic metrics such as multiple-choice accuracy and temporal IoU.No LLM-based evaluator is involved in the representative robotics or physical-embodiment frameworks described.Source: §3.5 · pp. 15–16
• Multimedia generation combines deterministic benchmarks with human studies, whereas GUI and web navigation uses restricted LLM-based evaluation alongside programmatic validators, including WebVoyager’s 85.3% agreement with human evaluators.WebArena uses GPT-4 fuzzy matching for some information-seeking tasks, but most evaluation relies on programmatic state validators and exact or substring matching.Source: §3.5 · p. 16

4 Application

• Agentic applications evolved from text-based information retrieval and software engineering toward prominent multimodal application areas developed in recent years.The survey uses Figure 3 to classify representative frameworks across major application areas.Source: §4 · p. 16

Robotics &

• Robotics and driving evaluations use task success, planning success, L2 distance, and collision rate, with EMMA listed as a representative driving framework.These metrics cover both task completion and driving-specific performance.Source: Application · p. 17, Robotics & · p. 17
• Robotics frameworks include SayCan, Inner Monologue, Code as Policies, PaLM-E, RT-2, OpenVLA, Octo, and Magma, targeting simulated and real-world task completion.Reported metrics include task success rate and planning success rate.Source: Application · p. 17

Navigation

• GUI and web-navigation frameworks include WebAgent, AutoDroid, AssistGUI, AutoWebGLM, SeeAct, CogAgent, SeeClick, GUI-Owl, Ferret-UI, Pix2Act, and WebGUM.Related systems include WebGPT, AppAgent, Mobile-Agent, and MM-Navigator.Source: Robotics & · p. 17, Navigation · p. 17
• GUI and web-navigation evaluations use state validation, exact or substring matching, element accuracy, grounding accuracy, and step success rate.These metrics assess both interface grounding and successful action sequences.Source: Robotics & · p. 17
• Navigation studies also use human preference and task completion, GPT-4V trajectory evaluation, GPT-4 fuzzy matching, and benchmarks including WebVoyager and the WebArena information-seeking subset.The listed evaluations combine human judgments, LLM-based assessments, and benchmark testing.Source: Navigation · p. 17

Application

• Across representative frameworks, deterministic metrics such as task success rate and grounding accuracy are generally preferred over LLM-based judgments.Table 1 summarizes the evaluation metrics used across these frameworks.Source: Table 1 · p. 17

Question Answering and Self-Correction

• ReAct (Yao et al., 2022b) improved multi-hop question answering by interleaving reasoning traces with environment actions, while Reflexion (Shinn et al., 2023a) used verbal reflection and textual memory for self-correction.WebGPT (Reiichiro Nakano et al., 2021) connected agents to internet search, and Reflexion achieved state-of-the-art performance on ALFWorld and HotpotQA within text-only contexts.Source: §4.1 · p. 17

Multi-Agent Collaboration and Software Development

• MetaGPT (Hong et al., 2023) assigned specialized roles to LLM instances, enabling collaborative generation of complex software repositories from high-level instructions.Generative Agents (Park et al., 2023) showed that 25 text-based agents with independent memories and reflections could develop emergent social behaviors, a principle used by ChatDev (Qian et al., 2023).Source: §4.1 · pp. 17–18

Generalist Tool and API Calling

• ToolFormer (Dwivedi-Yu et al., 2023), ToolkenGPT (Hao et al., 2023), RestGPT (Song et al., 2023), and ToolLLM (Qin et al., 2023) progressively expanded language-based API calling from learned invocations to 16,000+ real-world APIs.ToolkenGPT used learnable tool embeddings for thousands of tools, while RestGPT applied coarse-to-fine planning to documentation and JSON responses.Source: §4.1 · p. 18

Interface and Limitations

• Text-only applications treat language as the primary interface for reasoning, planning, and tool orchestration, often converting structured web or GUI inputs into text.This design is efficient and easy to control but limits access to visual and temporal details needed for more complex tasks.Source: §4.1 · p. 18

Grounding and Evaluation

• Embodied agents must ground perceptual inputs to plan successfully, motivating evaluation beyond text environments toward real-world actuation datasets and physical control.Earlier systems were evaluated in ALFWorld and WebShop, while newer single-model agents generate physical actuation signals and are evaluated on BridgeData V2 and GNM.Source: §4.2 · p. 18

Grounded Planning with LLMs

• Grounded planning evolved from SayCan’s semantic-affordance feasibility checks through feedback-driven replanning and code-based policy generation that handles spatial constraints.SayCan combines language and affordance probabilities; Inner Monologue uses action-success feedback; Code as Policies generates executable Python for complex actions.Source: §4.2 · p. 18

Multimodal Reasoning with MLLMs

• MLLM planners use visual reasoning and hierarchical or dual-loop designs to separate high-level intent from precise manipulation and adapt planning through environmental feedback.ViLa, GPTArm, COME-robot, and EMMA apply visual understanding to spatial reasoning, grasping, open-ended tasks, or trajectory prediction; COME-robot achieves significantly higher success rates in open-ended tasks.Source: §4.2 · pp. 18–19

Vision-Language-Action Models

• Explicit embodied reasoning further strengthens VLA control: CoT-VLA generates subgoal images before actions and outperforms prior VLAs by 17% on real-world manipulation tasks.HALO extends this approach with multimodal chain-of-thought reasoning that jointly performs textual reasoning, visual foresight, and action prediction for long-horizon and out-of-distribution scenarios.Source: §4.2 · p. 19

Multi-Agent Systems

• Multi-agent robotics uses language-mediated negotiation, shared visual maps, and skill-aware task decomposition to coordinate heterogeneous robots beyond individual-model capabilities.RoCo resolves assignments and spatial conflicts through dialogue, Co-NavGPT distributes exploration using visual sub-maps, and SMART-LLM forms coalitions when subtasks exceed one robot’s abilities; combining these strategies with RT-2 or OpenVLA remains underexplored.Source: §4.2 · pp. 19–20

Vision-Language-Action Models

• Vision-language-action models increasingly unify perception, reasoning, and control, mapping multimodal inputs to grounded or high-frequency robot actions across diverse embodiments.PaLM-E, RT-2, Octo, OpenVLA, π0, π0.5, VTAM, and GR00T N1 extend unified control through embedding-space grounding, tokenized actions, flow matching, tactile prediction, open-world scaling, or parameter-efficient adaptation.Source: §4.2 · pp. 19–20

Text-Based Tool-Augmented Agents

• Text-based agents convert HTML, UI XML, or visual-tool outputs into simplified text for LLM planning, enabling web, mobile, and desktop automation but losing spatial information under context limits.Representative systems include WebGPT, WebGLM, WebAgent, AutoDroid, AssistGUI, and AutoWebGLM.Source: §4.3 · pp. 21, 23

Fine-tuned Late-fusion Multimodal Agents

• Late-fusion multimodal agents encode screenshots with specialized vision modules and adapters, directly predicting actions or coordinates to preserve high-resolution and pixel-level UI information.CogAgent processes 1120 × 1120 images, Ferret-UI adapts to mobile aspect ratios, and SeeClick targets precise screen coordinates.Source: §4.3 · pp. 21–22

Natively Multimodal Agents

• Natively multimodal frameworks use LMMs as reasoning cores for visual web and mobile interaction, adding grounding, token-selection, trajectory-generation, memory, or specialized execution modules.ShowUI and FocusUI compress visual tokens while preserving grounding information; ShowUI-π unifies clicks and drags as cursor-waypoint trajectories; UI-TARS-2 and Agent S2 improve long-horizon reliability through reinforcement learning and delegated grounding.Source: §4.3 · p. 22
• 45.0% on OSWorld-Verified, OpenCUA-72B establishes a new state-of-the-art among open-source computer-use models with a 22,600-task cross-OS dataset.GUI-Genesis reduces environment latency by 10× and training costs by over $28,000 per epoch versus training on real applications.Source: §4.3 · p. 22

Multi-Agent Systems

• Modular and multi-agent systems divide perception, planning, decision-making, evaluation, or execution across specialized components to mitigate context degradation and generate training data.Cradle uses six modules, Mobile-Agent-v2 separates planning from decision-making, OpenAgents routes queries among specialized agents, and AgentTrek pairs an executing VLM with an evaluator.Source: §4.3 · pp. 22–23

Natively Multimodal Agents

• Natively multimodal agents avoid domain-specific training, but reliable GUI interaction requires auxiliary visual tagging and explicit self-reflection because spatial grounding remains weak.SeeAct iteratively refines plans from visual feedback, while Mobile-Agent corrects navigation errors when visual state changes violate expectations.Source: §4.3 · pp. 22–23

Text-Based Tool-Augmented Agents

• Text-based agents convert visual or audio inputs into text or code, routing semantic instructions to specialized foundation models for multimedia generation and editing.VISPROG (Tanmay Gupta, 2022), AudioGPT (Huang et al., 2023a), WavJourney (Liu et al., 2023b), WavCraft (Liang et al., 2024), and LAVE (Wang et al., 2024a) exemplify this approach.Source: §4.4 · pp. 24–25
• These early frameworks rely on neuro-symbolic orchestration, confining LLMs to semantic routing and code generation while specialized external models perform media manipulation.This design allows text-only LLMs to coordinate multimodal generation despite lacking direct non-textual perception.Source: §4.4 · p. 25

Natively Multimodal Agents

• Native multimodal agents directly perceive media and combine planning or tuning-free algorithmic constraints to reduce search costs and preserve visual or temporal consistency.GenArtist (Wang et al., 2024f) uses visual feedback for planning and self-correction, while CoSTA* (Gupta et al., 2025b), FaSTA* (Gupta et al., 2025a), FLATTEN (Cong et al., 2023), UniEdit (Bai et al., 2025a), and AnyV2V (Ku et al., 2024) use search, reusable subroutines, optical flow, inversion, or feature manipulation.Source: §4.4 · p. 25

Multi-Agent Systems

• Multi-agent frameworks assign specialized creative roles to iterative critique, revision, production, and refinement loops that support semantic alignment, creative diversity, synchronization, and reduced error accumulation.CREA (Venkatesh et al., 2025) surpasses single-agent baselines in semantic alignment and creative diversity, while ReelWave (Wang et al., 2025d), EditDuet (Sandoval-Castaneda et al., 2025), Paper2Poster (Pang et al., 2026), and Paper2Video (Zhu et al., 2025) extend role specialization across audiovisual production, editing, and academic media generation.Source: §4.4 · p. 25

Iterative Retrieval Agents

• Iterative retrieval agents reduce video-processing costs by querying only relevant frames, with VideoAgent achieving state-of-the-art EgoSchema performance using 20x fewer frames than dense sampling.AssistGPT interleaves planning, execution, inspection, and learning, while LLoVi performs temporal reasoning over textual descriptions generated from sampled frames.Source: §4.5 · p. 27
• Early retrieval frameworks overcome token bottlenecks through lossy external searches or text proxies that sacrifice fine-grained continuous visual details.Long-form video produces hundreds to thousands of visual tokens per minute, causing quadratic attention-cost growth that overwhelms standard context windows.Source: §4.5 · p. 27

Specialized Fine-tuned Reasoning Agents

• Specialized agents achieve deep temporal grounding without massive compute scaling by modularizing reasoning through dynamic LoRA adapters or redesigned video representations.VideoMind assigns Planner, Grounder, and Verifier roles via Chain-of-LoRA, while VLog enables generative retrieval over compositional narration units.Source: §4.5 · p. 27

Natively Multimodal Agents

• Native MLLMs ingest hour-scale videos directly, but benchmarks show strong short-term recognition alongside persistent difficulty with long-horizon causal reasoning.DoraemonGPT addresses this gap by explicitly modeling dynamic scenes and tracking object states over time; VideoRAG and VideoDeepResearch extend retrieval-based reasoning through multimodal tools and external databases.Source: §4.5 · pp. 27–28
• 74.2% accuracy on LVBench enables DVD to substantially surpass all prior works through autonomous adaptive planning over multi-granular video databases.DVD dynamically selects search-centric tools and iteratively refines reasoning instead of following rigid predefined workflows.Source: §4.5 · p. 28

5.1 Robotics and Physical Embodiment

• Early-fusion robotics models improve generalization: RT-2 reaches 62% on novel objects versus RT-1’s 32%, while 7B OpenVLA surpasses 55B RT-2-X by 16.5% across 29 tasks.RT-2 reaches 90% on Language Table; π0 provides 50 Hz control from a 3B VLM, and π0.5 performs 10–15-minute manipulation sequences in unseen homes.Source: §5.1 · p. 29

5.2 GUI and Web Navigation

• GUI agents improve from text-mediated baselines to native multimodal and decomposed systems, but GPT-4-based agents achieve only 14.41% on WebArena versus 78.24% for humans.UI-TARS-2 reaches 47.5% on OSWorld and 88.2% on Online-Mind2Web, while Agent S2 reaches 34.5% on OSWorld’s 50-step evaluation; persistent failures involve OCR, coordinate grounding, planning, and recovery.Source: §5.2 · p. 30

5.3 Long-Form Video Understanding

• Selective retrieval improves long-video understanding and efficiency: VideoAgent reaches 54.1% on EgoSchema using 8.4 frames on average, 3.8 points above LLoVi and with 2–30× efficiency gains.VideoMind and VLog further show that modular specialization and retrieval reformulation can replace parameter scale; VLog’s 124M model matches LLaMA-2-7B while achieving 10–20× speedup over generative baselines.Source: §5.3 · p. 31

5.4 Multimedia Content Generation and Editing

• Native visual feedback improves multimedia generation: GenArtist exceeds DALL-E 3 by over 20% in spatial-relation accuracy, with hierarchical planning adding 7–10% improvement.However, evaluation remains constrained by subjective success criteria, limited realistic multi-turn benchmarks, noisy or domain-specific datasets, and the need for custom benchmark construction.Source: §5.4 · p. 32

5 Performance Analysis

• Across domains, tighter perception–reasoning integration reduces information bottlenecks, with robotics gains of 20–30 percentage points, efficient video attention, and visual feedback enabling iterative refinement; GUI agents remain 60–70 points below humans.The section attributes these gains to preserved visual semantics, unified action tokenization, task-relevant computation, and output verification, while identifying grounding precision, planning depth, and failure recovery as unresolved GUI limitations.Source: §5 · pp. 28–29, §5.4 · p. 32

6 Efficiency

• Efficiency is defined by managing high-bandwidth visual, video, and audio inputs within temporal constraints using inference latency, throughput, and memory overhead.These metrics expose the trade-off between API-based modularity and the low-latency response times of optimized, fine-tuned models.Source: §6 · pp. 32–33

6.1 Robotics and Physical Embodiment

• π0 reaches up to 50 Hz, compared with 1–3 Hz for RT-2 and 6 Hz for OpenVLA, enabling closed-loop dexterous manipulation.OpenVLA uses a 7B backbone with 4-bit quantization to achieve approximately 6 Hz on a single consumer-grade GPU, while Octo uses 27M- and 93M-parameter checkpoints for faster deployment.Source: §6.1 · p. 34

6.2 GUI and Web Navigation

• WebGLM reduces average response time to approximately 5.36 seconds from WebGPT’s approximately 52 seconds, while GUI-Owl retains only the most recent 1 to 3 images in context.This context strategy saves GPU memory and improves inference speed.Source: §6.2 · p. 34

6.3 Multimedia Content Generation and Editing

• Delegated systems improve efficiency on simple multimedia tasks through selective tool execution, but FaSTA* uses fast–slow planning and toolpath reuse for costly multi-turn editing.VISPROG and AudioGPT avoid unified multimodal training, whereas CREA adds sequential planning, critique, and scoring calls that increase latency.Source: §6.3 · p. 35

6.4 Long-Form Video Understanding and Retrieval

• VideoAgent processes 20× fewer frames, while VLog achieves 10-20× speedup through generative retrieval, illustrating the efficiency of delegated and late-fusion strategies.Native systems also improve efficiency through scaling and architectural sparsity: Gemini 1.5 yields 26–75% real-world time savings, GPT-4o has approximately 232 ms audio latency and roughly 50% lower serving cost, and VideoLLM-MoD saves roughly 42% time and 30% memory.Source: §6.4 · pp. 36–37

7 Scalability

• Scalability is evaluated through data efficiency, modularity, and deployment costs, balancing upfront training infrastructure against recurring per-token API expenses.Data efficiency includes few-shot prompting and parameter-efficient fine-tuning; modularity measures integration of new modalities or tools without full retraining, while deployment compares proprietary APIs with local or domain-specific models.Source: §7 · p. 37

7.1 Robotics and Physical Embodiment

• In robotics, fine-tuned late-fusion agents such as Octo and OpenVLA appear most viable for scaling because efficient adaptation, LoRA, and quantization mitigate high training costs.OpenVLA required 21,500 A100 hours for pre-training, while Octo enables fine-tuning to new observation and action spaces within hours; API-based agents avoid training compute but incur recurring visual-query and planning costs.Source: §7.1 · p. 37

7.2 GUI and Web Navigation

• GUI and web agents scale more sustainably with specialized or locally trained models, since token-heavy proprietary APIs incur recurring costs at every interaction step.WebAgent uses a domain-expert planner, AutoWebGLM uses 6-billion-parameter ChatGLM3-6B, and trained models such as CogAgent-18B and GUI-Owl-7B offer favorable performance-to-parameter ratios after costly preparation.Source: §7.2 · p. 38

7.3 Multimedia Content Generation and Editing

• Multimedia generation and editing scale through modular execution and reuse of existing models, with FaSTA* and AnyV2V highlighted for efficient multi-turn editing.Tool-use systems decompose instructions into executable steps over specialized modules, while native multimodal systems reuse pretrained backbones, plug-in editors, fast–slow planning, and tuning-free branches without retraining.Source: §7.3 · pp. 38–39

7.4 Long-Form Video Understanding and Retrieval

• Native multimodal video agents additionally scale through unified long-context architectures, with Gemini 1.5 Pro processing millions of tokens across text, video, and audio without chunking or external retrieval.GPT-4o integrates text, vision, audio, and video in one architecture, while Video-MME evaluates scalability from short to hour-long videos as sequence length grows.Source: §7.4 · p. 39
• Long-form video agents scale by extracting key temporal cues through retrieval, abstraction, planning, parameter-efficient tuning, and compact representations rather than processing every frame.VideoAgent retrieves approximately 8.4 frames per query on average; VideoMind uses LoRA-based role adapters, while VLog and related systems compress long videos into reusable textual narrations.Source: §7.4 · p. 39

8 Latency

• Latency strategies include prompt pruning, token reduction, architectural innovations, and fewer model calls, with robotics and GUI/web navigation requiring stricter responsiveness than multimodal understanding or editing.Application requirements determine acceptable latency: real-time embodied and navigation systems are less forgiving than end-to-end understanding and editing systems.Source: §8 · p. 40

8.1 Robotics and Physical Embodiment

• GPTArm’s adaptive feedback reduced complex-task inference latency by ∼50% but lowered complex-task SR by ∼9%, while EMMA’s omitted CoT steps improved latency by 67%.Other robotics strategies include RT-2’s distributed cloud inference, Octo’s 27 - 93M local backbones, and OpenVLA’s 4-bit quantization achieving ∼3Hz on an NVIDIA A5000 GPU.Source: §8.1 · p. 40
• Robotics deployment remains difficult because external APIs add network interruptions, reduced visual verification can cause failures, and fused LiDAR/radar-camera contexts increase inference latency.Even single-digit control frequencies may be insufficient for smooth motor control in safety-critical environments.Source: §8.1 · p. 40

8.2 GUI and Web Navigation

• AutoDroid reduced inference latency by 21.3%, CogAgent cut visual processing cost by ∼25x, and WebGLM reduced webpage retrieval latency from ∼2 mins to 5 secs.AutoDroid also nearly halved prompt tokens and used memory shortcuts, while WebAgent summarized long pages and CogAgent used a locally deployed VLM with linear visual-feature complexity.Source: §8.2 · p. 40
• GUI and web agents remain constrained by proprietary APIs, large local-model compute, long-context failures, grounding errors, and dynamic webpages requiring continual reflection and state verification.WebVoyager can enter navigation loops or hallucinate, while SeeAct and MM-Navigator struggle to map high-level intentions to localized UI coordinates.Source: §8.2 · pp. 40–41

9 Reliability

• Reliability failures arise from out-of-distribution data, unexpected intermediate outputs, and cascading errors, including hallucinated bounding boxes, incorrect spatial captions, and outdated state representations.These failures are often harder to identify than latency, motivating targeted containment mechanisms across application domains.Source: §9 · p. 41

9.1 Robotics and Physical Embodiment

• Closed-loop verification, affordance scoring, and unified perception-to-action models contain cascading physical-task failures by validating states, action feasibility, and preserved spatial-semantic information.Inner Monologue (Huang et al., 2022) and COME-robot (Zhi et al., 2025a) verify outcomes and replan; SayCan (Ahn et al., 2022) scores semantic validity and physical feasibility; RT-2 (Ichter et al., 2023) and OpenVLA (Kim et al., 2024) reduce modular information loss.Source: §9.1 · p. 41
• Approximately 9% lower success on complex tasks follows reduced verification frequency, exposing a structural reliability-throughput tradeoff, while VLA models still degrade on unseen platforms and data despite Octo’s diverse-configuration training.Dense verification improves recovery but reduces control frequency through remote API calls, and out-of-distribution generalization remains open.Source: §9.1 · p. 41

9.2 GUI Grounding and Web Navigation

• GUI agents improve reliability through two-pass grounding, post-action state checks, compressed progress summaries, and retrieved app-behavior references that reduce dependence on zero-shot generalization.SeeAct (Zheng et al., 2024), WebVoyager (He et al., 2024a), Mobile-Agent (Wang et al., 2024c), Mobile-Agent-v2 (Wang et al., 2024b), and AppAgent-v2 (Li et al., 2024) implement these mechanisms.Source: §9.2 · pp. 41–42
• A persistent 60-70% performance gap versus humans on WebArena reflects GUI agents’ unresolved grounding, context, and live-environment reliability failures.Unexpectedly long contexts can cause WebVoyager (He et al., 2024a) navigation loops; interruptions and unpredictable loading require continuous verification, while Code2World (Zheng et al., 2026) evaluates candidate actions by simulating interface states before execution.Source: §9.2 · p. 42

Grounding and reasoning

• Agents remain ∼60-70% behind humans on WebArena and VisualWebArena, reflecting inadequate coordinate-level grounding and multi-step error recovery despite improved semantic perception.These gaps persist in high-precision GUI navigation tasks.Source: §10 · p. 42

Efficiency and scalability

• Native multimodal models offer strong generalization and zero-shot performance but incur prohibitive inference costs, whereas efficient fine-tuned models require substantial training compute and generalize less.No surveyed architecture balances state-of-the-art performance with deployment feasibility.Source: §10 · p. 42

Memory and long-horizon reasoning

• Long-horizon memory architectures accumulate errors, resist belief revision, and fail to maintain consistent, temporally coherent states, even when retrieval limits processing to relevant video frames.Current systems also lack robust mechanisms for forgetting outdated information during extended interactions.Source: §10 · p. 42

Verification and benchmarking

• Closed-source API orchestration prevents independent verification and systematic ablations, while internet-derived benchmarks may leak data and inflate apparent generalization through memorization.This combination limits understanding of failure modes and slows open scientific progress.Source: §10 · p. 43

Adversarial robustness

• Richer sensory inputs expand attack surfaces: covert visual prompt injections and subtle physical perturbations can corrupt perception, control flow, and safety-critical planning (Foerster et al., 2026; Eslami & Yu, 2025).These risks span digital agents and embodied systems such as autonomous driving.Source: §10 · p. 43

Safety and hallucination

• Weak grounding and limited foresight cause instruction drift, hallucinated actions, and unsafe navigation, while brittle self-correction cannot replace predictive world models for evaluating downstream consequences (Ma et al., 2024a; Jalaian et al., 2025; Ying et al., 2025; Chen et al., 2026a; Chen et al., 2026b).Agents may hallucinate affordances or invalid plans even when perception is accurate, especially in complex, dynamic, long-horizon environments.Source: §10 · p. 43

Unified Multimodal Representation and Reasoning

• Planning-oriented multimodal representations should dynamically reweight modalities, preserve spatial-temporal structure, support symbolic reasoning, and formalize links between perception, planning, and tool selection.These representations should be optimized for downstream planning and control rather than treated solely as embeddings.Source: §11 · p. 43

Scalable Memory Architectures for Lifelong Learning

• Lifelong-learning agents need multi-timescale memories that detect conflicting knowledge, trace its sources, revise beliefs coherently, and reason about their own knowledge state beyond retrieval-augmented generation.This direction moves beyond passive storage used by current systems (Shinn et al., 2023b; Zhang et al., 2025b).Source: §11 · pp. 43–44

Hybrid Architectures for Resolving Trade-offs

• Hybrid architectures could combine large natively multimodal and domain-specific models through hierarchical routing, distillation, or Mixture-of-Experts designs to reduce costs and computational overhead while preserving capability.Hierarchical systems could assign routine tasks to lightweight models and edge cases to larger models, while distillation could enable local deployment; the hierarchical direction is inspired by (Gupta et al., 2025a).Source: §11 · p. 44

Multimodal Integration Beyond Vision-Language

• Multimodal agents should extend beyond vision-language fusion to tactile and force sensing, 3D spatial reasoning, and cross-modal reasoning for embodied manipulation, navigation, and ambiguity resolution.Cross-modal reasoning can let one modality resolve ambiguity in another, as shown in (Galougah et al., 2025b).Source: §11 · p. 44

Multimodal Multi-Agent Communication

• Multi-agent systems need standardized, efficient, and verifiable multimodal communication protocols that address latency, ambiguity, information loss, interoperability, interpretability, and safety limitations.Promising directions include latent-space communication (Zou et al., 2025; Yu et al., 2026b), Universal Visual Codec alignment (Liu et al., 2026), structured spatial protocols (Padhan et al., 2026), and program-level embodied communication (Shi et al., 2026).Source: §11 · p. 44

• Tighter perception–reasoning integration yields the largest gains, with compact end-to-end robotics systems outperforming substantially larger pipelines that separate perception from action.Source: §12 · p. 44
• Web-navigation agents remain far from human-level reliability, especially on tasks requiring precise control and error recovery.Source: §12 · p. 44
• Native large multimodal models offer strong zero-shot capabilities, but token-heavy inference creates latency and cost that limit practical scalability.Source: §12 · p. 44
• Domain-specific and fine-tuned systems provide a more sustainable efficiency trade-off; OpenVLA (Kim et al., 2024) and CogAgent (Hong et al., 2024) amortize training costs through faster inference.Source: §12 · p. 44
• In long-form video generation, VideoAgent (Wang et al., 2024e), VLog (Lin & Shou, 2025), retrieval-based designs, and frozen backbones reduce redundant computation.Source: §12 · p. 44