PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, Jianfeng Gao

arXiv:2401.03568v2cs.AIcs.HCcs.LG

TL;DR

Agent AI addresses the challenge of building interactive multimodal systems that can interpret grounded environmental information and produce meaningful embodied actions. This paper surveys the field, its training approaches, applications, evaluation, and ethical boundaries, while highlighting multimodal action generation and cross-domain generalization as central goals.

  • Problem

    Agent AI must integrate multimodal environmental understanding with meaningful action across varied domains, where differing visual appearances and action spaces make generalist policies difficult to develop.

  • Method

    The paper provides a comprehensive survey of Agent AI foundations, architectures, training methodologies, applications, datasets, evaluation practices, ethical considerations, and future directions.

  • Results

    The survey synthesizes representative Agent AI research across multimodality, gaming, robotics, and healthcare, including evaluation guidance focused on effectiveness and generalization.

  • Takeaways & Limitations

    Grounded multimodal agents provide a framework for studying and advancing systems that connect perception, language, contextual understanding, and embodied action.

  • Takeaways & Limitations

    Policies learned in simulation may face distribution shift when applied to the real world, while Internet-collected datasets may contain biases and lack full representativeness.

Abstract

from arXiv · show

Multi-modal AI systems will likely become a ubiquitous presence in our everyday lives. A promising approach to making these systems more interactive is to embody them as agents within physical and virtual environments. At present, systems leverage existing foundation models as the basic building blocks for the creation of embodied agents. Embedding agents within such environments facilitates the ability of models to process and interpret visual and contextual data, which is critical for the creation of more sophisticated and context-aware AI systems. For example, a system that can perceive user actions, human behavior, environmental objects, audio expressions, and the collective sentiment of a scene can be used to inform and direct agent responses within the given environment. To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can produce meaningful embodied actions. In particular, we explore systems that aim to improve agents based on next-embodied action prediction by incorporating external knowledge, multi-sensory inputs, and human feedback. We argue that by developing agentic AI systems in grounded environments, one can also mitigate the hallucinations of large foundation models and their tendency to generate environmentally incorrect outputs. The emerging field of Agent AI subsumes the broader embodied and agentic aspects of multimodal interactions. Beyond agents acting and interacting in the physical world, we envision a future where people can easily create any virtual reality or simulated scene and interact with agents embodied within the virtual environment.

1 Introduction

This paper surveys Agent AI as a holistic research direction integrating language, vision, memory, reasoning, adaptability, and embodied interaction. It reviews foundations, methodologies, applications, evaluation, ethics, and emerging trends across multimodal agent systems.

  • Motivation: Agent AI research seeks to reunify AI capabilities that have developed across specialized subfields into more holistic systems.The paper connects this goal to historical efforts such as the Copy Demo and to recent progress in LLMs and VLMs.
  • Agent AI Overview: Multimodal Agent AI systems generate effective actions in environments by understanding multimodal sensory input.The paper focuses on multimodality, gaming, robotics, and healthcare as representative research areas.
  • Methodologies: The survey examines how LLMs and VLMs support Agent AI through methodologies illustrated in gaming, robotics, and healthcare case studies.Its scope includes foundation models, embodied task planning, and interactive learning approaches.
  • Evaluation and Future Directions: The paper provides guidance for evaluating Agent AI with relevant datasets, emphasizing effectiveness and generalization.It also discusses ethical considerations, societal impacts, responsible development, emerging trends, and future directions.
  • Paper Scope: The paper is organized around Agent AI technologies, training frameworks, methodologies, agent types, applications, cross-domain versatility, continual learning, datasets, and ethics.This structure is intended to provide general and comprehensive knowledge of current Agent AI research.

2 Agent AI Integration

Agent AI integrates foundation models with embodied, multimodal interaction to address limited performance in unseen environments and enable scene understanding, generation, editing, and collaboration across physical and virtual worlds. The section also frames grounding, expert demonstrations, external knowledge, and human guidance as complementary ways to improve agent behavior and mitigate hallucinations and bias.

  • Motivation: Foundation models still show limited embodied-AI performance in understanding, generating, editing, and interacting within unseen environments, producing sub-optimal agent outputs.Current approaches often depend on accessible world-state representations and domain-specific data, which limits deployment in new settings.
  • Infinite AI agent: Infinite agents transfer memory from general foundation models to novel physical or virtual domains for scene understanding, generation, and interactive editing.The proposed direction targets new tasks without requiring extensive training data for every domain.
  • Grounding and risks: External retrieval and grounded interaction are presented as routes for reducing hallucinations, while multimodal agents remain vulnerable when relying mainly on pretrained internal knowledge and limited environment-specific fine-tuning.Vision-language hallucinations can arise from over-reliance on object co-occurrence and visual cues in training data.
  • Biases and inclusivity: Foundation-model agents can reproduce societal, cultural, linguistic, and overgeneralization biases because training data and model behavior reflect limitations in their sources and learned patterns.The paper specifically highlights internet-heavy and often English-only training data as embedding norms disproportionately associated with WEIRD societies.
  • Imitation learning: Decoupled imitation learning uses diverse expert state-action pairs rather than task-specific rewards, allowing policies to generalize across tasks without explicit reward-function dependence.The agent learns a state-to-action policy by imitating expert behavior, while an infinite-memory agent collects physical-world demonstrations.
  • Emergent interaction: Knowledge-guided interaction combines foundation-model knowledge, web information, and human collaboration to support adaptation and scene generation in complex or unseen environments.The paper names this mechanism Mixed Reality with Knowledge Inference Interaction and describes it as supporting exploration of unseen environments for virtual-reality adaptation.

3 Agent AI Paradigm

The proposed Agent AI paradigm combines pretrained multimodal capabilities with agent-specific mechanisms for planning, memory, environmental feedback, and specialized action. It supports both modular LLM/VLM-bootstrapped systems and unified transformers with agent tokens.

  • Agent AI Paradigm: The framework aims to bootstrap agents with pretrained models while supporting long-term planning, learned memory, and environmental feedback.These goals target understanding of text and visual inputs, retrieval of learned knowledge, and learning which actions to take.
  • Agent AI Paradigm: LLMs and VLMs can initialize agent components because they provide task-planning, world-knowledge, reasoning, and language-aligned visual capabilities.The passage identifies LLMs for planning and reasoning and VLMs such as CLIP for visual encoding and zero-shot recognition.
  • Agent Transformer Definition: A unified agent transformer combines visual, language, and agent tokens instead of relying only on frozen LLM and VLM submodules.Agent tokens reserve input and output space for domain-specific behaviors such as robotics actions, game actions, and tool calls.
  • Agent Transformer Definition: Compared with large proprietary language models, smaller agent transformers can be customized for specific actions and may reduce cost or address strict data-privacy requirements.The cited motivation includes domains such as healthcare and law, where data privacy is especially important.
  • Agent AI Paradigm: The paradigm can use foundation-model outputs to train specialized agent transformers tailored to particular tasks and domains.The process defines domain objectives and action spaces, collects evaluable data, and then applies continuous improvement using monitoring and feedback.

4 Agent AI Learning

Agent AI learning combines interactive feedback, foundation-model assistance, reinforcement learning, imitation learning, prompting, memory, and spatial-temporal optimization. The section also emphasizes data, reward-design, long-horizon, and generalization challenges.

  • Interactive Learning: Interactive agent improvement can collect user feedback, action information, and knowledge, either by updating models or by improving contextual prompts at test time.The strategy may use a trained agent to gather information while leaving the underlying LLM/VLM unchanged.
  • Reinforcement Learning: Reinforcement learning learns state-action relationships from rewards or penalties but faces difficulties in reward design, data efficiency, and long-horizon credit assignment.Longer action sequences increase ambiguity between actions and rewards and expand the state space requiring exploration.
  • Reinforcement Learning: LLMs can support long-horizon robotics by handling high-level task planning while RL-based policies address low-level control.Task-and-motion planning separates high-level subtask sequencing from physically consistent, collision-free trajectories.
  • Imitation Learning: Imitation learning uses expert data to reproduce experienced behavior, with behavioral cloning directly copying expert actions.This contrasts with RL, which learns through exploratory interactions and reward maximization.
  • Optimization and Infrastructure: Agent development also uses synthetic data, data augmentation, in-context examples, batch or self-play RL, memory, and spatial-temporal optimization.Self-play can improve teams over time but may produce brittle agents that overfit to self-play and fail with humans or independent agents.
  • Agent Modules: Agent Modules explore minimalist training or in-context learning systems with prompt-based memory to improve planning and inform future actions.The broader infrastructure effort is constrained by the cost and effort required for high-quality datasets and agent systems.

5 Agent AI Categorization

Agent AI spans generalist, embodied, action, interactive, XR, emotional, and localized multimodal systems. These systems connect multimodal understanding with environmental action, creation, communication, or user assistance.

  • Generalist Agent Areas: Multimodal Agent AI systems generate effective environmental actions from multimodal sensory input and share interests in data, benchmarking, and ethics.The field includes applications such as user-action prediction, task planning, video understanding, robotics, and knowledge-informed dialogue.
  • Action Agents: Action agents execute physical actions in simulated or real environments, with gaming and robotics as principal application categories.Gaming agents may communicate with humans and other entities while optimizing task-specific rewards.
  • Interactive Agents: Interactive agents have a broader scope than action agents because they can communicate information or modify environments without requiring physical action.Examples include dialogue-based question answering and assistance with parsing existing information.
  • Embodied Agents: Simulators support agent learning when physical trial-and-error is costly, time-consuming, or dangerous.The passage highlights autonomous driving and underwater vehicles as settings where physical failures can be hazardous.
  • XR Agents: XR agents aim to reduce the skilled labor required to create virtual characters, environments, objects, and interactions.The proposed tools include user-defined object behaviors, sketch-based level geometry, prompt-generated effects, and runtime environment modification.
  • Generative Agents: Few-shot GPT models have been used in Unity without additional fine-tuning to call engine methods, download assets, and assign behaviors and animations.A proposed near-term application is a storyboarding or prototyping tool that could make rough functional experience sketches an order of magnitude faster.
  • Emotional Reasoning: Empathy-oriented agents use image-commenting data with human emotion annotations and affect-conditioned generation to produce more engaging and socially appropriate comments.The NICE dataset contains almost two million images and corresponding human-generated comments and emotion annotations.
  • Neuro-Symbolic Agents: Localized visual commonsense models accept multiple image regions and can be trained from automatically generated localized knowledge without aligned human-authored image-text pairs.A separately trained critic selects higher-quality examples from the automatically constructed corpus.

6 Agent AI Application Tasks

Agent AI application tasks span gaming, navigation, multimodal video understanding, knowledge-intensive question answering, and interactive assistance. Across these settings, systems combine foundation-model capabilities with embodied action, external knowledge, multimodal inputs, or human feedback.

  • Gaming: Gaming applications use agents to support human interaction, adaptive behavior, and meaningful actions beyond scripted NPC responses.Games serve as sandboxes for testing collaboration and decision-making, while conventional scripted NPCs tend to be predictable or repetitive.
  • Gaming: Multi-agent gaming infrastructure uses GPT-4 as a central planner to support collaboration across domains and introduces Collaboration Score (CoS) for efficiency evaluation.The CuisineWorld benchmark demonstrates multi-agent and human-AI collaboration in real-world video games.
  • Interactive assistance: Scene analysis converts instructions and initial visual data into object, graspability, and spatial-relation information aligned with the human’s manipulation action.GPT-4V selected task-relevant objects, such as including a table for relocating a container but ignoring it for a fridge-opening task.
  • Embodied navigation: 10% on SPL: the RCM model significantly outperforms previous VLN methods, while self-supervised imitation learning reduces the seen–unseen success-rate gap from 30.7% to 11.7%.The approach combines global instruction–trajectory matching with local visual grounding and self-imitation for exploration.
  • Knowledge-intensive agents: Knowledge-augmented and vision-language agents extend multimodal reasoning through explicit external knowledge and learned cross-modal representations.KAT outperformed other systems by 6% on the 2022 OK-VQA task, while VLC showed effective image classification and image–text patch matching.
  • Video-language understanding: Audio transcriptions improve video descriptions by correcting visually misleading captions and capturing detailed physical actions.The multimodal agent produced more informative descriptions when audio, video, and language interactions were integrated.

7 Agent AI Across Modalities, Domains, and Realities

Generalist Agent AI must bridge missing multimodal training data, domain-specific visual and action differences, and the persistent gap between simulated and real environments.

  • Cross-modal Understanding: Large-scale datasets combining vision, language, and agent behavior are scarce, so multimodal systems commonly rely on separately trained frozen language and visual modules.RT-2 suggests that jointly tuning the visual encoder and language model can improve robotics and vision-language performance.
  • Cross-domain Understanding: Distinctive visual appearances and disparate action spaces make it difficult to train one agent policy across domains; separate domain-specific fine-tuning fails to exploit shared data.The challenge spans domains such as real-world environments, video games, robotics, and healthcare.
  • Cross-reality Understanding: Agents also struggle to understand real and virtual environments simultaneously because those environments differ visually and follow separate physics.Sim-to-real transfer is therefore identified as a particularly important cross-reality problem.
  • Sim-to-real Transfer: Simulation-trained embodied agents often perform poorly in the real world because simulations do not fully reproduce disturbances, lighting, gravity, and other physical properties.This discrepancy motivates sim-to-real transfer techniques.
  • Sim-to-real Transfer: Domain randomization, domain adaptation, and improved simulation are established approaches for reducing simulation-to-reality discrepancies.These methods vary simulated parameters, incorporate real-world images, or identify realistic physical and visual simulation parameters.
  • Sim-to-real Transfer: Sim-to-real transfer remains a central challenge requiring continued theoretical and empirical research.

8 Continuous and Self-improvement for Agent AI

Agent AI can improve through diverse interaction and model-generated data, but current foundation-model-based agents generally lack continuous environmental learning.

  • Learning Sources: Foundation-model-based agents can learn from multiple data sources, including human interaction data and artifacts generated by existing models.However, their dependence on pretrained foundation models limits continuous learning from environmental interaction.
  • Human-based Improvement: Human-agent interactions can provide training examples for future agent iterations when successful and unsuccessful examples are filtered.Filtering may use rules, models, or manual inspection and modification.
  • Human-based Improvement: Human preference learning presents users with multiple outputs and uses their selections to identify outputs aligned with their preferences.The passage cites ChatGPT and GPT-4 as examples of this strategy.
  • Human-based Improvement: Red-teaming uses human or computer adversaries to expose agent weaknesses and vulnerabilities, supporting improvements to safety measures and reductions in harmful outputs.
  • Model-generated Data: Prompting and data-pairing techniques can extract or generate training data from foundation-model artifacts.
  • Model-generated Data: Instruction tuning uses outputs from larger proprietary language models and human participants to fine-tune smaller open-source models.Alpaca and Vicuna are cited examples based on the open-source LLaMA family.
  • Model-generated Data: Vision-language systems can expand pretraining data by automatically generating captions and other text for visual content.LLaVA uses 150,000 mainly language-model-generated instruction-following examples from textual and visual inputs.

9 Agent Dataset and Leaderboard

The paper proposes benchmarks, datasets, evaluation systems, and leaderboards for multi-agent gaming and analytical multimodal video understanding.

  • Overview: The proposed research infrastructure includes benchmarks for multi-agent gaming and agentic visual-language tasks, plus the CuisineWorld and VideoAnalytica datasets.
  • CuisineWorld: CuisineWorld is a text-based cooperative game designed to evaluate whether agents understand goals and coordinate effectively in dynamic multi-agent scenarios.
  • CuisineWorld: The CuisineWorld benchmark provides extensible task definitions, multi-agent and human-machine interaction interfaces, and the CoS collaboration-efficiency metric.
  • CuisineWorld: CuisineWorld includes collaboration tasks, an interaction API, and an automatic evaluation system.
  • CuisineWorld: The paper proposes Microsoft MindAgent and associated CuisineWorld data, with benchmarks, baseline results, and ranking for submitted algorithms.
  • Evaluation: CuisineWorld evaluation combines averaged collaboration-efficiency metrics with human judgments of engagement, breadth, and overall interaction quality.
  • Evaluation: The planned leaderboard uses automated test-set evaluation, code submissions, optional reruns for leading contenders, and organizational human evaluation.
  • VideoAnalytica: VideoAnalytica evaluates analytical comprehension of long instructional videos beyond recognition and basic comprehension.It emphasizes integrating audio, video, language, and domain-specific knowledge.

10 Broader Impact Statement

The paper positions multimodal Agent AI as a catalyst for applications and collaboration while emphasizing that priorities, limitations, and hazards vary by deployment domain.

  • Contribution: The forum aims to catalyze innovative research and collaboration around multimodal agents and future human-AI interactions.
  • Applications: It grounds research in applications including gaming, robotics, healthcare, and long-video understanding, with potential effects across industries.
  • Domain-specific Concerns: Agent development priorities differ by environment: healthcare requires attention to hallucination risks, while gaming emphasizes open-ended generation, creativity, and adaptation.
  • Contribution: Expert presentations, posters, and leaderboard winners are intended to summarize current trends, research directions, and concepts in multimodal agents.

11 Ethical Considerations

The paper highlights both the applications and risks of multimodal Agent AI, emphasizing responsible development and mitigation strategies. Key concerns include manipulation, bias, privacy, and the transfer of simulated policies to the real world.

  • Applications: Multimodal Agent AI could support content generation, productivity tools, action prediction, and synthesis of 2D or 3D scenarios.The paper connects these applications to broader efforts to model embodied and empathetic agents in simulated or real environments.
  • Risks and mitigation: Agent AI-generated content could enable manipulation or deception, motivating disclosure that content is AI-generated and user controls for customization.The paper also identifies detection of manipulated content as a possible use of Agent AI.
  • Domain-specific concerns: Healthcare deployment raises risks from biased training data, inaccurate diagnoses for underrepresented groups, and exposure of sensitive patient information.The paper frames privacy and confidentiality as significant concerns for patient data.
  • Risks and mitigation: Policies learned through simulation may behave unpredictably in real-world settings because of distribution shift.The paper recommends robust testing and continual safety monitoring, while noting that the VideoAnalytica corpus is collected from the Internet and may contain bias.
  • Responsible development: The project plans to discuss ethical issues, explore mitigation strategies, and deploy a responsible multimodal AI agent.

12 Diversity Statement

The diversity statement positions the project as a broad effort to examine multimodal and agentic AI across environments, modalities, methods, and communities. It includes both technical directions and efforts to address limitations such as hallucinations and bias.

  • 12 Diversity Statement: The project aims to build a diverse community by examining multimodal and agentic AI from varied perspectives and across domains.It presents diversity of subjects, perspectives, and solutions as part of the project’s scope.
  • 12 Diversity Statement: The project focuses on multimodal systems that interact with humans in both physical and virtual environments.It seeks participation from experts across technical specialties, cultures, countries, and scholarly fields.
  • Technical directions: One technical direction integrates audio, image, text, and sensor inputs to improve agents’ recognition and response capabilities.
  • Technical directions: Other directions include large-scale end-to-end systems, cross-modal grounding, and intuitive human interfaces.These directions target versatile systems, coherent multimodal processing, and meaningful human-agent interaction.
  • Technical directions: The project also explores approaches for addressing hallucinations and biases in LLM and VLM outputs.

B GPT-4V for Bleeding Edge

In Bleeding Edge, GPT-4V is prompted as an agent for a team-based combat game, but its visual and rule understanding is qualitatively less thorough than in Minecraft.

  • Game context: Bleeding Edge is a third-person team-based combat game centered on capturing objectives or collecting more resources than the opposing team.
  • Prompted agent: The paper provides an example input and output from prompting GPT-4V for Bleeding Edge.
  • Observed behavior: Compared with Minecraft, GPT-4V qualitatively shows less thorough understanding of Bleeding Edge’s visual content and game rules.The paper attributes this likely to differences in Minecraft’s representation in training data and the game’s visual complexity.

C GPT-4V for Microsoft Fight Simulator

A GPT-4V-based agent can describe high-level actions in Microsoft Flight Simulator from cockpit and external visual views. Its description covers flight controls, aircraft state, and navigation.

  • Agent behavior: The GPT-4V-based agent provides a high-level action description for the player in Microsoft Flight Simulator.
  • Visual interpretation: The description identifies cockpit and external views of the aircraft while the player is flying.
  • Action description: The agent describes managing flight controls and instruments to maintain airspeed and altitude while navigating virtual airspace.

D GPT-4V for Assassin’s Creed Odyssey

GPT-4V provides a high-level description of a character engaged in nighttime combat during a historical battle, while managing abilities and combat resources.

  • D GPT-4V for Assassin’s Creed Odyssey: The character fights multiple enemies with a spear and special abilities, including Bull Rush, during a nighttime historical battle.The scene is presented from a third-person action role-playing perspective, with enemies marked by level indicators.
  • D GPT-4V for Assassin’s Creed Odyssey: The description includes management of the character’s health and ability meters during combat.

E GPT-4V for GEARS of WAR 4

GPT-4V describes GEARS of WAR 4 gameplay from video frames, identifying combat, tactical movement, resource management, allies, and interactive environmental elements.

  • E GPT-4V for GEARS of WAR 4: The player navigates an industrial environment, uses cover, and engages enemies with firearms or melee attacks.The description also identifies visual cues associated with damage and health management.
  • E GPT-4V for GEARS of WAR 4: The player shifts between cover positions, activates a protective shield, reloads, and appears to land successful hits.
  • E GPT-4V for GEARS of WAR 4: Other characters appear nearby and participate in similar combat actions, indicating cooperation with allies.
  • E GPT-4V for GEARS of WAR 4: Overall, the scenes emphasize fast-paced action, strategic movement, shooting, and situational awareness against adversaries.

F GPT-4V for Starfield

GPT-4V describes Starfield gameplay across sampled frames, covering exploration, NPC interaction, mission guidance, machinery, and resource collection in a mining environment.

  • F GPT-4V for Starfield: The player explores underground caverns and mining interiors while following objectives indicated by waypoints and glowing markers.The sequence includes movement toward a highlighted waypoint and traversal through darker cavern areas.
  • F GPT-4V for Starfield: The player interacts with Supervisor Lin, receives mission updates, and follows the NPC toward a resource-related objective.
  • F GPT-4V for Starfield: The player appears to observe or operate machinery with another character during technical work.
  • F GPT-4V for Starfield: The player examines glowing objects and a HUD-marked Beryllium Deposit, indicating resource gathering or mining activity.
  • F GPT-4V for Starfield: HUD waypoints, dialogue text, and mission updates provide context for the player’s actions and game objectives.
Loading 2401.03568v2…