Agent AI: Surveying the Horizons of Multimodal Interaction

1.1 Motivation

• The authors argue that LLMs and VLMs enable a return to holistic AI by combining language, visual cognition, memory, reasoning, and adaptability in agent systems.They also revisit design around Aristotle’s Final Cause—the reason a system exists.Source: §1.1 · p. 5

1.2 Background

• LLMs’ general problem-solving potential (Bubeck et al., 2023; Mirchandani et al., 2023) and robot- and game-planning work (Liang et al., 2022; Wang et al., 2023a,b; Yao et al., 2023a; Huang et al., 2023a) support their use as general-purpose agents.Embodied systems also decompose language instructions into subtasks and use low-level controllers and environmental feedback (Ahn et al., 2022a; Huang et al., 2022b; Liang et al., 2022; Wang et al., 2023a; Ikeuchi et al., 2023).Source: §1.2 · pp. 5–6
• Interactive-learning agents can adapt through direct user feedback and observational learning, with human oversight intended to prevent biases or incorrect patterns (Li et al., 2023b; Yu et al., 2023a; Parakh et al., 2023; Zha et al., 2023; Wake et al., 2023a,b,c).The passage describes learning from user corrections as well as patterns in users’ interactions.Source: §1.2 · p. 6

1.3 Overview

• Multimodal Agent AI systems generate actions from multimodal sensory input, and this survey focuses on multimodality, gaming, robotics, and healthcare and shared concerns including data, benchmarks, and ethics.It aims to provide a foundation for understanding the field and advancing its research.Source: §1.3 · p. 6
• The survey covers Agent AI training, agent types and applications, sim-to-real transfer, continual learning, datasets, evaluation, ethics, limitations, and societal impact.It also discusses emerging trends and future directions across domains.Source: §1.3 · pp. 6–7

2.1 Infinite AI agent

• The proposed infinite agent transfers memory from general foundation models to novel physical or virtual scenarios for scene understanding, generation, and interactive editing, addressing costly per-task data collection.RoboGen (Wang et al., 2023d) applies this transfer in robotics through autonomous cycles of task proposition, environment generation, and skill learning.Source: §2.1 · pp. 7–8

2.2 Agent AI with Large Foundation Models

• Foundation models support constrained robotic manipulation and navigation, as well as language-conditioned, scene-adaptive motion generation.Examples include using an image-editing model to generate future subgoals for low-level robot policies and combining LLM landmark extraction with VLM visual grounding for navigation instructions (Black et al., 2023; Shah et al., 2023a).Source: §2.2 · p. 8

2.2.1 Hallucinations

• LLM and VLM agents can hallucinate when pretrained co-occurrence priors and limited environment-specific tuning lead them to misrepresent world state; retrieval grounding and contradiction checks are proposed mitigations.Vision-language hallucinations can arise from over-reliance on object and visual-cue co-occurrences in training data.Source: §2.2.1 · p. 9

2.2.2 Biases and Inclusivity

• Agent biases can reflect WEIRD-skewed and historically biased training data, language and context limitations, and dominant cultural viewpoints; biases persist despite mitigation efforts.The section calls for inclusive, respectful interactions and continued efforts to reduce bias and improve fairness.Source: §2.2.2 · pp. 9–10

2.2.3 Data Privacy and Usage

• Responsible agent deployment requires transparent data collection, consent-bound use, secure storage and sharing, retention and deletion controls, portability, correction, and anonymization.Users should understand how their interaction data is used and be able to access or correct it, request deletion, and limit data collection.Source: §2.2.3 · p. 11

2.2.4 Interpretability and Explainability

• An infinite-memory agent learns to imitate expert state-action demonstrations rather than depend on sparse task rewards, using in-context prompts or implicit reward signals to support generalization across tasks.Decoupling policy learning from task-specific reward functions can enable transfer between domains with minimal fine-tuning.Source: §2.2.4 · pp. 11–12

2.2.5 Inference Augmentation

• Inference augmentation combines external data, improved algorithms, human judgment, real-time feedback, cross-domain transfer, and use-case-specific tuning to improve agent inferences.Human input can correct errors in ambiguous or ethically sensitive cases, while environmental feedback can correct actions that violate simulation rules.Source: §2.2.5 · pp. 12–13
• Inference augmentation enhances agents’ natural inference through additional data, improved algorithms, human input, and other techniques to support complex tasks and accurate outputs.The section also calls for continuous refinement as data and user needs change, while considering fairness, transparency, privacy, and harmful stereotypes, especially for sensitive inferences.Source: §2.2.5 · p. 13

2.2.6 Regulation

• The proposed human-machine collaboration system uses LLM/VLM dialogue and action capabilities to identify human needs and perform requested actions, with prompts and human oversight to manage unpredictable outputs.Environmental information in prompts is reported to yield more stable robotic task-planning outputs than text alone; explanatory prompts and pre-execution verification or modification under human guidance are also proposed.Source: §2.2.6 · pp. 13–14
• A robot-teaching workflow combines task planning from instructions and environmental information with user demonstrations, review, and the option to revisit deficient steps.The figure also shows a web application for uploading demonstration data and interacting with ChatGPT.Source: Figure 4 · p. 14

2.3 Agent AI for Emergent Abilities

• The proposed Mixed Reality with Knowledge Inference Interaction uses foundation-model knowledge to support collaboration in complex real-world tasks and adaptation to unseen environments.It combines cross-modality micro-reactions, which gather task-relevant knowledge from explicit web sources and pretrained-model inference, with reality-agnostic macro-behavior that adapts interaction patterns and roles. The authors report improved generalization, consciousness, and interpretability in their setting.Source: §2.3 · pp. 14–15

3 Agent AI Paradigm

• The framework bootstraps agents from pretrained models, supports long-term planning and retrievable memory, and uses environmental feedback to teach action selection.Source: §3 · p. 15

3.1 LLMs and VLMs

• LLMs can bootstrap planning, world knowledge, and reasoning, while VLMs provide language-aligned visual encoding and zero-shot recognition (Gong et al., 2023a; Yu et al., 2023b; Creswell et al., 2022; Radford et al., 2021).LLaVA (Liu et al., 2023c) and InstructBLIP use frozen CLIP models as visual encoders.Source: §3.1 · p. 15

3.2 Agent Transformer Definition

• A unified transformer adds agent tokens to visual and language tokens for domain-specific actions or tools, enabling customization and action-probability inspection, with potential privacy and cost advantages over proprietary LLMs.Source: §3.2 · pp. 15–16, Figure 7 · p. 16

3.3 Agent Transformer Creation

• Creation defines domain objectives, action spaces, and unique action tokens, then gathers success-rule, foundation-model, or human-annotated data and improves performance through monitored feedback, fine-tuning, and bias checks.The process can initialize from LLM/VLM-bootstrapped agents or foundation-model-generated data, tailoring agents to specific tasks and domains.Source: §3.3 · pp. 16–17

4.3 Agentic Foundation Models (pretraining and finetune level)

• Foundation models ground language in visual scenes for navigation and manipulation, with examples including LM-Nav, open-vocabulary detectors, and GPT-4V prompting.LM-Nav combines GPT-3-generated textual landmarks with robot images for navigation (Shah et al., 2023a); manipulation work combines LLMs with detectors such as Detic (Parakh et al., 2023), while prompting GPT-4V supports multimodal task planning (Wake et al., 2023b).Source: §4.1 · p. 17, §4.3 · pp. 19–20

4.1.1 Reinforcement Learning (RL)

• LLMs and VLMs can support RL reward design and data efficiency, while LLM task planning paired with RL control decomposes long-horizon tasks through TAMP.Long action sequences worsen credit assignment and increase the states and data to explore; task-and-motion planning separates high-level task planning from physically consistent motion planning. Examples of LLM task planning with RL controls include Xu et al. (2023), Sun et al. (2023a), Li et al. (2023b), and Parakh et al. (2023).Source: §4.1.1 · pp. 17–18

4.1.2 Imitation Learning (IL)

• Behavioral cloning trains robots to copy expert actions, and RT-1 and RT-2 map image and language inputs to action sequences (Brohan et al., 2022; Brohan et al., 2023).The passage reports that these transformer-based models show high generalization after training on large amounts of data.Source: §4.1.2 · p. 18

4.1.3 Traditional RGB

• RGB-based agent learning addresses visual input’s dimensionality through more data or architectural inductive biases such as 3D structures and maps.Approaches include synthetic data from graphics simulators, efforts to close the sim-to-real gap, large-scale datasets, and data augmentation.Source: §4.1.3 · p. 18

4.1.4 In-context Learning

• Few-shot in-context examples can improve foundation-model outputs, and environment-specific feedback after actions can further adapt in-context learning for agents.The passage describes few-shot visual understanding with Flamingo and BLIP-2 (Alayrac et al., 2022; Li et al., 2023c).Source: §4.1.4 · p. 18

4.1.5 Optimization in the Agent System

• Agent-system optimization spans spatial coordination and resource allocation as well as temporal scheduling and efficient task sequencing.Self-play can improve agents when task-specific multi-agent datasets are rare, but may produce brittle policies; diverse conventions and foundation models can support coordination with humans and independent agents.Source: §4.1.5 · pp. 18–19

4.2.1 Agent Modules

• MindAgent organizes interactive multimodal agents into five modules for perception and task planning, learning, memory, action prediction, and cognition.The initial modules support training or in-context learning and use prompt-based memory to inform planning and future actions.Source: §4.2.1 · p. 19

4.2.2 Agent Infrastructure

• Agent infrastructure could reduce the cost of interactive content creation, while analyzing user behavior and offering support when users cannot complete tasks.The passage notes that current multimodal interactions are largely rule-based and that implementing specific behaviors can be costly.Source: §4.2.2 · p. 19

5.1 Generalist Agent Areas

• Agent AI spans multimodal and general agents, with embodied, action, language, vision-language, knowledge, gaming, robotics, navigation, simulation, and XR applications.The section also identifies healthcare, rearrangement, agentic foundation models, and embodied vision-language as areas of interest.Source: §5.1 · p. 20

5.2.1 Action Agents

• Action agents act in simulated or real environments, spanning gaming AI and robotics; Diplomacy training with conversation data and an RL policy achieved human-level play (Meta Fundamental AI Research (FAIR) Diplomacy Team et al., 2022).In town simulations, foundation models and external memory can support natural interactions, schedules, relationships, and virtual lives (Park et al., 2023a).Source: §5.2.1 · pp. 20–21

5.2.2 Interactive Agents

• Interactive agents can share information or modify environments without requiring physical action, enabling dialogue, diagnostic, and knowledge-retrieval (Peng et al., 2023) applications.Examples include answering questions and helping users navigate information, like a chatbot.Source: §5.2.2 · p. 21

5.3 Simulation and Environments Agents

• Simulators enable policy learning when physical trial-and-error is costly, time-consuming, or dangerous, with realistic rendering and physics engines supporting transfer-relevant skills.Platforms include Habitat, VirtualHome, and Overcooked-AI; some incorporate human intervention or demonstrations for policy learning.Source: §5.3 · p. 21

5.4 Generative Agents

• Generative agents can add user-defined interaction rules, create level geometry from sketches, retexture scenes, and generate shaders or visual effects from prompts.A proposed near-term use is a VR storyboard or prototype tool for creating a functional experience or game much faster than current workflows.Source: §5.4 · pp. 21–22

5.4.1 AR/VR/mixed-reality Agents

• XR agents aim to connect large AI models with rendering engines; GPT models have used Unity methods and APIs to place 3D assets and assign behaviors from prompts (Huang et al., 2023a).XR copilots could assist creators with assets and boilerplate, while runtime code generation could let users modify environments, dynamics, and agent mechanics.Source: §5.4.1 · p. 22

5.5 Knowledge and Logical Inference Agents

• Knowledge agents combine static implicit model knowledge with continuously updated explicit sources, while logic and neuro-symbolic approaches target structured reasoning and symbolic representations.GPT-style models can perform logical tasks but do not use explicit formal rules; TP-N2F encodes natural-language structure with tensor-product binding and generates relational-tuple programs.Source: §5.5.1 · p. 23, §5.5.2 · p. 23, §5.5.4 · p. 24

5.5.3 Agents for Emotional Reasoning

• The NICE dataset and MAGIC model support empathetic image commenting, and experiments found the approach effective for more human-like, engaging agents.The section also flags bias in current language models’ emotional understanding and empathetic reasoning (Mao et al., 2022; Wake et al., 2023d).Source: §5.5.3 · p. 23

6.1.1 NPC Behavior; 6.1.2 Human-NPC Interaction

• LLMs and VLMs could make NPCs less scripted by adapting their behavior to players and enabling more human-like, organic interactions.The passages describe conventional NPC reactions as preset and often repetitive, and identify player-behavior analysis as a way to make interactions more nuanced and engaging.Source: §6.1.1 · pp. 24–25, §6.1.2 · p. 25

6.1.3 Agent-based Analysis of Gaming

• For game analysis, GPT-4V can use action history and goals to predict high-level actions and perceived inventory, but raw low-level control may require supplemental modules.LLMs and VLMs can also analyze game text and video, while player interactions, feedback, pixels, and natural-language planning can support agents that imitate human players.Source: Figure 9 · p. 26, §6.1.3 · p. 27

6.1.4 Scene Synthesis for Gaming

• LLMs and VLMs can support game-scene synthesis by generating terrain and assets, guiding contextually appropriate object placement, and informing realistic or responsive lighting and weather.The passage describes using internet-scale knowledge for varied landscapes and adapting atmospheric effects to player actions, game states, or external inputs.Source: §6.1.4 · p. 27

6.1.5 Experiments and Results

• Gaming experiments demonstrate GPT-4V action-description generation, a small model’s predictions on unseen Minecraft scenes, and a multi-agent collaboration infrastructure across game domains.The infrastructure uses GPT-4 as a central planner and introduces CuisineWorld with a Collaboration Score (CoS) to quantify collaboration efficiency.Source: §6.1.5 · p. 28, Figure 11 · p. 28

6.2 Robotics

• Robotic visual-motor control uses visual feedback to adjust actions and verify outcomes, while language-conditioned tasks and physical skills require grounding instructions and learning nuanced cues.The text notes that contact points and arm posture are difficult to express in language, motivating enhanced datasets and learning from human demonstrations.Source: §6.2 · pp. 28–30

6.2.1 LLM/VLM Agent for Robotics.

• Robotics research spans multimodal end-to-end systems, task planning and skill training, on-site optimization, conversation agents, and language- or object-guided navigation.These approaches use linguistic and visual inputs, adapt execution to environmental conditions, and include navigation in unfamiliar environments.Source: §6.2.1 · pp. 30–31

6.2.2 Experiments and Results.

• A robotics pipeline uses few-shot ChatGPT planning followed by visual demonstrations that supply affordances, which are converted into reusable skills for varied real-world conditions.Demonstrations provide information such as waypoints, grasp types, and arm postures rather than exact motion paths; the resulting skills are acquired through reinforcement learning.Source: §6.2.2 · pp. 31–32
• In VirtualHome, the task planner produced reasonable plans and adjusted them after execution errors; multimodal feedback also supports more robust recognition and scene-aware planning.GPT-4V analyzes demonstrations and text, selects objects relevant to the demonstrated action, and generates task sequences. Users can correct its video-recognition results.Source: §6.2.2 · pp. 32–33, Figure 14 · p. 32
• RCM outperforms previous VLN methods by 10% on SPL, and SIL reduces the success-rate gap between seen and unseen environments from 30.7% to 11.7%.RCM combines reinforcement learning with imitation learning and cross-modal grounding; SIL explores unseen environments by imitating its own good past decisions.Source: §6.2.2 · pp. 34–35

6.3 Healthcare

• Healthcare agents could support diagnosis, patient care, therapy, and remote monitoring, but misuse or hallucinations could cause serious patient harm.Medical knowledge retrieval paired with diagnostic agents is proposed as a way to reduce hallucinations and improve response quality and precision.Source: §6.3 · pp. 35–36

6.3.1 Current Healthcare Capabilities

• GPT-4V demonstrates medical-image knowledge but does not always answer prescriptive or diagnostic queries, and its video performance is examined on bedside activities and ultrasounds.The examples include safety-related refusals to diagnose some medical images and provide detailed analysis of an echocardiogram video.Source: §6.3.1 · p. 36, Figure 19 · p. 37, Figure 20 · p. 38

6.4.1 Image-Language Understanding and Generation

• Image-language agents must connect image content to language across captioning, referring expressions, and visual question answering.The section describes this as interpreting visual content and generating associated linguistic descriptions.Source: §6.4.1 · p. 36

6.4.2 Video and Language Understanding and Generation

• Video-language generation requires coherent descriptions across frames, but current video foundation models struggle with video-only tasks and fine-grained reasoning due to limited, noisy video-text data.The section identifies encoding frame sequences and generating topically coherent multi-sentence text as a goal for agent-aware synthesis.Source: §6.4.2 · pp. 37, 39

6.4.3 Experiments and Results

• KAT outperforms other methods by 6% on the 2022 OK-VQA open-domain multimodal task, while VLC shows effective visual representations and image-patch-to-text matching.KAT combines implicit GPT-3 knowledge with explicit website knowledge for concurrent reasoning; VLC is pretrained on image-caption pairs and uses a simple linear projection for image embeddings.Source: §6.4.3 · pp. 39, 41

6.5 Video-language Experiments

• With audio transcriptions, GPT-4V produced more informative video summaries and captured detailed actions that frame-caption-only summaries missed.Caption-only summaries perpetuated fabricated events; adding audio helped the agent depict physical actions accurately.Source: §6.5 · pp. 43–44
• For video-language experiments, descriptions should align audio-grounded visual details with timecodes, remain plausible, and omit contextually unsupported events.The descriptions should be delivered confidently, while the final output is a JSON list of video segments with start and end times and a single-sentence description.Source: §6.5 · p. 44
• The video-to-text retrieval task uses four caption-like queries: one video-aligned rephrasing and three compelling, subtly mismatched negatives that require visual understanding to reject.The task supplies the original query, video description, audio transcription, and potentially noisy time-specific descriptions; queries include justifications for why the negatives are incorrect.Source: §6.5 · pp. 44–45

6.6.1 LLM agent

• Language-grounded agents can be improved through tool use and knowledge-base querying, stronger reasoning and planning, and adaptation to system or human feedback.The section gives Toolformer, Retrieve What You Need (Wang et al., 2023g), ReAct (Yao et al., 2023a), search-structured reasoning (Yao et al., 2023b), and AutoGen as examples.Source: §6.6.1 · p. 45

6.6.2 General LLM agent

• Planning and human feedback are identified as key components of human-machine agents, including AutoGen and Retrieve What You Need (Wang et al., 2023g).The section frames their joint study as supporting deeper understanding of agent planning and feedback for knowledge inference and natural-language generation.Source: §6.6.2 · pp. 45–46

6.6.3 Instruction-following LLM agents

• Instruction-following LLM agents have been trained with RLHF, while more efficient instruction tuning uses instruction-response pairs generated by humans or LLMs such as Alpaca.RLHF trains a proxy reward model to simulate human preferences; the section presents the Alpaca training pipeline in Figure 28.Source: §6.6.3 · p. 46, Figure 28 · p. 46

6.6.4 Experiments and Results

• Dialogue agents add explicit knowledge search and retrieval for more interpretable responses and deeper human-agent social interaction; logical-transformer agents detect and integrate text logic through hierarchical projections, consistently outperforming baseline transformers (Wang et al., 2023e).Source: §6.6.4 · p. 47
• Iterative training of a knowledge-selector agent and reader improves predictive performance on some public open-domain QA benchmarks.The selector uses reader feedback and policy gradients to choose a small, informative passage set without annotated query-document pairs or brute-force search.Source: §6.6.4 · pp. 47–48

7.1 Agents for Cross-modal Understanding

• RT-2 improved performance on robotics and vision-language tasks by jointly tuning the visual encoder and LLM, challenging the frozen-submodule strategy used by systems such as Flamingo, BLIP-2, and LLaVA.The paper argues that progress in cross-modal agent understanding will likely require moving beyond separately trained frozen encoders and LLMs.Source: §7.1 · p. 48

7.2 Agents for Cross-domain Understanding

• Separate domain-specific fine-tuning of pretrained models fails to capture cross-domain commonalities and limits training data, while varied appearances and action spaces make a shared policy difficult.Existing LLMs and VLMs often face a mismatch between training data and deployment domains, including robotics and healthcare.Source: §7.2 · p. 48

7.3 Interactive agent for cross-modality and cross-reality

• Agents struggle to understand real and virtual environments together because their visual appearances and environment physics differ, making sim-to-real transfer an important cross-reality challenge.The section notes recent success in image and scene generation (Huang et al., 2023a).Source: §7.3 · p. 49

7.4 Sim to Real Transfer

• Sim-to-real transfer is difficult because simulations do not fully reproduce real-world disturbances, lighting, gravity, and other physical properties, and remains a central challenge for embodied agents.The section describes domain randomization, which varies simulated parameters such as object appearance and sensor noise to prepare policies for real-world variation.Source: §7.4 · p. 49
• Other sim-to-real approaches bridge domains with simulated and limited real-world images, or improve simulation through system identification and photorealistic simulators.Unpaired image translation methods include CycleGAN (Zhu et al., 2017b), with RL-CycleGAN and RetinaGAN adaptations; simulation improvement includes system identification (Zhu et al., 2017c; Allevato et al., 2020) and photorealistic simulators (Sasabuchi et al., 2023).Source: §7.4 · p. 49

8 Continuous and Self-improvement for Agent AI

• Foundation-model agents generally do not learn continuously from their environments, though Bousmalis et al. (2023) demonstrated unsupervised robotic agents that continuously improve through environmental interaction.The section identifies continuous environmental learning as an exciting future direction.Source: §8 · p. 49

8.1 Human-based Interaction Data

• Human-agent interactions can improve future agents through filtered training examples, user preference selections, and adversarial red-teaming that exposes vulnerabilities for corrective safety training.Examples can be filtered by rules, models, or manual review; red-teaming seeks consistent ways to induce unwanted outputs so training can correct them.Source: §8.1 · pp. 49–50

8.2 Foundation Model Generated Data

• LLM instruction-tuning distills larger models into smaller open-source students (Wang et al., 2022b), transferring style and some instruction-following while leaving factuality and capability gaps (Gudibande et al., 2023).Alpaca and Vicuna are LLaMA-family models tuned on outputs from ChatGPT and human participants (Touvron et al., 2023).Source: §8.2 · p. 50
• LLaVA used 150,000 mainly LLM-generated instruction examples from text and images (Liu et al., 2023c), while image and video recaptioning improved downstream image-generation or VLM quality (Segalis et al., 2023; Wang et al., 2023f; Zhao et al., 2022).Source: §8.2 · p. 50

9 Agent Dataset and Leaderboard

• The authors propose two benchmarks, datasets, and baseline models, with leaderboards for multi-agent gaming and agentic visual-language tasks.CuisineWorld and VideoAnalytica are intended to encourage submissions on test sets.Source: §9 · p. 50

9.1 “CuisineWorld” Dataset for Multi-agent Gaming

• CuisineWorld is a text-based cooperative game for evaluating agents’ goal understanding and coordination in centralized-dispatcher or decentralized modes.Participants can choose a play mode and submit to its leaderboard.Source: §9.1 · p. 50

9.1.1 Benchmark

• CuisineWorld provides collaboration tasks, agent-interaction and human-machine interfaces, extendable task definitions, and automatic evaluation for strategies intended to maximize collaboration efficiency.The competition aims to compare grounded-LLM and interactive techniques and establish baselines; its CoS metric evaluates collaboration efficiency.Source: §9.1.1 · p. 51

9.1.3 Metrics and Judging

• CuisineWorld’s CoS rating averages collaboration-efficiency scores across tasks, while human evaluators judge responses, engagement, breadth, and overall interaction quality.The metric is identified as coming from MindAgent (Gong et al., 2023a).Source: §9.1.3 · p. 51

9.1.4 Evaluation

• CuisineWorld leaderboard submissions undergo automated test-set CoS evaluation and human evaluation, with code checks for top contenders and final winner ratings announced.Teams submit code and a Readme; the organizers conduct human evaluation, and EvalAI checks submission files.Source: §9.1.4 · p. 51

9.2 Audio-Video-Language Pre-training Dataset.

• VideoAnalytica evaluates multimodal, domain-aware reasoning over long instructional videos through video-text retrieval and video-assisted informative question answering.Retrieval includes human-validated hard negatives generated by large language models; question answering requires analytical reasoning and thorough comprehension. Its leaderboard reports performance on both tasks.Source: §9.2 · pp. 51–52
• The VideoAnalytica leaderboard raises ethical concerns because video understanding could enable surveillance or other privacy-invasive uses.The authors encourage participants to consider potential misuse and promote ethical AI use.Source: §9.2 · p. 52

10 Broader Impact Statement

• The forum encourages real-world-driven multimodal-agent research in gaming, robotics, healthcare, and long-video understanding, with potential effects on industries and people worldwide.Source: §10 · p. 52
• It helps practitioners tailor development to domain-specific challenges, including dangerous healthcare hallucinations and gaming's need for creative, open-ended behavior amid unpredictable interactions.Source: §10 · p. 52
• Expert presentations, posters, and the event's two leader-board winners offer attendees and website visitors a concise resource on trends and research directions in multimodal agents.Source: §10 · p. 53

• Grounded multimodal systems could support content generation, productivity, and scenario replay, paraphrasing, action prediction, or synthesis, with many applications potentially beneficial.The authors plan to discuss ethical issues and mitigation strategies and deploy a responsible multimodal AI agent, inviting researchers to address these questions together.Source: §11 · p. 53
• Generated content could manipulate or deceive users, so systems should disclose AI generation and provide user controls.Agent AI may also help develop methods for detecting manipulative content.Source: §11 · p. 53
• In healthcare, biased training data could worsen disparities through inaccurate diagnoses for underrepresented groups, while patient-data handling raises privacy and confidentiality concerns.Gaming and adaptive robotics may shift developer and manufacturing roles toward refining agent learning and acquiring new skills, creating potential socioeconomic disruption.Source: §11 · p. 53
• Applying collaboration policies learned in simulation to the real world risks unpredictable behavior under distribution shift, requiring robust testing and continual safety monitoring.The Internet-collected VideoAnalytica dataset is not fully representative; despite ethical and legal review, its biases remain a concern, and its captured agent-level distributions do not encompass everything relevant to real tasks.Source: §11 · p. 53

12 Diversity Statement

• The project seeks to build a diverse community and broaden collective understanding of agentic AI’s potential and limitations through varied perspectives.It aims to enrich individual perspectives, collective knowledge, and a holistic view of challenges faced by multimodal AI agents.Source: §12 · pp. 53–54
• It intends to engage experts and practitioners across technical specialities, cultures, countries, and scholarly fields to discuss multimodal systems for physical and virtual environments.The systems are intended to interact effectively with humans.Source: §12 · p. 54
• Discussion topics include agents integrating audio, image, text, and sensor inputs, general-purpose end-to-end models trained on large-scale data, and methods for grounding modalities.These efforts aim to improve recognition and responses, adaptability, and coherent multimodal processing.Source: §12 · p. 54
• The project also considers meaningful human-agent interaction and approaches to address hallucinations and biases in large-scale models.Source: §12 · p. 54

A GPT-4V Agent Prompt Details

• For Minecraft, the appendix uses GPT-4V’s default system prompt unless specified and details the prompting and response-generation process in Fig. 31.Source: A GPT-4V Agent Prompt Details · p. 69

B GPT-4V for Bleeding Edge

• Bleeding Edge is a third-person team combat game focused on capturing objective points or collecting more resources than the opposing team.Source: B GPT-4V for Bleeding Edge · p. 69
• Compared with Minecraft, GPT-4V shows less thorough understanding of Bleeding Edge’s visual content and game rules.Source: B GPT-4V for Bleeding Edge · p. 69
• The authors suggest that more Minecraft data in GPT-4V’s training data may help explain the difference.Source: B GPT-4V for Bleeding Edge · p. 69
• They also suggest that Bleeding Edge’s greater visual complexity than Minecraft may contribute to GPT-4V’s weaker understanding.Source: B GPT-4V for Bleeding Edge · p. 69

C GPT-4V for Microsoft Fight Simulator

• The agent describes the player managing flight controls to maintain airspeed and altitude while navigating virtual airspace.The description is grounded in cockpit and external views of the aircraft (Fig. 33).Source: C GPT-4V for Microsoft Fight Simulator · p. 69

D GPT-4V for Assassin’s Creed Odyssey

• GPT-4V identifies nighttime combat in a historical battle, the spear-wielding character’s Bull Rush against multiple level-marked enemies, and health and ability meters in a tutorial sequence.The scene is described as a dark, fiery third-person action role-playing setting, with a Bull Rush prompt on screen.Source: D GPT-4V for Assassin’s Creed Odyssey · p. 69

E GPT-4V for GEARS of WAR 4

• The description also identifies tactical shifts between cover, defensive shield use, reloading, and impacts indicating successful hits.These details suggest the player responds to incoming fire and manages combat resources.Source: E GPT-4V for GEARS of WAR 4 · p. 69, Figure 35 · p. 74
• It notes nearby allied characters and interactive exploding barrels as elements that may affect the player’s combat choices.Source: E GPT-4V for GEARS of WAR 4 · p. 69
• GPT-4V describes the player navigating a futuristic industrial combat environment, using cover and firearms against enemies amid damage cues and close-quarters fighting.The figure caption says the description is based on 10 selected key frames.Source: E GPT-4V for GEARS of WAR 4 · p. 69, Figure 35 · p. 74
• The scenes are characterized as fast-paced, requiring strategic movement, shooting, and situational awareness to overcome adversaries.Source: E GPT-4V for GEARS of WAR 4 · p. 75

F GPT-4V for Starfield

• GPT-4V describes the Starfield sequence as a mining mission involving exploration, NPC interaction, resource collection, and possible machinery operation, contextualized by HUD cues.The account mentions waypoints, dialogue, and mission updates as context for the player’s actions and objectives.Source: F GPT-4V for Starfield · p. 75, Figure 36 · p. 76
• Figure 36’s setup sends ten selected video key frames individually to GPT-4V, without arranging them in a grid, to generate a high-level action description.The example uses footage from the Xbox console game Starfield.Source: F GPT-4V for Starfield · p. 75, Figure 36 · p. 76

Author Biographies

• Rohan Taori’s research spans machine-learning foundations in real-world systems, open-source language models, and multimodal reasoning; he received a computer science bachelor’s degree from UC Berkeley.He is a PhD student at the Stanford AI Lab.Source: Author Biographies · p. 78
• A biography notes more than 16 years of experience developing efficient infrastructure for Xbox One and Xbox Cloud Gaming.Source: Author Biographies · p. 78
• A researcher at the Allen Institute is described as working on commonsense reasoning in NLP, machine learning, and AI, with broader interests in computer vision and digital humanities.The biography lists the Anita Borg Early Career Award, a MacArthur Fellowship, and multiple paper awards.Source: Author Biographies · p. 78
• A researcher joined Microsoft as a principal researcher in 2015 after 19 years at the University of Tokyo’s Institute of Industrial Science and supervised more than 50 PhD students.Source: Author Biographies · p. 79
• His research interests include computer vision, robotics, and computer graphics, and his awards include the David Marr Prize and IEEE Robotics and Automation Society K. S. Fu memorial best transaction paper.Source: Author Biographies · p. 79
• His service includes leadership and editorial roles across major conferences and journals; he was also named a fellow of IEEE, IEICE, IPSJ, and RSJ.Other honors listed include IEEE-PAMI’s Distinguished Researcher Award, Japan’s Medal of Honor with Purple Ribbon, and the Okawa prize.Source: Author Biographies · p. 79