Source-linked AI summary
A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations): A Visual-Symbolic Framework for Virtual Humans
Alessandro Emmanuel Pecora, Stefano Calzolari, Francesco Strada, Andrea Bottino
TL;DR
Virtual humans need coherent integration of multimodal perception, reasoning, and action, yet grounding these capabilities in interactive 3D settings remains challenging. A.D.A.M.O. combines egocentric vision, symbolic state, and tool use in a language-driven control loop, and evaluates behavior with a task-difficulty taxonomy. In controlled scenes, semantic labeling reduces perceptual ambiguity and shifts failures toward execution, while reasoning errors remain comparatively rare.
Problem
Integrating visual observations, symbolic reasoning, and embodied control in dynamic 3D settings remains challenging for language-driven virtual humans.
Source: §1 · p. 2Method
A.D.A.M.O. uses a pretrained VLM with tool calling to combine egocentric visual input and synchronized symbolic state in a language-driven control loop.
Source: §1 · p. 2Results
Semantic labeling reduces perceptual ambiguity and shifts failures toward downstream execution effects, while reasoning-related errors remain comparatively rare.
Source: §6 · p. 10Takeaways & Limitations
The Capability-Difficulty evaluation offers a diagnostic setup for identifying bottlenecks across perception, memory, and action execution in virtual humans.
Source: §1 · p. 2, §6 · p. 10Evaluation is limited to controlled simulated environments, and the absence of persistent spatial or episodic memory makes long-horizon tasks more challenging.
Source: §5.4 · p. 10
Abstract
from arXiv · showhide
Creating believable vh requires the coherent integration of perception, reasoning, and action mediated by language. A central challenge is to combine these components into a control loop grounded in interactive 3D environments. To this end, we present A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations), a visual-symbolic framework for language-driven vh that leverages a pretrained vlm with tool calling to unify perception, reasoning, and action within a single control loop. A.D.A.M.O. maintains a dual visual-symbolic world model that combines egocentric visual input and synchronized symbolic state to support grounded task-oriented behavior from natural language prompts. To support diagnostic evaluation, we introduce a controlled task suite organized by a cd taxonomy that breaks down spatial tasks into procedural and linguistic complexity. Experiments in controlled scenes show that semantic labeling strongly influences task completion and failure modes, reducing perceptual ambiguity while shifting failures toward downstream execution, whereas reasoning errors remain comparatively rare.
1 Introduction
Virtual humans need to connect multimodal perception, language, reasoning, and goal-directed action, but these capabilities remain difficult to integrate in 3D settings. A.D.A.M.O. combines visual and symbolic inputs in a language-driven control loop and evaluates behavior with a task-difficulty taxonomy.
1 Introduction
- Existing virtual-human systems often depend on manual scripting, while integrating visual observations, symbolic reasoning, and embodied control in dynamic 3D settings remains challenging.Language-driven ECAs and symbolic-state agents each lack some of the grounding needed for situated interaction.Source: §1 · p. 2
- A.D.A.M.O. unifies language-based decision making, egocentric VLM perception, synchronized symbolic state, and tool-based action in one control loop.The visual and symbolic streams are cross-referenced to support grounded multimodal reasoning.Source: §1 · p. 2
- The Capability-Difficulty taxonomy organizes spatial tasks by procedural and linguistic complexity, while experiments find semantic labeling reduces ambiguity and shifts failures toward execution.Reasoning-related errors remain comparatively rare.Source: §1 · p. 2
2 Related Work
Prior virtual-agent work often separates linguistic interaction from embodied behavior, and existing evaluation can obscure task difficulty and error sources. A.D.A.M.O. combines egocentric vision with structured symbolic memory and introduces a taxonomy for diagnosing embodied task difficulty.
2 Related Work
- Research diverged into ECAs prioritizing coordinated verbal and nonverbal human-like interaction yet decoupling communicative intent from physical realization, and GCAs prioritizing embodied behavior, often via symbolic world states with limited perceptual grounding [16] [17].Source: §2 · p. 3
- Earlier spatial-awareness systems used handcrafted rules or exhaustive scene annotation, limiting adaptability and scalability while offering little visual disambiguation.These omniscient setups cannot exploit visual cues for disambiguation.Source: §2 · p. 3
- A.D.A.M.O. uses egocentric vision for perceptual verification and structured symbolic memory for explicit spatial reasoning within one language-based control layer.Its Capability-Difficulty taxonomy separates procedural capability from linguistic complexity to help explain where agents struggle.Source: §2 · p. 3
3 Methods
A.D.A.M.O. is designed for language-driven control in controlled 3D environments, with a logical architecture for perception and action and an infrastructure layer that separates reasoning from simulation execution.
3.1 Logical Architecture
- A.D.A.M.O. adapts the perceive–reason–act paradigm and ReAct to a multimodal virtual-human setting, using a closed Thought–Act–Observation loop under partial observability.The VLM produces a reasoning note and tool choice; primitive tools include Look, Walk, Pick, and Drop.Source: §3.1 · p. 4
- Each observation pairs an egocentric RGB view with object tags, bounding boxes, sparse spatial anchors, and synchronized symbolic state containing object and agent information.A rolling multimodal short-term memory retains recent visual inputs, symbolic state, reasoning, tool calls, and execution feedback.Source: §3.1 · p. 4
3.2 Infrastructure Architecture
- The infrastructure separates high-level reasoning from simulation and action execution to keep cognitive decisions independent of real-time environment dynamics.The Cognitive Server manages prompts, memory, and tool invocations, while the Unity Runtime executes actions and returns updated observations.Source: §3.2 · pp. 4–5
- At runtime, the VLM selects a tool or terminates; tool execution updates the environment and returns observations and feedback to memory for the next loop iteration.The cycle continues until task completion, after which the Cognitive Server signals termination.Source: §3.2 · p. 5
4 Experiments
The experiments evaluate spatial reasoning and manipulation in controlled scenes using a task suite, a difficulty taxonomy, and repeated runs that measure completion and model or control activity.
4 Experiments
- The benchmark contains 15 object-centric manipulation episodes across a simple tabletop scene and a more complex living-room layout with the same 19 interactable objects.Prompts vary pick/place actions and spatial constraints such as near and on; the checker evaluates task completion.Source: §4 · p. 5, Figure 3 · p. 6
- The study compares GPT-4o-vision and Claude-Sonnet-3.5 with semantic or opaque object labels to examine grounding and action efficiency.Semantic tags encode class and instance information, whereas opaque tags use numeric identifiers only.Source: §4 · pp. 5–6
4.1 Capability-Difficulty (C-D) Taxonomy
- The agent-agnostic Capability-Difficulty taxonomy sums costs for actions, target references, and final spatial relations to order task difficulty.Locomotion and perceptual actions are implicit rather than explicitly scored; the example task has ACT=2, REF=3, REL=2, and C-D level 7.Source: §4.1 · p. 6, Table 1 · p. 6
4 Experiments
- Each episode is run 15 times per model and labeling scheme, with completion, tool-call and generation-time, and action-count measures reported.Runs use a 10-minute cap; the checker returns a normalized score from 0 to 1 based on satisfied atomic conditions.Source: §4 · p. 5, §4.2 · p. 6
5 Results
Across the evaluated tasks, semantic labeling substantially improved completion and shifted failures away from perception, while reasoning errors remained rare. Performance also varied with scene and task complexity, and ablations showed trade-offs among context, sampling, spatial anchoring, and image resolution.
5.1 Model and Labeling Scheme
- Semantic labeling raised completion from 0.48→0.88 (+83%) for G4O and 0.38→0.83 (+118%) for S35; perceptual failures fell while execution-related failures rose.Reasoning-related errors remained rare (2–3%). G4O averaged 0.68 completion versus 0.61 for S35, with lower per-call latency but more actions per episode.Source: §5.1 · p. 7
5.2 Tasks and Scenarios
- Completion was higher in S1 than S2 for both models: G4O scored 0.93 versus 0.83, and S35 scored 0.90 versus 0.76.S2’s more dispersed, cluttered layout increased exploration; G4O’s broader exploration partly compensated for reduced visibility at the cost of additional actions.Source: §5.2 · p. 8
- Basic single-object tasks were near-perfect, and comparative, relational, and moderate multi-object tasks remained highly reliable despite labels not explicitly encoding some task relations.T3’s color–shape discrimination was a notable exception among basic tasks; the text attributes reliable relational performance to complementary visual and symbolic representations.Source: §5.2 · p. 8
- For SEM tasks, C-D level negatively correlated with completion (Spearman ρ = −0.53, p < 0.05); attribute-heavy and long-horizon tasks had the lowest completion, especially in cluttered scenes.T14 exposed conflicts between pretrained category–attribute priors and visual evidence, while T15 combined fine-grained attribute recognition with extended action sequences.Source: §5.2 · pp. 8–9
5.3 Ablation Studies
- Completion rose from 0.82 at k=4 to 0.86 at k=10 and peaked at 0.93 for k=50; quasi-deterministic decoding achieved 0.93, versus 0.82 with moderate stochasticity and 0.85 with high stochasticity.Smaller buffers prompted more exploration and model calls; larger buffers preserved temporal coherence at higher latency. Overall, stochasticity degraded tool-execution reliability despite partial recovery at high stochasticity.Source: §5.3 · p. 9, Table 4 · p. 9, Table 5 · p. 9
- Increasing 3D reference-point density from 3×3 to 6×6 raised completion to 0.93, while further densification to 7×7 lowered it to 0.87; 1920×1080 input raised completion to 0.93 versus 0.76 at 640×360.At 6×6, model calls were 4.67 and tokens were 32,977; per-call time rose from 2.65 s at 640×360 to 4.71 s at 1920×1080.Source: §5.3 · p. 9, Table 6 · p. 9, Table 7 · p. 9
5.4 Limitations and Future Work
- Evaluation is limited to controlled simulated environments and relies on semantic scaffolding and VLMs with native tool-calling support; raw visual grounding alone is not established as sufficient.The implementation lacks persistent spatial or episodic memory and does not explicitly model perceptual noise, exogenous dynamics, or uncertainty-aware action selection.Source: §5.4 · p. 10
6 Conclusion
A.D.A.M.O. combines egocentric visual input, structured symbolic state, and tool-based actions in a unified framework for language-driven virtual humans. In controlled evaluation, semantic labeling reduced perceptual ambiguity, while task difficulty and fine-grained visual attributes remained bottlenecks.
6 Conclusion
- A.D.A.M.O. integrates multimodal perception, reasoning, and embodied control in a unified decision loop for language-driven virtual humans.Source: §6 · p. 10
- Combining egocentric visual input, structured symbolic state, and tool-based actions supports interpretable language-mediated behavior in controlled 3D environments.Source: §6 · p. 10
- Semantic labeling reduced perceptual ambiguity and shifted failures toward downstream execution, while reasoning-related errors remained comparatively rare.Task difficulty and fine-grained visual attributes were the main bottlenecks under increased complexity.Source: §6 · p. 10