PaperlayerView exact revision on arXiv

Source-linked AI summary

TARSphere: A Robot-Oriented Full-Stack Framework for Long-Horizon Embodied Agents in Evolving Physical and Social Environments

Yicheng Xu, Zhikai Xu, Ziyuan Chen, Yuhang Dong, Tao Zhang, Zhucun Xue, Yong Liu, Jiangning Zhang

arXiv:2610.30043v1cs.RO

TL;DR

Existing embodied-agent evaluations often focus on specified goals or abstract execution, despite robots needing to adapt to physical and social changes. TARSphere combines simulated robot execution, human interaction, a feedback-driven MLLM harness, and a paired benchmark; across 8 MLLMs, general-purpose models can outperform embodied models, while execution safety remains a concern.

  • Problem

    Many embodied-agent systems focus on specified goals or abstract execution into symbolic actions, limiting evaluation of long-horizon behavior in evolving physical and social environments.

    Source: §1 · p. 2

  • Method

    TARSphere combines simulated robot execution with embodied human interaction, an event-driven task-state evaluator, and a frozen-MLLM harness using observations, memory, executable skills, and feedback.

    Source: §1 · p. 2

  • Results

    Across 8 MLLMs, general-purpose models can outperform embodied models, with capability gaps varying by task family.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TARSphere supports trajectory-based comparisons of model decisions, task completion, and execution quality as demands change during execution.

    Source: §5 · p. 11

    The evaluation covers one robot body and two indoor service-task families in simulation; real-robot transfer must address differences in sensing, contact, and human behavior.

    Source: §5 · p. 11

Abstract

from arXiv · show

Robots must act under physical and social constraints while adapting to changes in their environment and human needs. Existing embodied-agent systems often focus on specified goals or abstract execution into symbolic actions, limiting evaluation in an evolving world. We introduce TARSphere, a robot-oriented full-stack framework for physically and socially grounded long-horizon embodied agents. TARSphere-Infra runs robot policies in a physics simulator with embodied NPCs and constrained dialogue. An event-driven finite-state machine (FSM) uses completed actions and delivered information to update task states and evaluate outcomes. TARSphere-Harness, its embodied brain harness, lets a frozen multimodal large language model (MLLM) act through multi-level executable skills, using observations, hierarchical memory, and execution feedback to revise its decisions. On this foundation, agent-in-the-loop curation aligns task semantics, causal dependencies, and scene geometry to construct TARSphere-Bench: 360 episodes across 30 indoor scenes. Five paired conditions assess perceptual robustness, disruption adaptation, task update, human coordination, and intent understanding for navigate-and-report and fetch-and-deliver tasks. Our evaluation of 8 MLLMs shows that general-purpose models can outperform embodied models, while task updates and human interaction remain difficult. Frequent collisions and falls further motivate evaluating execution safety alongside task success, even with a shared action interface and decision budget.

1 Introduction

Long-horizon embodied agents must handle physical and social constraints while adapting to changing conditions and human needs. TARSphere combines grounded execution, feedback-driven task updates, and a paired benchmark; evaluation finds capability gaps across model and task families, alongside persistent execution-safety challenges.

1 Introduction

  • Robots in everyday environments must account for physical constraints, human safety, appropriate interaction, and social norms as conditions and people’s needs change during execution.Long-horizon tasks require tracking world states, interpreting human intentions, and adjusting plans using execution feedback.Source: §1 · p. 2
  • Existing systems often target specified goals or abstract physical execution into symbolic actions, leaving physically and socially grounded evaluation in an evolving world underexamined.The cited systems include VLA and world-action models, embodied-agent benchmarks, and robot-execution settings with virtual NPC dialogue or human collaboration.Source: §1 · p. 2
  • A valid action call does not guarantee successful execution because contact, friction, balance, and human responses can require revising decisions from physical feedback.Physical conditions, human activity, and task requirements may also change while a plan unfolds.Source: §1 · p. 2
  • TARSphere-Infra runs robot actions under physical and social constraints, while its event-driven FSM uses completed actions and delivered information to update task states and evaluate outcomes.TARSphere-Harness uses observations, memory, executable skills, and execution feedback to guide model decisions.Source: §1 · p. 2
  • Across 8 MLLMs, general-purpose models can outperform embodied models, with capability gaps varying by task family; collisions and falls remain common.The findings motivate evaluating execution safety alongside task success and combining general reasoning, adaptation, and safe execution.Source: §1 · p. 2
  • TARSphere-Bench comprises 360 episodes in 30 scenes, pairing two service-task families with five capability conditions that probe changing perception, access, goals, and human needs.Paired variants retain their Base scene and task family.Source: §1 · p. 2

2 Related Work

Prior work advances embodied simulation, foundation models, and robotic agent systems, but the cited approaches cover different parts of physical execution, human interaction, and evolving task requirements. TARSphere combines these elements in an event-driven environment and evaluates decisions alongside execution and completion.

2.1 Embodied Interactive Simulation Environments

  • TARSphere couples physical robot execution, embodied NPC behavior, and changing task requirements in one event-driven environment, with paired conditions comparing adaptation and interaction in matched scenes and task families.Prior environments span household programs, navigation, manipulation, social navigation, collaboration, and NPC dialogue.Source: §2.1 · p. 3

2.2 Embodied Foundation Models

  • TARSphere evaluates frozen multimodal models through a shared harness with bounded primitives, higher-level skills, hierarchical memory, and measured execution feedback.The setting examines decisions under evolving goals, access, and human needs, alongside component ablations.Source: §2.2 · p. 3

2.3 Robotic Agent Systems and Interfaces

  • TARSphere retains robot dynamics and records collisions, falls, motion residuals, and task outcomes, with event-driven evaluation tied to the requirements active when evidence was produced.This supports analysis of how physical execution, adaptation, and human interaction jointly affect completion.Source: §2.3 · p. 3

3 Methodology

TARSphere formulates embodied execution as a feedback loop in a changing world and implements it with simulated robot and human interaction, an MLLM harness, and a curated benchmark. The framework ties task-state changes and evaluation to executed actions, delivered information, and active requirements.

3.1 Problem Formulation

  • At each decision, the policy selects an action sequence from observations and interaction history; actions execute sequentially, and new observations and measured feedback inform the next decision.The world update incorporates robot, object, human, and active task states as well as human or environmental events.Source: §3.1 · p. 3

3.2 TARSphere Framework

  • TARSphere-Infra manages physical execution, embodied human interaction, task states, and evaluation, while TARSphere-Harness organizes observations, memory, executable skills, and feedback for a frozen MLLM.The simulator executes robot control under contact and friction; NPC queries and speech depend on proximity, visibility, hearing range, and availability.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.1 TARSphere-Infra

  • The event-driven FSM admits events only when prerequisites and live conditions hold, commits transitions after confirmed action completion or message delivery, and evaluates outcomes against applicable requirements.Handover requires physical execution and recipient acceptance; task-update events are not scored.Source: §3.2.1 · p. 4

3.2.2 Embodied Brain Harness

  • The harness closes the loop by returning current observations, public events, and execution feedback, including motion residuals and collision bearings, so decisions can be revised at tool boundaries.Hierarchical memory retains recent actions and outcomes, the active plan, dialogue, and robot state; semantic spatial memory grounds selected entities in measured locations.Source: §3.2.2 · pp. 4–5, Figure 3 · p. 5
  • A unified skill interface combines bounded body, gaze, and end-effector primitives with manipulation and navigation skills; social actions use the same interface with spatial preconditions.Controller backends can be replaced by vision-language navigation and action policies.Source: §3.2.2 · p. 5

3.3 TARSphere-Bench

  • The benchmark contains 360 episodes across 30 indoor scenes, covering navigate-and-report and fetch-and-deliver tasks under five paired capability conditions.Task update revises the goal; human coordination requires another person’s information or work, while intent understanding requires clarification or contextual inference.Source: §3.3 · p. 5
  • Agent-in-the-loop curation jointly aligns task semantics, causal dependencies, and scene geometry, then uses agent and human review and simulation validation to check task transitions and execution.Human critiques guide episode corrections and SKILL.md refinement; simulation checks event delivery and physical execution.Source: §3.3 · pp. 5–6, Figure 4 · p. 6

4.1 Experimental Setup

The setup evaluates frozen MLLMs through a shared harness and benchmark, using staged task-quality scoring alongside efficiency and safety measures. The benchmark covers Base tasks and five conditions, with statistics on difficulty and human exchanges.

4.1 Experimental Setup

  • Eight frozen MLLMs use the same harness, benchmark episodes, prompts, observations, actions, 100-decision cap, and termination rules.Each decision uses the current image and 10 text-history steps to propose 1–5 actions; models receive no benchmark-specific training or adaptation.Source: §4.1 · p. 6
  • Task quality is scored across five stages, from task understanding and target observation to onsite completion, recipient observation, and final completion.Eligibility depends on event dependencies and live conditions; plan, inspection, and delivered-report quality are continuous, while observation, information receipt, and physical outcomes are binary.Source: §4.1 · p. 6
  • Figure 5 summarizes Base and five conditions across 360 episodes, alongside Base-task difficulty, human exchanges, and opening-utterance terms.Its panels cover the difficulty distribution for 60 Base tasks and exchange counts across episodes, including the opening, with a mean in the last column.Source: Figure 5 · p. 7
  • Progress includes initial plan and inspection quality, while Success depends on task completion and preserves earned Progress after terminal physical failure.Ordinary contact alone does not veto Success; scores average valid runs, and Avg. equally weights the six conditions.Source: §4.1 · p. 7
  • Efficiency measures successful Base-run path length against the authored route, while collision and fall incidence and decision/token cost are also reported.Path length integrates pelvis XY movement at native physics steps, excluding reset and warmup; unavailable efficiency entries are shown as dashes.Source: §4.1 · p. 7

4.2 Main Results

Across both task families, proprietary models lead, and Gemini-3.8-Flash substantially exceeds the runner-up; capability gaps vary by task family. Collisions and falls remain frequent, including for the strongest model.

4.2 Main Results

  • 40.6% navigation and 47.8% delivery success for Gemini-3.8-Flash exceed Claude Sonnet-5’s 16.9% and 24.6%, respectively.Proprietary models occupy the top two ranks in both task families, while the best open VLM outperforms embodied models.Source: §4.2 · p. 7, Table 2 · p. 8
  • 84–100% collision incidence and 9–47% fall incidence across model–task groups show that physical failures persist alongside task results.Even Gemini-3.8-Flash has collisions in 84% of navigation and 98% of delivery episodes, and falls in 14% and 16%, respectively.Source: Table 2 · p. 8, §4.2 · p. 8
  • Task-family capability gaps differ: PR/DA retain near-Base Progress, while navigation deficits are largest for IU/TU and delivery deficits for HC/IU.Gemini-3.8-Flash has identical 60.0% Base success in both families, yet its PR/DA success is higher in navigation and TU/HC/IU success is higher in delivery.Source: §4.2 · p. 8

4.3 Ablation Study and Analysis

The ablations show that execution assistance improves completion without removing task-update and human-coordination deficits, while harness choices trade local gains against average performance and cost. Failure attribution, execution checks, and curation ablations further expose where task completion and benchmark quality depend on reliable execution and review.

4.3.1 Harness Ablations

  • All six harness-component removals lower average Progress and Success; removing the grasp skill causes the largest Success decline, at 25.0 points.Removing LiDAR and motion residuals lowers Success by 12.5 and 11.1 points, respectively; semantic spatial memory removal raises PR and HC Success locally but lowers average performance.Source: §4.3.1 · pp. 8–9, Table 3 · p. 9
  • The default harness configuration achieves the highest tested average Progress and Success, balancing performance against decision and token costs.Ten text steps, the current image, and adaptive one-to-five-action decisions average 43.2 decisions and 10.5k tokens per decision.Source: §4.3.1 · p. 9

4.3.2 Controlled Intervention Study

  • Full assistance improves Success across conditions but leaves task-update and human-coordination deficits unchanged at 13.3 and 16.7 points.It narrows the disruption-adaptation deficit by 3.3 points; human coordination has the largest Progress gain (+7.3) without a smaller Success gap.Source: Table 4 · p. 9, §4.3.2 · p. 10

4.3.3 Failure Analysis

  • Failure analysis distinguishes the first unmet scored outcome from its attributed mechanism, with tool-interface errors accounting for 52.9% of Qwen3.7-Plus failures.Task-state errors account for 48.9% and 46.1% of failures for Qwen3.8-27B and Hy-Embodied-VLM-1.0; qualitative cases show recovery, wrong-person interaction, and unsupported completion failures.Source: §4.3.3 · p. 10, Figure 7 · p. 10, Figure 8 · p. 10

4.3.4 Execution Checks

  • Valid-call execution checks find translation errors of 1.3–4.9 cm, composite-motion endpoints 2.6 cm and 0.15° from target, and 229/240 successful grasps.Grounding identifies 285 correct instances and localizes 260 within 10 cm across 300 visible-target views, counting detection failures in both denominators.Source: §4.3.4 · pp. 10–11, Table 5 · p. 11

4.3.5 Curation Ablations

  • The full curation pipeline yields 96.7% usable episodes, while removing multi-agent review reduces yield to 45.0%.Removing human audit, SKILL.md refinement, or simulation validation yields 65.0%, 72.5%, or 83.3%, respectively.Source: §4.3.5 · p. 11, Table 6 · p. 11

4.4 Human Evaluation

Human evaluation compares semantic and human scores on plans and inspection/inquiry reports, and reviews task soundness and feasibility in human-coordination and intent-understanding tasks.

4.4 Human Evaluation

  • Plan scores reach SRCC/PLCC of 0.86/0.88, while inspection/inquiry report scores reach 0.91/0.92.Causal soundness and task feasibility each reach 100.0% across the reviewed HC and IU tasks.Source: §4.4 · p. 11

5 Conclusion

TARSphere couples robot dynamics, embodied NPC interaction, and state-machine evaluation with a harness that lets frozen MLLMs act through memory, executable skills, and measured feedback. Its benchmark pairs two service task families with five capability demands, while evaluation remains limited to one simulated robot body and two indoor task families.

5 Conclusion

  • TARSphere combines physically and socially grounded tasks, an execution-and-evaluation framework, a memory-and-skill harness, and a benchmark for changing task demands.The benchmark supports trajectory-based comparisons of decisions, completion, and execution quality, including revised goals and human participation.Source: §5 · p. 11
  • Evaluation covers one robot body and two indoor service task families in simulation, leaving real-robot transfer to address differences in sensing, contact, and human behavior.The authors plan Mobile-Franka transfer and task-adapted VLA, navigation, and manipulation backends.Source: §5 · p. 11
Loading 2610.30043v1…