PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards the Harness of Embodied Agents

Qi Wang, Tianyi Wang, Chengyang Li, Shikun Ban, Yurun Chen, Yizhong Ge, Jason Qin, Chengtai Li, Wentao Zhu

arXiv:2608.11246v1cs.AIcs.LGcs.RO

TL;DR

Physical agents lack the readable world state and action verdicts that software environments provide. Thea adapts the coding-agent harness with a persistent scene graph, callable robot tools, and an evaluator; it achieves the highest success rate at all three task-complexity levels and composes tools into long-horizon interaction. Its current loop does not act in real time.

  • Problem

    Physical agents receive neither a complete, persistent representation of the world nor a built-in judgment of whether an action has succeeded.

    Source: Gap 1: Readability of the World · p. 4, §4.2 · p. 19

  • Method

    Thea orchestrates robot capabilities as tools and uses a persistent scene graph for world state and an evaluator for action outcomes.

    Source: §4.1.2 · p. 17, §4.2 · p. 19, §7 · p. 27

  • Results

    Thea achieves the highest success rate at all three task-complexity levels and composes tools into representative long-horizon navigation and manipulation trajectories.

    Source: §5.2 · p. 23, §5.4 · p. 24

  • Takeaways & Limitations

    Representative trajectories are composed at run time from general tools rather than following task-specific prepared scripts.

    Source: §5.4 · p. 24

    Every decision passes through a language model, so the system does not yet act in real time.

    Source: §7 · p. 28

Abstract

from arXiv · show

The success of coding agents has established the harness as a paradigm: what an agent achieves depends not on the model alone, but on the infrastructure around it. We ask whether the same paradigm extends to embodied agents in the physical world. We present Thea, a harness in which an agentic loop orchestrates robot capabilities, each wrapped as a callable tool. It inherits the core components of coding agents, modified as the physical world requires. The world, however, withholds two abilities that software grants for free: reading the state of the world, and judging the outcome of an action. To bridge these gaps, Thea introduces Scene Graph as Context, a persistent, symbolic representation of the world, and Evaluation as Exit Codes, which detects when an action should terminate, judges whether it succeeded, and on failure diagnoses the cause. Together they close the loop between the agent and the physical world. Rich behaviors then emerge from the composition of tools, and the closed loop carries long-horizon tasks to completion in real environments.

2. Why the Harness Works

Thea connects a model and robot body through tools, with scene-graph state and evaluator outcomes closing the loop.

2. Why the Harness Works

  • The harness connects a swappable model and body, which the model controls through callable tools.The harness ports across embodiments and was validated on three robot platforms.Source: Figure 1 · p. 3
  • The model reads the resulting world state from the scene graph and the action outcome from the evaluator.Source: Figure 1 · p. 3
  • The harness ports across embodiments and was validated on three different robot platforms.Source: Figure 1 · p. 3

1. Introduction

Following coding agents’ harness paradigm, Thea uses a tool-based loop while rebuilding the world-state and outcome signals that software supplies for free.

1. Introduction

  • Coding agents’ harnesses close the loop between intent and outcome through infrastructure around the model.The paper asks whether this harness paradigm also holds in the physical world.Source: §1 · pp. 3–4
  • Thea adapts the coding-agent loop to robotics: a language model selects callable capability tools, observes results, and composes actions step by step.The loop accesses policies and platforms through tool interfaces, decoupling orchestration from execution.Source: §1 · p. 4

Gap 1: Readability of the World

  • Physical agents must construct a persistent world representation because sensing supplies only local, momentary slices rather than a complete, readable state.The input is a continuous stream of RGB-D frames, and the world keeps no record for the agent to consult.Source: Gap 1: Readability of the World · p. 4

Gap 2: Verifiability of Outcomes

  • Unlike software environments, the physical world provides no natural termination signal, success judgment, or failure diagnosis for an action.A robot’s continuous action stream does not indicate whether, for example, a grasp succeeded or the object slipped.Source: Gap 2: Verifiability of Outcomes · p. 4
  • Thea supplies the missing signals with a persistent scene graph and an evaluator that detects termination, judges success, and diagnoses failures.Together they close the loop the physical world otherwise leaves open.Source: Gap 2: Verifiability of Outcomes · p. 5
  • The harness is designed for extensibility and portability: new policies enter as tools, while models and embodiments can change without dedicated redesign.The agent also bridges user dialogue with autonomous physical interaction.Source: Gap 2: Verifiability of Outcomes · p. 5

2. Why the Harness Works

The section models task reliability as a product of step success probabilities, then shows how an evaluator-and-retry loop can improve it. The gains depend on evaluator accuracy, and adaptive replanning can make the model a lower bound on a well-designed harness.

2. Why the Harness Works

  • Open-loop task success is the product of per-step success probabilities, so each factor below one compounds loss as task length n grows.The product decays geometrically with n.Source: §2 · p. 6
  • The harness retries a step when its evaluator reports failure, and task reliability is the product of the resulting per-step success probabilities.The evaluator can correctly pass a success, retry a success it misclassifies, retry a detected failure, or pass through a missed failure.Source: §2 · p. 6
  • With a perfect evaluator, k retries compress per-step failure exponentially, while random guessing never exceeds the single-attempt success rate.The retry formula assumes independent attempts with the same per-step success probability.Source: §2 · p. 6
  • Evaluator accuracy limits harness gains, while each additional retry yields geometrically less improvement.The formula’s limit is capped directly by evaluator accuracy α, making reliable evaluation a primary design challenge.Source: §2 · p. 6
  • The model assumes independent, identically distributed retries and a fixed plan, but diagnosis can adapt attempts and replanning can raise step success, making it a lower bound on harness advantage.For example, the harness may adjust robot position or choose a more suitable grasp policy after failure.Source: §2 · pp. 6–7

3. System Architecture

Section 3 describes how Thea adapts coding-agent architecture for physical robots, combining a model-driven tool loop with context, tools, skills, memory, and safety components reshaped for the physical world. It also covers memory that carries lessons across tasks, safety checks before physical actions, and tools for bringing users into the loop.

3.1. Agentic Loop

  • Each turn, the model calls one tool and observes its result before choosing again; a turn without a tool call ends the task, while evaluation runs automatically after manipulation.This reactive loop treats evaluation as an explicit tool because the physical world supplies no exit code or stack trace; a post-execution hook triggers it after every manipulation.Source: §3.1 · pp. 7–8

3.2. Context Engineering

  • Context engineering separates stable resident knowledge, freshly refreshed physical evidence, and accumulated interaction history, matching each input to its role and lifetime.Refreshed context pairs a persistent scene-graph brief about the present world with current camera images and clearance measurements; accumulated messages record prior actions and results and can be compacted.Source: §3.2 · pp. 8, 10

3.3. Tool Protocol

  • The tool protocol gives capabilities a shared interface and result envelope, with contracts specifying use and success conditions and harness-owned hooks preflighting calls and triggering follow-up work.Lightweight tools run in-process and heavyweight backends in subprocesses, while the registry presents both uniformly; failed calls return reasons rather than exceptions.Source: §3.3 · pp. 11–12

3.4. Skills

  • Skills provide optional text-based domain knowledge, while physical actions remain tools so their contracts, safety checks, permissions, and evaluation attach at the call boundary.The model loads skill instructions when relevant, and new knowledge can be added by editing a file without retraining or code changes.Source: §3.4 · p. 13

3.5. Memory

  • Task Notes track active-task progress, while durable memory stores cross-session knowledge and per-tool experience, which the harness consolidates at task end and reloads at task start.Tool-specific experience captures how policy-backed tools actually succeed or fail, since their descriptions promise tendencies rather than guaranteed outcomes.Source: §3.5 · p. 13, Figure 5 · p. 14
  • Thea stores cross-task lessons in Memory and tool-specific lessons in each tool’s experience file, writing only candidates that exceed a confidence threshold.Task Notes preserve task-local progress, Memory remains resident context, and tool lessons are appended to the corresponding tool descriptions. Evaluator verdicts enter consolidation only as trace evidence, not as direct writes.Source: §3.5 · p. 14

3.6. Safety

  • Thea enforces physical safety through deterministic hooks and execution checks, including a motion filter that refreshes clearance before planning and rechecks admissibility before movement.Physical actions run one at a time, and a failure budget halts the loop; the agent can also stop and ask the user. The paper notes that force limits, safety around people, and deployment beyond the lab remain open.Source: §3.6 · p. 15

3.7. User Interaction

  • Thea’s query_user and notify_user tools let the agent request human input or report events, while remaining ordinary choices within the agentic loop.Questions are a last resort after consulting the scene graph and current observations, and can include candidate references and views as evidence; skills can guide notifications, while Memory stores settled answers as standing preferences.Source: §3.7 · pp. 15–16

4. Bridging the Gaps

The section builds infrastructure for the physical world’s missing readability and verifiability, while making body-specific capabilities explicit. Scene graphs, evaluation, and embodiment profiles adapt the coding-agent harness to these requirements.

4. Bridging the Gaps

  • The physical world lacks built-in readability and action verification, so Thea adds a scene graph, an evaluator, and a profile of the active robot’s capabilities.Source: §4 · p. 16

4.1. Scene Graph as Context

  • The scene graph pairs a persistent, symbolic world state with a context interface that gives the model a compact brief and ref-keyed access to more detail.Source: §4.1 · p. 16

4.1.1. Persistent World State

  • Stable object refs, positions, and freshness turn momentary physical observations into a world representation the agent can reference across turns.Source: §4.1.1 · p. 16
  • Observation-derived updates refresh graph state, while action-derived changes are admitted only after the evaluator confirms the tool’s post-condition.Source: §4.1.1 · p. 17

4.1.2. Accessing the World State

  • The per-turn brief includes object refs, coarse positions, confidence, freshness, robot state, and tracked container contents; less-used graph details remain available on demand.Source: §4.1.2 · p. 17
  • Ref-keyed queries retrieve relations or stored images to resolve descriptions to symbolic instances, but identity and coarse location do not establish manipulation readiness.Source: §4.1.2 · p. 18

4.2. Evaluation as Exit Codes

  • After each manipulation, the harness triggers an evaluator that reads the current observation and tool-specific post-condition, then returns a verdict and reason.Source: Figure 7 · p. 19, §4.2 · pp. 19–20
  • Evaluator timing depends on policy type: end-to-end runs use process, success, or failure at action-segment boundaries, while generated programs terminate through their own control flow.Source: §4.2 · p. 19

5. Experiments

The experiments compare Thea with alternative architectures on real robots, assess evaluator accuracy, and examine capabilities composed in full deployments. Thea leads the task-success comparison at every difficulty level, while demonstrations span extended tasks, active perception, recovery, user interaction, and portability.

5.1. Task Setup

  • The evaluation spans three task levels, from fixed-table pick-and-place to repeated manipulation and navigation between two workspaces, with 20 trials each on L1 and L2 and 15 on L3.The main quantitative experiments use the Astribot S1; additional deployments use AgileX Cobot Magic and Unitree G1.Source: §5.1 · p. 22

5.2. Scaling with Task Complexity

  • Thea achieves the highest success rate at all three task levels, with the advantage widening as task horizon and compositional complexity grow.The comparison covers L1–L3; Figure 8 reports success rates, and each trial requires all instructed objects and navigation and manipulation stages to be completed.Source: Figure 8 · p. 23, §5.2 · p. 23

5.3. Evaluator Accuracy

  • 93.3% average accuracy: the evaluator makes no false-success judgments on successful checkpoints, but labels 6.7% of failures and 13.3% of in-progress checkpoints successful.The test comprises 90 checkpoints sampled equally from successful, in-progress, and unrecovered-failure states across three embodiments.Source: §5.3 · pp. 23–24, Figure 9 · p. 24

5.4. Emergent Capabilities

  • Long-horizon demonstrations compose navigation and manipulation at run time, using scene-graph positions and evaluator verdicts to adjust, retry, or continue without task-specific scripts.Thea approaches with navigate_to and refines its base pose with move_base before manipulation.Source: §5.4 · p. 24
  • When targets are absent from the scene graph or outside the camera view, Thea explores and inspects drawers or tilts its head to gather new evidence; failure reasons also guide recovery actions.Successful recovery can be consolidated into a tool’s experience.Source: §5.4 · pp. 25–26
  • Thea asks users to resolve ambiguous choices rather than selecting a substitute itself, and the same harness runs across three embodiments through robot-specific profiles and tools.The loop above those profiles and tool implementations carries over unchanged.Source: §5.4 · p. 26

6. Related Work

The related work traces harness design from coding agents to embodied foundation models and robot-skill orchestration. Thea’s contribution is to carry the harness paradigm over as a complete deployed system that closes the loop from instruction to physical completion.

6.1. Coding Agents

  • Coding-agent harnesses combine reactive tool selection, recovery, managed context, and persistent memory; physical deployment requires rebuilding readable state and verifiable outcomes.The discussion identifies these two missing software affordances as properties without physical counterparts.Source: §6.1 · p. 26

6.2. Embodied Foundation Models

  • Embodied foundation models provide atomic capabilities, while single-step failures compound across long tasks, motivating harness-level evaluation and recovery rather than waiting for perfect policies.The section discusses both VLA models and World Action Models as complementary capability lines.Source: §6.2 · pp. 26–27

6.3. Orchestration for Embodied Agents

  • Prior orchestration work supplies pieces of the loop, but Thea carries the coding-agent paradigm over as the deployed system that closes instruction-to-completion execution in the physical world.The cited discussion notes missing general step-success judgment in earlier approaches.Source: §6.3 · p. 27

7. Discussion

The discussion presents Thea as an exploratory working system for embodied-agent harnesses and argues that harness components can evolve through improved representations, evaluation, and deployment. It also identifies current implementation limits, including perception errors, evaluator mistakes, and lack of real-time action.

Discussion

  • Thea is an exploratory implementation of an embodied-agent harness, adapting the loop, tools, context, and memory while rebuilding readability and verifiability as a scene graph and evaluator.The authors offer it as evidence that the harness paradigm can transfer to embodied AI.Source: §7 · p. 27
  • Reliability can be engineered in the harness without changing model weights, while new policies and embodiments can be added as tools and profiles without retraining the surrounding system.The discussion also presents tool choice, scene-graph reading, and recovery as potential training targets for future embodied models.Source: §7 · pp. 27–28
  • Current limits include coarse scene-graph positions and object-association errors, false-positive and false-negative evaluator verdicts, and decisions too slow for real-time action.Proposed directions include more durable representations, evaluator training and evidence gathering, and compact local evaluation models.Source: §7 · p. 28

A. Tool Registry

The appendix lists the Astribot S1 tool registry, describes the context assembled for a model decision, and provides execution logs from demonstrations on two embodiments.

A. Tool Registry

  • The Astribot S1 registry groups tools by function and distinguishes deployment-backed physical capabilities from in-process context and interaction tools.The evaluate_run tool is registered but invoked by a harness post-hook rather than selected by the model.Source: Table 4 · p. 34, A. Tool Registry · p. 34

B. Context Structure

  • A decision’s context concatenates three lifetimes: stable resident inputs, accumulated instructions and tool history, and refreshed scene-graph and observation evidence.The Scene Graph Brief is regenerated before each model call.Source: B. Context Structure · pp. 35–36

C. Demo Execution Logs

  • The demo logs cover four runs: the first three use Astribot S1, while the final run uses Unitree G1 with the same harness.Final responses have no tool calls and therefore no turn numbers.Source: C. Demo Execution Logs · p. 36

C.1. Beverage Retrieval

When no water bottle is available, Thea asks the user to choose an available drink; after the user selects orange juice, the robot delivers it.

C.1. Beverage Retrieval

  • After the user selects orange juice because no water bottle is available, the robot delivers the drink.The grasp and placement are each verified by the evaluator.Source: §C.1 · pp. 36–38
  • The robot searches the pantry, but its fresh view reveals no unambiguous plain water bottle, so it asks the user to choose an alternative.The robot reaches the pantry and advances to improve its view and manipulation distance.Source: §C.1 · pp. 36–37
  • After binding the selected drink by name, the robot navigates to desk_24, approaches the placement pose, and places the drink.The robot retains the drink during navigation; the evaluator verifies the placement.Source: §C.1 · pp. 37–38

C.2. Power-Bank Search

The robot searches the cabinet drawers in order, finds the power bank in the lower drawer, and delivers it to the desk.

C.2. Power-Bank Search

  • The robot finds the power bank in the lower drawer and successfully places it on desk_96.The evaluator verifies the placement.Source: §C.2 · pp. 38, 40
  • Because the power bank is absent from the visible scene graph, the robot navigates to cabinet_87 and inspects its drawers in order, starting with the upper drawer.The upper drawer is opened successfully; after the power bank is absent there, the robot closes it before opening the lower drawer.Source: §C.2 · pp. 38–39
  • The evaluator verifies the pickup and closure of the lower drawer before the robot carries the power bank to the desk.The cabinet search concludes with both compartments closed.Source: §C.2 · p. 39

C.3. Desk Cleanup

A standing monitor waits for an unoccupied workstation with objects on the desk; Thea then identifies an empty orange-juice bottle as trash and completes the cleanup.

C.3. Desk Cleanup

  • A standing monitor waits until the workstation is unoccupied while objects remain on the desk, then returns an event that supplies the task goal.The monitor is an ordinary tool that waits for a natural-language condition to hold.Source: §C.3 · p. 40
  • Thea discards the empty orange-juice bottle while leaving the laptop and charging power bank untouched.The robot drops the bottle into the bin and sends a completion notification.Source: §C.3 · pp. 40, 42
  • Thea treats only the confirmed empty bottle as discardable; the laptop, charging power bank, and items of uncertain state must remain untouched.It sends a non-blocking notification before acting.Source: §C.3 · p. 41
  • After grasping and carrying the bottle to the bin, the evaluator verifies the drop by confirming both grippers are empty and no bottle is visible outside the bin.The cleanup result is then delivered to the user.Source: §C.3 · pp. 41–42

C.4. Basket Delivery

The robot packs snacks into a basket, carries it to Jason in the living room, and hands it over; the evaluator verifies the release.

C.4. Basket Delivery

  • The robot carries the snack basket to Jason and releases it securely with the recipient.The evaluator verifies the release rather than relying on the controller state alone.Source: §C.4 · pp. 42–43
  • The snack is verified in the basket, and the loaded basket is lifted and prepared for carrying.The robot waits for packing to finish before lifting the basket.Source: §C.4 · pp. 42–43
  • The robot reaches Jason in the living room, then uses four short forward motions to approach the seated recipient before extending its arm and opening its hand.The reasoning explicitly cautions against inferring transfer from the controller state alone.Source: §C.4 · p. 43

Related papers

Loading 2608.11246v1…