PaperlayerView exact revision on arXiv

Source-linked AI summary

Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools

Shijia Ge, Alex Zhou, Jianshu Zeng, Yexing Wan, Di Wu, Zelin Zheng, Yazhe Wang, Zhiqi Jia, Xuan Shangguan, Jay Zhu, Yijun Liu, Lingyu He, Sihang Wu, Xiao He, Hongcheng Gao

arXiv:2609.39018v1cs.ROcs.AI

TL;DR

Reasoning through every robot motion step can make manipulation slow and token-intensive. URAI has programming agents build and revise tools for complete motions while execution agents choose the next tool from feedback. Across five RoboDojo tasks, it raises aggregate success over direct fingertip control.

  • Problem

    Repeated model-selected motions add waiting time and tokens, motivating whether tools can handle complete motions while the policy retains model decisions.

    Source: §1 · pp. 1–2

  • Method

    URAI pairs a programming agent that writes and validates reusable or task-specific tools with a frozen execution agent that selects tools from observations and feedback.

    Source: §1 · pp. 2–3

  • Results

    53.0% versus 18.0% aggregate success across five RoboDojo tasks for URAI and direct fingertip control, respectively, across four execution agents.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    URAI organizes robot control around reusable tools that agents can invoke and revise while the execution agent decides between tool calls.

    Source: §1 · p. 2, §6 · p. 8

    Five episodes per cell do not establish per-task differences.

    Source: §5 · p. 8

Abstract

from arXiv · show

Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot-Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. Three of the four agents also finish episodes 1.3-1.5 times faster with 1.5-1.7 times fewer execution-agent output tokens; DeepSeek-V4-Flash's cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise.

1 Introduction

The paper asks whether agents can author and revise robot tools so that each model decision covers a complete motion, addressing the time and token costs of deciding every motion step. URAI keeps the execution agent deciding between tool calls while a programming agent builds and validates the tools.

1 Introduction

  • Repeated observation–reasoning–action cycles add waiting time and tokens, motivating the question of whether agents can author tools for complete motions while retaining policy decisions.The paper frames the approach as analogous to developers running and revising code written by coding agents.Source: §1 · pp. 1–2
  • A programming agent writes and revises reusable or task-specific tools, while a frozen execution agent selects a tool from observations and decides again after its motion completes.Each tool call can cover approach, grasp, transfer, placement, and retreat without a separate model call for each phase.Source: §1 · p. 2
  • Validated tool revisions persist across episodes without foundation-model weight updates, and humans and agents access the same tools through a shared GUI and API.The backend grounds requests, checks planned motion, and executes locally.Source: §1 · p. 3
  • URAI distinguishes itself by returning control to a frozen execution agent after every complete tool execution and sharing one tool contract with humans through a GUI.Related systems share elements such as coding agents, feedback, skill synthesis, or tool-maker/user separation.Source: §1 · p. 3
  • URAI raises five-task RoboDojo success from 18.0% to 53.0% over direct fingertip control; deciding after each call reaches 56% against 24% for a program written in advance over the same tools.Three of four agents also finish 1.3–1.5× faster with fewer output tokens.Source: §1 · p. 3

2 Related Work

Prior work spans frontier-model control, learned action policies, program-based execution, and interfaces for robot interaction. URAI’s related-work distinction is its combination of complete tool calls, renewed model decisions between calls, and a shared human-agent interface.

Frontier-model control and action granularity

  • Prior systems include hierarchical program generation and feedback-driven skill synthesis, coding agents that revise programs from physical feedback, and persistent skill or policy-repair approaches.Examples include Code as Policies [26], CaP-X [11], Agent as Policy [21], Voyager [40], ASPIRE [30], ENPIRE, and RHO [10].Source: §2 · p. 3
  • Other approaches use value maps, visual marks, constraints with local optimization, scene-graph memory, or language-model planning with environment feedback.The section also contrasts step-by-step frontier-VLM control on real robots [8].Source: §2 · p. 3
  • Robot policies range from learned motor-token or continuous action outputs to fixed vocabularies of discrete translation, rotation, and gripper commands.The cited examples include OpenVLA [24], Qwen-RobotManip [44], RoboDawn [14], and Show-Harness [7].Source: §2 · pp. 3–4

Interfaces and efficiency

  • Unlike interfaces built around paths, sketches, waypoints, or single-step commands, URAI’s shared human-model action vocabulary consists of complete timed tools with explicit metric arguments.The paper characterizes this as reducing the number of decisions per episode rather than the cost of each decision.Source: §2 · p. 4

3 Code as Policy with the Agent in the Loop

URAI separates fixed foundation-model weights from an external, revisable tool policy: an execution agent selects a tool, and its code runs a timed motion locally. The interface connects tool authoring, feedback-driven selection, motion specification, and local validation and execution.

3.1 The Policy: Agent-Written Tools and Feedback-Driven Selection

  • The policy separates fixed execution-agent weights from an external tool collection, executable code, and execution strategy; the programming agent proposes revisions and retains only validated versions.The execution agent maps instruction, observation, and history to a tool call whose code runs a complete multi-phase motion locally.Source: §3.1 · p. 4

3.2 Interaction Contract

  • The programming agent can revise reusable tools or create task-specific ones, while agents or humans invoke available tools through a GUI or API with shared execution semantics.The backend grounds and validates requests, executes locally, and can pass recurrent failures back for versioned revision.Source: §3.2 · p. 4

3.3 A Motion Representation with Explicit Timing

  • A motion request specifies a metric path, desired end-effector orientation, translational speed profile, gripper events and holds, and execution settings.Path progress is not wall-clock time; gripper events are tied to motion progress and execution conditions.Source: §3.3 · pp. 4–5

3.4 Grounding, Editing, and Local Execution

  • The GUI and API expose corresponding motion controls, while the backend checks the full trajectory, including the approach from the current pose, against configured constraints before local execution.On the AgileX backend, references stream locally at a nominal 50 Hz.Source: §3.4 · p. 5

4 Experiments

URAI was evaluated across five RoboDojo tasks, seven AgileX tasks, and a case study revising a clothes-folding tool. In RoboDojo, URAI improved aggregate success over native control, while real-robot comparisons and tool evolution provide additional evidence about performance and reuse.

4.1 RoboDojo: five-task success rate

  • 53.0% vs 18.0% overall success favored URAI; with the same tools, between-call decisions yielded 28/50 successes versus 12/50 for advance-written programs.The URAI advantage remained without swap blocks: 42.5% vs 22.5%.Source: Table 1 · p. 6, §4.1 · pp. 6–7
  • The RoboDojo study compared four frozen execution agents using the same 13 tools across five tasks, with five episodes per task and task-specific budgets.A first round for pour balls prompted development of a task-specific pouring tool, after which URAI and program episodes were rerun.Source: §4 · p. 5, §4.1 · p. 6

4.3 Tool evolution: revising one tool for fold clothes

  • For clothes folding, the programming agent revised a two-point PnP tool seven times, adding sheet pinching, synchronized transport, and depth-based release while preserving its call signature.The final tool succeeded in 3/20 and 2/20 episodes across two seed sets; a human choosing pixels through the GUI succeeded in 4/5.Source: §4.3 · p. 7

4.1 RoboDojo: five-task success rate

  • URAI used 3.5–10.0 calls per episode lasting 29–72 s each, versus 21.0–25.6 calls of 19–20 s natively; time outside commands fell for three agents but rose for DeepSeek.The time outside robot and observation/probe commands fell about 45%, 38%, and 32% for Fable, Opus, and Astra, respectively.Source: §4.1 · p. 7, Table 2 · p. 8

4.2 Real-robot evaluation

  • All three real-robot tasks with published references succeeded in 3/3 URAI trials; tic-tac-toe was 2.7× faster at the same 3/3 success than GPT-Policy.Unscrew bottle cap was 3.5× faster than R3; block into bowl was 2.4× faster with 4.7× fewer tokens than Robocurve.Source: §4.2 · p. 7, Table 3 · p. 8

5 Limitations and Discussion

The reported results are bounded by small per-cell samples, limited controls, unmatched real-robot comparisons, and unmetered authoring costs; portability beyond the tested platforms is not established.

5 Limitations and Discussion

  • Five episodes per cell do not establish per-task differences, despite significant pooled success gains with and without swap blocks.The pooled McNemar p-values were 7.9 × 10^-8 with swap blocks and 0.0025 without them.Source: §5 · p. 8
  • The advance-written program control covered two agents on seeds 0–4, and no prompt- or memory-only revision control was run.Source: §5 · p. 8
  • Real-robot ratios are not causal speedup estimates because references used other hardware and no same-robot step-by-step baseline was run.Tool programming and author steering were unmetered; the interface was tested on one simulator and one position-controlled dual-arm platform without force regulation, so portability is unestablished.Source: §5 · p. 8

6 Conclusion

URAI pairs agent-written tools with feedback-driven tool selection, and its evaluations span simulated and real-robot tasks. The authors conclude that agents deciding after each call outperform advance-written programs in the reported RoboDojo comparison.

6 Conclusion

  • URAI pairs agent-written tools with feedback-driven selection by the execution agent through an interface shared with humans.Source: §6 · p. 8
  • Figure 3 shows seven AgileX tasks; clothes folding is a demonstration-video run, while the other rows are evaluated trials.Source: Figure 3 · p. 9
  • Across five RoboDojo tasks, four frozen agents succeed more often through URAI than with direct fingertip control; three are also faster and use fewer output tokens.Source: §6 · p. 9
  • The authors plan to extend the tool collection to more tasks and robot platforms and meter tool-authoring costs alongside execution costs.Source: §6 · p. 9

A Extended Related Work

Related work spans frontier-model spatial reasoning and physical control, program- and tool-based robot policies, and interfaces that shape action granularity. URAI’s focus is a persistent collection of executable tools that a frozen agent selects between calls, with shared human and agent operation and explicit motion geometry and timing.

Frontier models in spatial software work and physical manipulation

  • Frontier models have demonstrated spatial software work and physical manipulation, but software examples alone do not establish manipulation performance.In physical control, the model selects end-effector targets and gripper commands from visual observations, while robot inverse kinematics and a local controller realize each command.Source: A Extended Related Work · p. 13

Frontier models as robot decision makers

  • URAI differs from prior robot policies by keeping the frozen execution agent responsible for choosing the next complete tool after each execution.Related approaches place run-time decisions in generated programs, composed objectives, or step-by-step model choices; URAI uses persistent executable tools written and revised by coding agents.Source: A Extended Related Work · p. 13

Persistent program and policy improvement

  • URAI retains validated tool revisions across episodes through natural-language-steered programming rather than reward-driven search.The execution agent invokes a frozen tool version during evaluation, while humans use the same tools through a GUI with the same execution semantics.Source: A Extended Related Work · pp. 13–14

Recent frontier-model control evaluations

  • Published frontier-model evaluations use different hardware, task sets, action budgets, and protocols, so their results are references rather than direct controls for URAI.For example, GPT-Policy’s two stationary tabletop tasks are run on URAI’s AgileX platform, while its mobile exploration task is excluded.Source: A Extended Related Work · p. 14

Action tokens and execution granularity

  • URAI’s discrete calls can invoke reusable or task-specific tools whose internal motion phases run locally, unlike fixed discrete action symbols mapped to pose increments.The distinction concerns the duration and responsibility of a model-level decision, as well as the cost of authoring the tool.Source: A Extended Related Work · p. 14

Sketches and intermediate representations for manipulation

  • Prior sketch and interface systems establish precedents for shared human/model inputs, while URAI emphasizes explicit metric geometry and timing at the command boundary.URAI’s timed tools with explicit metric arguments differ from fixed-increment semantic controls; its interface perspective also targets the number of decisions per episode rather than only the cost of each decision.Source: A Extended Related Work · p. 15

B Implementation Details

The implementation connects visual observations and editable paths to locally executed, checked robot motions. Its GUI and API share execution semantics, while tool calls plan complete staged motions and report their outcomes.

Tool calls

  • Each tool call plans all stages before motion, and its reply reports completed stages rather than declaring the task complete.A dry run returns the plan without motion; pre-motion refusals identify the failed stage, and each post-motion call must use a fresh observation because its identifier expires when the robot moves.Source: B Implementation Details · p. 16

Path inputs and timing rules

  • The prototype interpolates rotations on SO(3), with separate timing rules for pure rotations, endpoint ramps, and stationary holds.It also accounts for angular speed and robot-dependent motion limits when generating references from requested speeds.Source: B Implementation Details · p. 16

Gripper events

  • Ordinary placement opens the gripper after reaching the configured arrival condition, whereas dynamic release ties dispatch to motion progress and a calibrated delay.For ordinary placement, the sequence is descent, arrival condition, opening, and retreat.Source: B Implementation Details · p. 16

Visual grounding, GUI, and API

  • Depth at a valid image pixel is back-projected using camera intrinsics and the camera-to-world transform to locate an observed surface point.That surface point does not determine the desired height of an airborne motion, so free-space path editing is distinct from surface-grounded input.Source: B Implementation Details · p. 16
  • The GUI supports path drawing, arm and orientation selection, speed profiles, and explicit gripper events, with higher-level modes generating multi-phase motions from sparse input.Humans can call tools by clicking pixels.Source: B Implementation Details · p. 16
  • The API exposes corresponding control points, profiles, motion settings, and tool calls, and its bounded arithmetic interpreter can specify a curve without mouse actions for each point.Equivalent visual and API requests use the same backend and execution semantics.Source: B Implementation Details · p. 16

Local execution backend

  • The backend constructs an approach when the requested start differs from the current end-effector pose and checks the approach and task trajectory against kinematic and geometric constraints.For dual-arm motions, selected arms can reach their start poses before their task trajectories begin together; changed robot state requires refreshing and rebuilding the versioned plan.Source: B Implementation Details · p. 16
  • On the AgileX backend, references are streamed locally at a nominal 50 Hz, with model-based checks for the robot and calibrated table and gripper feedback as an execution signal.Figure 2 details the two loops summarized by Figure 1.Source: B Implementation Details · pp. 15–16

C Experimental Setup

The appendix specifies the configurations used for the simulated and real-robot evaluations.

C Experimental Setup

  • The appendix provides the configuration of the simulated evaluation in Section 4.1 and the real-robot evaluation in Section 4.2.Source: C Experimental Setup · p. 16

C.1 RoboDojo

The RoboDojo setup evaluates five tasks under URAI, native fingertip control, and a program condition, using frozen tools and four frozen execution agents. The protocol defines observations, budgets, accounting, and integrity checks, including a common environment fix for Swap Blocks.

Tasks

  • The five RoboDojo tasks are Pour Liquid Into Cup, Pour Balls Into Vase, Fold Clothes, Swap Blocks, and Stack Blocks, each with a task-specific step budget and public success criterion.Partial credit for sleeves alone or two of three stacked blocks does not count as success.Source: §C.1 · p. 17

Execution agents

  • Four frozen execution agents—Claude Fable 5.1, Claude Opus 5.5, GPT-6 Astra, and DeepSeek-V4-Flash—run episodes in fresh sessions without access to other episodes or the checker.The task brief supplies the instruction, public criterion, step budget, and command syntax; every command is logged.Source: §C.1 · pp. 17–18

Observations and depth probe

  • Observations provide a 640×480 head-camera image, two wrist-camera images, both arms’ state, and remaining budget; depth probes return world coordinates for a head-camera pixel.The same observation and probe commands are available in every condition, and only played simulator steps count against the budget.Source: §C.1 · p. 17

Conditions

  • URAI agents choose and parameterize one tool call at a time, native agents command fingertip poses or grippers, and the program condition runs a single written program without the model.The program is authored in a planning session that permits observations, depth probes, schemas, and dry runs but no robot motion, then runs unchanged using the same tools.Source: §C.1 · p. 17

Stopping, limits, and continuation

  • Episode completion requires success before stopping criteria such as judged completion or impossibility, exhausted motion budget, or 100 commands; failures and agent-side failures count as failures.Wall-clock limits differ by condition, and a program stopped at its limit or crashing counts as a failure.Source: §C.1 · pp. 17–18

Timing and token accounting

  • Timing spans the agent session, including planning and program execution where applicable, and means include failed episodes.Output-token and call counts come from each harness’s logs; Codex polling counts as a model call, so call counts are not comparable across harnesses.Source: §C.1 · p. 18

Swap-blocks environment fix

  • Swap Blocks uses a common reset bootstrap fix because the unpatched button state made the checker’s post-press condition impossible to satisfy.The change reapplies declared drive targets after reset, touches neither tools nor scoring, and all conditions use the patched environment.Source: §C.1 · p. 18

Tool collection and freezing

  • The same thirteen reusable tools are exposed in URAI and the program condition, with task-specific frozen code trees and no execution-agent access to tool code.The pour-balls tool and fold-clothes pick-and-place tool have task-specific revisions frozen before evaluation.Source: §C.1 · p. 18

C.2 AgileX real robot

The real-robot evaluation uses a two-arm AgileX platform for seven tabletop tasks, including three with published task references. Those references use different hardware and operating limits, so they are not same-robot controls.

C.2 AgileX real robot

  • The platform uses two AgileX PiPER-X arms with parallel-jaw grippers and four RealSense RGB-D cameras, with both arms reaching a common workspace.The arms are mounted about 0.59 m apart; the cameras comprise two wrist cameras and two fixed scene cameras.Source: §C.2 · p. 19
  • Seven tabletop tasks use GPT-6 Astra with URAI tools; three have published task references, while four have no published same-task reference.The referenced tasks are unscrew bottle cap, tic-tac-toe, and block into bowl; clothes folding is the physical counterpart of a simulated task.Source: §C.2 · p. 19
  • Task success requires whole-task completion, with task-specific criteria including winning or drawing at tic-tac-toe and placing all six blocks in the bowl.Progress is also tracked for hot-dog serving, block tossing, and clothes folding.Source: §C.2 · p. 19
  • The three tasks with published references were run for three trials each at 1× robot speed; the other four ran for ten trials each, with time and tokens measured over three.Logged pour-blocks trials used a 0.8 robot speed scale.Source: §C.2 · p. 20
  • Episode time includes inference, network waiting, motion, recovery, verification, and human-move waiting in tic-tac-toe; token totals count execution-agent outputs.Means include failures, while model calls include observations, status queries, retries, recovery, and replies; loops inside tools are excluded.Source: §C.2 · p. 20
  • Published comparisons use different hardware and operating limits, and the reported decision counts are narrower than URAI’s model-call counts.GPT-Policy reports no per-task token counts for R3 or R8, and the compared references are not same-robot controls.Source: §C.2 · p. 20

Related papers

Loading 2609.39018v1…