PaperlayerView exact revision on arXiv

Source-linked AI summary

Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens

Ruiyang Si, Jianxin Bi, Shunyu Yang, Rui Ni, Wenbo Huang, Qiang Wang, Shulong Jiang, Duomin Wang, Xiuyu Li, Haiwen Feng, Zhen Dong, Daquan Zhou

arXiv:2610.01939v1cs.CV

TL;DR

VLM robot agents can incur substantial overhead from repeated invocations and redundant observations. PyRUA-Lean uses executable code to compose primitives and select feedback; across 700 simulated tasks, it improves overall success over tool calling and uses fewer calls and input tokens on jointly solved instances.

  • Problem

    Tool-calling robot agents generally require another VLM invocation for dependent operations, while intermediate results and automatically returned images add to subsequent context.

    Source: §1 · p. 2

  • Method

    PyRUA-Lean lets a VLM compose robot primitives in Python cells, check execution feedback, and return only explicitly requested outputs and camera images.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across 700 simulated task instances, overall success rises from 63.1% to 71.7%; on jointly solved instances, PyRUA-Lean uses 49% fewer LLM calls and 65% fewer input tokens.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Fewer LLM invocations account for most token savings, while gains are smaller on VLA-dominated tasks.

    Source: §6 · p. 8

    Evaluation is limited to GPT-6 Astra, RPent’s robot stacks and primitive libraries, and simulation, with one run per agent per task instance.

    Source: §5 · p. 8

Abstract

from arXiv · show

Vision language model (VLM) agents can control robots through visual feedback and action primitives, but repeated model invocations and redundant observations incur substantial token overhead. We introduce PyRUA-Lean, an interactive code-execution framework that couples feedback-driven primitive composition with selective observation: the agent composes classical robot primitives and learned vision-language-action (VLA) policies into Python cells that perform conditional checks and local retries, returning only explicitly requested images and state feedback for replanning. Across 700 simulated task instances from LIBERO-PRO, RoboTwin 2.0, and RoboCasa365, we compare PyRUA-Lean with a tool-calling baseline using the same GPT-6 Astra planner and underlying robot primitives. Under equal LLM-call budgets, PyRUA-Lean increases overall success from 63.1% to 71.7%. On instances solved by both agents, it uses 49% fewer LLM calls and 65% fewer input tokens.

1 Introduction

PyRUA-Lean addresses the overhead of repeated VLM calls and redundant observations by letting agents compose robot primitives and select feedback through executable code. Across 700 simulated instances, it improves success over tool calling and reduces calls and input tokens on jointly solved instances.

1 Introduction

  • Tool-calling agents generally need another VLM invocation for operations that depend on prior results, while intermediate outputs and automatically returned images add to later context.Tasks involving repeated localization, motion, and recovery can incur substantial inference overhead even when individual primitives are reliable.Source: §1 · p. 2
  • Executable code can compose primitives, inspect outputs, and run conditional branches or retries before returning control to the VLM.Prior work in robotics and general-purpose agents motivates asking how primitive composition and selective observation can reduce token use while completing tasks reliably.Source: §1 · p. 2
  • PyRUA-Lean keeps intermediate results in persistent program state and returns only explicitly printed outputs and requested camera images to the VLM.The interface supports feedback-driven primitive composition and lets the agent control which observations enter its context.Source: §1 · p. 2
  • The comparison uses the same GPT-6 Astra planner, frozen VLA policies, task instances, and LLM-call budgets, with no cross-episode memory.The complete interfaces are compared over 700 simulated instances from three benchmarks.Source: §1 · p. 2
  • Approximately 14% higher overall success than tool calling; on jointly solved instances, PyRUA-Lean uses 49% fewer VLM calls and 65% fewer input tokens.The comparison covers 700 simulated instances across LIBERO-PRO, RoboTwin 2.0, and RoboCasa365.Source: §1 · p. 2

2 Related Work

Related work spans VLM-based robot control, executable robot programs, and code-based interfaces for general-purpose agents. PyRUA-Lean focuses on interactive code execution over shared robot primitives and compares it with tool calling without cross-episode skill accumulation.

2 Related Work

  • VLM robot-control work ranges from language-grounded skill selection and feedback-informed planning to semantic actions, embodied action generation, and VLA policies.Harness VLA integrates frozen VLA policies with analytic primitives through RPent [33].Source: §2 · p. 2
  • Robot-programming research includes language-generated programs, spatial value maps, execution feedback, program adaptation, and reusable skills [14] [25] [11] [9] [17] [28] [16].These studies treat program synthesis, feedback-driven execution, and skill reuse as complementary approaches to embodied control.Source: §2 · p. 3
  • The paper compares interactive code execution with tool calling over the same robot primitives, including VLA policies, while disabling cross-episode memory.Within-episode state and code reuse remain allowed.Source: §2 · p. 3
  • General-purpose agent studies examine reasoning with environment interaction, API use, executable Python, and code-mediated processing of intermediate results [32] [24] [29] [31].PyRUA-Lean applies these interface principles to visual robot manipulation and evaluates success alongside call use, input tokens, and inference cost.Source: §2 · p. 3

3 PyRUA-Lean: Primitive Composition with Selective Observation

PyRUA-Lean exposes robot primitives through an interactive Python interface that combines execution-feedback-driven programs with selective observation. Cells retain intermediate state for conditional actions and retries, then return requested outputs and images for VLM replanning.

3 PyRUA-Lean: Primitive Composition with Selective Observation

  • PyRUA-Lean is an interactive code-execution interface that combines robot-primitive composition with control over which execution feedback enters the VLM context.It builds on RPent’s robot stacks and primitive implementations, including classical motion and perception primitives and learned VLA policies [33].Source: §3 · p. 3

3.1 Robot Interface and Execution Model

  • The agent calls primitives through a Python robot object and code tool; a persistent namespace retains variables and helper functions across cells, but not across episodes.An uncaught exception returns a traceback while preserving variables assigned before the exception; a watchdog enforces the two-hour episode limit.Source: §3.1 · p. 3

3.2 Feedback-Driven Primitive Composition

  • A cell can condition later primitive calls on earlier outcomes, with checks, branches, and retries running without additional LLM invocations.At cell end, feedback allows the LLM to replan; programs can also compute task-specific values from observations, such as a bowl placement target from a world-coordinate map, and retain helper functions for later cells.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Selective Observation and Context Management

  • Primitive results stay in the runtime unless explicitly printed or requested, and requested images and state messages return together at cell end.In the placement example, the cell requests an image only if the task remains unfinished, so successful placement returns no image; selective feedback controls new context rather than removing existing conversation history.Source: §3.3 · p. 4, Figure 2 · p. 4

4 Experimental Results and Analysis

Across four benchmark groups, PyRUA-Lean improves task success under equal LLM-call budgets and uses fewer calls and input tokens on jointly solved instances. The analysis finds that fewer model invocations account for most token savings, while ablations and failure cases qualify where the gains hold.

4.2 Task Success and Inference Cost

  • 71.7% versus 63.1% overall success, an 8.6-point gain, for PyRUA-Lean over tool calling under equal LLM-call budgets.Gains span all benchmark groups; the reported relative improvement is approximately 14%.Source: Table 2 · p. 5, §4.2 · p. 6

4.1 Experimental Setup

  • The evaluation pairs both interfaces on 700 simulated task instances, using the same planner, robot stacks, frozen VLA policies, and LLM-call budgets.Instances span LIBERO-PRO, RoboTwin 2.0, and RoboCasa365; the agents have no cross-episode memory.Source: §4.1 · p. 5

4.2 Task Success and Inference Cost

  • On jointly solved instances, PyRUA-Lean uses 49% fewer LLM calls and 65% fewer input tokens than tool calling.Estimated cost falls from $1.63 to $0.74 per episode; caching discounts partly explain the smaller monetary reduction.Source: Table 2 · p. 5, §4.2 · p. 6

4.3 Analysis of LLM Calls and Observation Feedback

  • Fewer invocations account for most token savings: on LIBERO-PRO, the 4.48× token ratio comprises 2.55× fewer calls and 1.76× fewer tokens per call.On RoboCasa365 atomic tasks, PyRUA-Lean calls are larger on average, but fewer invocations still reduce total token usage.Source: §4.3 · p. 7, Figure 5 · p. 7
  • Returning images only on request does not reproduce PyRUA-Lean’s gains: the modified baseline’s success falls from 83.0% to 68.5%.On the 131 instances solved by all three configurations, it makes more calls and consumes approximately the same total input tokens as the original baseline.Source: §4.3 · p. 7

4.4 Ablation Studies

  • Without VLA policies or operating guides, PyRUA-Lean retains higher overall success and lower input-token use on jointly solved instances; guide removal can also raise tool-calling success.On RoboTwin 2.0 without VLA policies, removing guides raises tool-calling success from 42.8% to 60.4%.Source: §4.4 · p. 7

4.5 Failure Cases and Limitations

  • Of the 96 instances solved only by PyRUA-Lean, 52 baseline runs exhaust the call budget, while 44 end unsuccessfully before doing so.The success difference cannot be attributed entirely to better recovery logic; some cases include VLA run-to-run variation.Source: §4.5 · p. 7
  • Efficiency gains are smallest on RoboCasa365 atomic tasks, where larger API descriptions offset some savings and all-attempt estimated inference costs are similar.The estimated tool-calling-to-PyRUA-Lean cost ratio is approximately 1.1× when failed episodes are included.Source: §4.5 · pp. 7–8

5 Limitations and Future Work

The evaluation leaves generalization beyond one planner, one primitive stack, and simulation untested, and does not isolate the effects of individual interface features.

5 Limitations and Future Work

  • Evaluation covers only GPT-6 Astra, RPent’s robot stacks and primitive libraries, and simulation, with one run per agent per task instance.Generalization to other planners and primitive configurations, repeated-run variability, and physical-robot performance remain untested.Source: §5 · p. 8
  • The comparison evaluates complete interfaces, so it does not fully isolate primitive composition, persistent state, and selective feedback.Future work will measure execution latency and physical-robot recovery, and examine primitive granularity and cross-episode reuse of validated routines.Source: §5 · p. 8
  • Future studies will examine how primitive granularity and cross-episode reuse affect success, inference cost, and adaptability.The authors also plan to measure execution latency and recovery on physical robots.Source: §5 · p. 8

6 Conclusion

PyRUA-Lean composes existing robot primitives and controls which execution feedback enters the LLM context. In 700 simulated instances, it improves success over tool calling and uses fewer calls and tokens on jointly solved instances.

6 Conclusion

  • PyRUA-Lean composes existing robot primitives, including VLA policies, while controlling which execution feedback enters the LLM context.The results motivate further study of feedback management in robot agents.Source: §6 · p. 8
  • Overall success increases from 63.1% to 71.7% across 700 simulated task instances under equal LLM-call budgets.The tool-calling baseline uses the same GPT-6 Astra planner and underlying primitives.Source: §6 · p. 8
  • On instances solved by both agents, PyRUA-Lean uses 49% fewer LLM calls and 65% fewer input tokens.Fewer LLM invocations account for most token savings, with smaller gains on VLA-dominated tasks.Source: §6 · p. 8

A The episode of Figure 2, call by call

An illustrated LIBERO-PRO episode contrasts tool-by-tool control with code that groups dependent actions and selectively requests images. Additional episodes are selected from tasks solved by both agents to show token consumption across benchmarks.

A The episode of Figure 2, call by call

  • The code agent localizes both objects first, then approaches and grasps, closes the gripper and lifts, checks the grasp, and carries and places the bowl.These steps are spread across six calls; the tool-calling agent moves one step per call and receives three camera images after every move.Source: A The episode of Figure 2, call by call · p. 11
  • In the illustrated LIBERO-PRO episode, tool calling solved in 17 calls after 33 images, while code solved in 6 calls with 5 images (816k vs 156k tokens).Figure 6 follows the whole episode and marks the solving call and token consumption.Source: Figure 6 · p. 11, A The episode of Figure 2, call by call · p. 11

B More episodes

  • Table 5 and Figure 7 cover five jointly solved episodes per benchmark, selected by how close each task’s token factor is to that benchmark’s median.The table reports whole-episode calls and token consumption for tool calling and code; shading indicates how many times fewer tokens code consumed.Source: B More episodes · p. 13, Table 5 · p. 13

C What the code agent writes

Code cells combine robot actions, checks, and computation locally, while selecting when to return visual feedback to the planner. Their behaviors range from guarded action sequences and retries to perception and reusable helper functions.

C What the code agent writes

  • Code cells run 2.1 acting primitives on average, and 45% run at least two; tool-calling LLM calls average 0.65 acting primitives.Of tool calls that invoke a tool, 89% invoke exactly one; only 45 of 18,585 calls run two acting tools together.Source: C What the code agent writes · p. 13
  • Guarded chains make up 19% of code cells and 45% on LIBERO-PRO, linking actions through conditional checks within a cell.With tool calling, each link in such a chain requires another LLM call.Source: C What the code agent writes · pp. 13–14
  • Retries occur in 8% of cells overall and 18% on RoboCasa365 atomic tasks, rerunning a VLA policy until the task reports success.With tool calling, every retry is a round trip that returns three camera images.Source: C What the code agent writes · p. 14
  • In 30% of cells, the agent uses NumPy for perception or geometry, including locating objects and estimating grasp orientation from image and point-cloud data.Tool calling instead relies on outputs computed by its tools, such as images, named pixel back-projections, and box statistics.Source: C What the code agent writes · p. 14
  • The agent defines helper functions in 2% of cells and calls previously defined helpers in 7%, including a guarded descent routine reused to grasp without a VLA policy.The descent moves in 8 mm steps and stops when a step fails to reach its target, descent stops, or sideways drift occurs.Source: C What the code agent writes · p. 14
  • The agent looks in 20% of cells, choosing when and which camera to use, although 84% of its cells request an image.Tool calling returns three images after every move whether needed or not.Source: C What the code agent writes · p. 14
  • Single-step cells account for 35% of cells overall and 54% on RoboTwin 2.0, where the agent often checks one VLA-policy chunk at a time.Code saves the fewest LLM calls on RoboTwin 2.0.Source: C What the code agent writes · p. 14

D Instances only one agent solved

PyRUA-Lean solved 96 instances that tool calling did not, while tool calling alone solved 36. The traces show different recovery patterns after failures, including local scripted recovery and additional point-cloud computation.

D Instances only one agent solved

  • 96 instances were solved only by PyRUA-Lean, compared with 36 solved only by tool calling.In 52 of the 96, tool calling exhausted its LLM-call budget; in the other 44, it ended the episode and reported failure.Source: D Instances only one agent solved · p. 15
  • In about a quarter of the 44 episodes tool calling ended, the code agent called the VLA policy and succeeded where tool calling had not.In most of the remaining episodes, both agents encountered the same setback, commonly a missed VLA grasp or a fallen object.Source: D Instances only one agent solved · p. 15
  • After shared setbacks, the code agent sometimes closed the gripper itself, scripted and checked recovery steps, or computed from camera point clouds.Tool calling instead often asked the policy again or stopped with most of its call budget unused.Source: D Instances only one agent solved · p. 15

E Cost accounting

The cost accounting includes every episode call and uses GPT-6 Astra list prices, while LLM calls—not elapsed time—are the efficiency measure. Across all episodes, tool calling used more input tokens on each benchmark, though atomic RoboCasa365 costs were similar.

E Cost accounting

  • Every episode call was billed at GPT-6 Astra’s September 2026 list prices, and post-success calls were included in whole-episode metrics.The accounting used $10 per million input tokens, $1 for cached input, $12.50 for cache writes, and $50 for output; no call exceeded 272K prompt tokens.Source: E Cost accounting · p. 15
  • The comparison counts LLM calls rather than episode time because shared-gateway call latency varied too much for a time comparison.Source: E Cost accounting · p. 15
  • 3.1×, 1.8×, and 1.6× are the tool-calling-to-PyRUA-Lean input-token ratios across all episodes on LIBERO-PRO, RoboTwin 2.0, and RoboCasa365 composite tasks.For RoboCasa365 atomic tasks, the estimated inference-cost ratio was 1.1×; failed episodes tended to approach the 40-call limit for both agents, while PyRUA-Lean used more input tokens per call.Source: E Cost accounting · p. 16
  • Median first prompts were 20k versus 22k tokens on LIBERO-PRO, 15k versus 21k on RoboTwin 2.0, and 17k versus 24k on RoboCasa365 for tool calling versus code.Output tokens, including reasoning, were at most 17% of either agent’s bill on any benchmark.Source: E Cost accounting · p. 16

F Benchmarks and task instances

The main comparison covered 700 task instances and 1,400 paired episodes across LIBERO-PRO, RoboTwin 2.0, and RoboCasa365, with specified seeds and a few rerun or substitution rules.

F Benchmarks and task instances

G Ablation settings

The ablations reuse main-comparison task instances and seeds; one LIBERO-PRO setting tests tool calling with camera images requested on demand. Two instance traces illustrate cases uniquely solved by code.

G Ablation settings

  • Ablation settings use every main-comparison task instance at its original seeds, with each group’s first setting reusing the main-comparison episodes.Source: G Ablation settings · p. 17
  • On 200 LIBERO-PRO instances, the image-on-demand baseline returned tool text only unless the agent called the camera tool, while code episodes were reused from the main comparison.The baseline camera tool returned three views per call; Table 8’s means cover the 131 instances all three settings solved.Source: G Ablation settings · p. 17, Table 8 · p. 17
  • In the stove-and-moka-pot trace, code recovered a fallen pot by estimating its lean from lid points, turning it upright, and placing its base over the burner.Tool calling tried to grasp the fallen pot with Pi0 four times and ended the episode.Source: Figure 8 · p. 18
  • In the cabinet-door trace, code used three handle positions to estimate the hinge-centered arc, then pulled along it with checks for short moves or grip slips.Tool calling was still pulling when its 40 calls ran out; a VLA call later completed the code agent’s task.Source: Figure 9 · p. 19

Related papers

Loading 2610.01939v1…