PaperlayerView exact revision on arXiv

Source-linked AI summary

RobotUse: Allocating Computation, Context, and Decisions

Junhoo Lee, Injun Baek, Seungyeon Kim, Suhyun Jeon, Minkyu Kim, Baekseung Kim, Nojun Kwak

arXiv:2610.04929v1cs.RO

TL;DR

Robot agents need to connect intended actions to outcomes and retain context for revision, while existing interfaces can hide task-relevant choices or leave agents to manage execution code and its growing history. RobotUse exposes visual targets, positions, orientations, and adjustments to the agent, delegates geometric realization to the backend, retains detailed subgoal interactions locally, and uses returned reports and a persistent playbook to guide decisions. 45.00% success on RoboLab (54/120 episodes), exceeding CaP-X by 6.67 percentage points.

  • Problem

    Robot agents need to connect intended actions to outcomes and retain context for revision, while existing interfaces can hide task-relevant choices or leave agents to manage execution code and its growing history.

    Source: §1 · p. 1

  • Method

    RobotUse exposes visual targets, positions, orientations, and adjustments to the agent, delegates geometric realization to the backend, retains detailed subgoal interactions locally, and uses returned reports and a persistent playbook to guide decisions.

    Source: §1 · p. 2, §4.1 · p. 4, §4.3 · p. 6

  • Results

    45.00% success on RoboLab (54/120 episodes), exceeding CaP-X by 6.67 percentage points.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    Success increases from 19/40 to 22/40 across revisions.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Robot agents must connect their intended actions to observed outcomes while retaining the context needed to revise their choices over repeated attempts. Existing interfaces often leave these choices inside predefined tools or require agents to manage detailed execution code and its growing history. We introduce RobotUse, a robot agent harness that organizes computation, context, and decisions around specifying and revising physical actions. Agents visually select targets and poses, while the backend handles geometry, motion planning, and control. Subagents retain detailed interactions within each subgoal and return the information needed for subsequent decisions. Continual harnessing lets agents learn from execution by updating a persistent playbook. On RoboLab, RobotUse achieves 45% task success, outperforming CaP-X by 6.7 percentage points while maintaining compact decision contexts and reducing reliance on predefined action abstractions. Furthermore, we show that RobotUse learns from real-world execution despite imperfect feedback and transfers what it learns to subsequent tasks. Project page is available at https://robotuse-team.github.io/.

1 INTRODUCTION

Robot agents must connect physical intentions to outcomes and retain enough context to revise actions, but existing interfaces either hide task-relevant choices in skills or burden agents with execution code. RobotUse allocates visual action choices to agents, detailed execution to subagents and the backend, and cross-episode learning to a persistent playbook.

1 INTRODUCTION

  • Physical actions require scene-dependent choices about position, orientation, and approach, while code-based policies expose execution logic and high-level skills can conceal choices needed for adjustment.Accumulating abstractions can make the connection between agent decisions and physical outcomes harder to inspect and repair.Source: §1 · p. 1
  • RobotUse keeps targets, poses, and adjustments available for agent decisions while delegating geometric realization and control to the backend.Subagents retain detailed interaction histories and return outcomes, current state, and unresolved constraints; a playbook accumulates conditional guidance across episodes.Source: §1 · p. 2
  • Across successive playbook refinements, accumulated experience improves task performance, and the evaluation also examines grasp-tool dependence and task completion on RoboLab.The studies consider inference cost, token usage, and execution time alongside task completion.Source: §1 · p. 2

2 RELATED WORK

Prior work organizes agent reasoning, execution, context, and experience in different ways, including harnesses for long-horizon work and systems for robot manipulation. RobotUse applies these allocation questions to physical action, retaining task-relevant choices at the agent interface while delegating geometric realization.

2 RELATED WORK

  • Agent-harness research spans reasoning with action feedback, recursive handling of long inputs (Zhang et al., 2026), subagent delegation, and retaining or refining experience (Karten et al., 2026a; Karten et al., 2026b).RobotUse brings this allocation of computation and context to physical action, separating local interaction histories while preserving information for subsequent decisions.Source: §2 · p. 2
  • Robot-manipulation work includes vision-language-action models, skill selection and feedback, visual action refinement, and generated or revised executable behavior (Lu et al., 2026; Chen et al., 2026).RobotUse retains concrete physical choices at the agent interface while delegating their geometric realization to the backend.Source: §2 · p. 3

3 PROBLEM FORMULATION

A robot harness determines the commands available to an agent, how the backend realizes them, and what observations and history inform later decisions. The formulation asks how to delegate execution while keeping task-relevant choices accessible for revision and managing the context needed to connect choices with outcomes.

3 PROBLEM FORMULATION

  • A robot harness organizes available commands, their backend realization, and the observations and history supplied to the agent, thereby allocating decisions between agent and backend.Commands may expose skill invocations, executable code, individual movements, or target poses even when robot capabilities are unchanged.Source: §3 · p. 3
  • High-level skills can determine task-relevant choices internally, while exposing perception and geometry code can still leave choices to an internal scoring rule.The harness must make intended choices available for selection and revision while deciding which computations to delegate.Source: §3 · p. 3, Figure 2 · p. 4
  • Because action outcomes depend on geometric and contact conditions, feedback must connect outcomes to the choices that need revision, such as target selection, approach direction, or contact location.Revision also needs current observations, prior choices, results, and attempted adjustments, with detailed local attempts retained separately from what passes to the next task-level decision.Source: §3 · pp. 3–4
  • The paper studies how to organize commands and context so agents can specify and revise task-relevant choices while delegating the computation needed to realize them.Source: §3 · p. 4

4 ROBOTUSE

RobotUse keeps task-relevant physical choices available to the agent while delegating geometric realization, and organizes local execution histories, reports, and persistent guidance around that division. Its four components define the decision interface, subgoal handoff, closed-loop revision, and continual playbook updates.

4.1 DESIGN PRINCIPLE

  • RobotUse exposes grasp candidates and pose previews for agent selection and revision, while the backend performs geometric conversion, motion planning, and control.The playbook guides delegation and local execution with conditional guidance across episodes.Source: §4.1 · p. 4

4.2 HANDOFF UNIT

  • A handoff gives a subagent a language-defined subgoal, constraints, and relevant context; its report returns the outcome, reason, current image, revisions, and unresolved issues.Detailed interaction histories remain local to the subagent, while the main agent accumulates handoffs and reports to organize progress.Source: §4.2 · p. 5

4.3 CLOSED-LOOP INTERACTION

  • Within a handoff, the subagent observes, specifies a spatial choice, inspects feedback, and revises or executes it using pointing, candidate selection, or pose editing.Previews and feasibility diagnostics support inspection before motion; execution feedback supports later revision, and local context retains observations, adjustments, and feedback.Source: §4.3 · p. 6

4.4 CONTINUAL HARNESS

  • An outer-loop refiner updates a persistent playbook from main-agent trajectories, recording conditions for effective choices, recovery strategies, and delegation context.New evidence can qualify or revise guidance; the refiner leaves model parameters and backend code unchanged.Source: §4.4 · p. 6

5 EVALUATION

The evaluation tests RobotUse on RoboLab manipulation tasks, examining task performance, action flexibility, context handoff, and learning across harness revisions. Results show higher overall success than the compared baselines, lower context costs with subgoal delegation, and improved performance across revisions.

5.1 EXPERIMENTAL SETUP

  • The RoboLab evaluation covers 40 tabletop tasks (21 Simple, 13 Moderate, 6 Complex), measures full-task success with the final task verifier, and examines performance across revisions after execution experience.Language-agent results use three trials per task (120 episodes per method).Source: §5.1 · p. 7

5.2 TASK PERFORMANCE

  • 45.00% success (54/120 episodes) puts RobotUse 6.67 percentage points above CaP-X; it leads overall, Simple, and Complex tasks, while Cosmos 3 leads Moderate tasks.RobotUse also exceeds Open Robot Skill by 25.00 points, mini-SWE (20.83%), π0.5 (30.75%), and Cosmos 3 (42.25%).Source: §5.2 · p. 7
  • Without the grasp tool, RobotUse retains 75% of its original success rate, falling from 50.00% to 37.50% across 40 tasks.CaP-X falls from 45.00% to 17.50%; Figure 5 compares success with and without the tool.Source: §5.2 · p. 8, Figure 5 · p. 8
  • In the butter-box example, the agent corrects a measured 27 mm offset with a 20 mm shift and raises placement after three rejected paths, retaining the selected target.On Stack3RubiksCube, it lowers the gripper by 85 mm and rotates it by 38° after inspecting the wrist image; CaP-X instead uses a fixed top-down orientation in its remaining programs.Source: §5.2 · p. 8

5.3 CONTEXT HARNESSING

  • Across 120 episodes, single-agent execution raises mean input per call 4.2×, episode peak input 3.5×, total tokens 4.9×, and API cost 2.4×; success falls from 45.0% to 40.8%.Figure 6 compares RobotUse with a single agent using the same tools and role instructions in one continuous conversation.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4 LEARNING FROM EXPERIENCE

  • Success rises from 19/40 to 22/40 across four harness revisions as the playbook incorporates lessons about target tracking, robot-frame directions, and re-observing the original stack.Round 3 evaluates one frozen policy in 40 new episodes; Figure 8 illustrates the resulting guidance and choices.Source: §5.4 · pp. 8–9, Figure 8 · p. 9, Figure 7 · p. 9

6 CONCLUSION

RobotUse organizes computation, context, and decisions around specifying and revising physical actions. Its RoboLab results, tool-removal evaluation, and simulation and real-robot studies illustrate task performance, context handoff, and learning through playbook refinement.

6 CONCLUSION

  • RobotUse retains physical choices for revision, keeps execution details within subgoals, and carries experience into later attempts through a persistent playbook.On RoboLab it achieves 45% task success and supports manipulation without the grasp tool; the real-robot evaluation uses a Franka Panda and freezes the playbook after its first learning success.Source: §6 · p. 9, Figure 9 · p. 9

AI USE STATEMENT

The authors report using LLM-based assistants for manuscript editing and experimental-code implementation support, with research ideas, designs, and analyses directed by the authors.

AI USE STATEMENT

  • Claude and ChatGPT assisted with manuscript clarity and experiment-code writing and debugging; the authors say they reviewed the AI-assisted content and take responsibility for the paper.The statement says the tools did not conceive or direct the research ideas, experimental designs, or analyses.Source: AI USE STATEMENT · p. 10

A.1 EVALUATION PROTOCOLS

The appendix specifies RoboLab evaluation conditions, resource accounting, harness-revision protocol details, and the grasp-tool comparison setup.

A.1 EVALUATION PROTOCOLS

  • The main RoboLab comparison evaluates 40 tasks across three runs, one episode per task per run (120 episodes per method); full-task success requires a successful final verifier.Seed 0 uses fixed initial placements; seeds 1 and 2 randomize dynamic-object positions independently within ±0.10 m in X and Y.Source: §A.1 · p. 13

A.2 HARNESS REVISION PROTOCOL

  • Reported resource measures average API cost, turns, tokens, and process time over episodes; effective tokens exclude cached input, and local perception-model and GPU costs are excluded.The revision sequence uses a 60-minute episode limit, with round 3 evaluating a single frozen policy; evaluation costs exclude development probes and playbook updates.Source: §A.1 · p. 13, §A.2 · p. 13, Table 2 · p. 14

A.3 GRASP-TOOL COMPARISON

  • The grasp-tool comparison uses seed 1 and randomized initial placements; without the tool, RobotUse specifies a click location and relative approach height, then adjusts position and rotation.The appendix reports a smaller observed success reduction for RobotUse in these completed runs.Source: §A.3 · p. 14

A.4 CONTEXT HANDOFF ABLATION

The context-handoff ablation compares RobotUse with an otherwise matched single-agent condition; separate accounting specifies how main-agent requests and input context are measured.

A.4 CONTEXT HANDOFF ABLATION

  • The single-agent comparison held the model, tools, role instructions, playbook, turn budgets, and backend constant while assigning subgoal work to one conversation across 120 episodes.The evaluation covered 40 tasks across three seeds.Source: §A.4 · p. 14
  • RobotUse used 596.8k tokens and $0.511 per episode versus 2,909k and $1.221 for single-agent execution, with 54/120 versus 49/120 successes.The conditions averaged 41.6 and 41.1 model calls per episode, respectively.Source: §A.4 · p. 14

A.5 MAIN-AGENT TURN AND CONTEXT ACCOUNTING

  • Main-agent turns count unique role-prime input events; started decisions with unanswered requests count as turns, but HTTP retries do not.Source: §A.5 · p. 14
  • Main-agent context averages and peaks use provider prompt tokens, excluding completion and subagent calls; CaP-X's seed-0 comparison counted 259 code requests with complete input-token usage.The cross-method main-agent comparison covers seed 0.Source: §A.5 · p. 14

A.6 CONTEXT AND LEARNING RESULTS

The supplied results cover task-attribute success, main-agent interaction and context, resource comparisons, and the scope of context and playbook-revision evaluations.

A.6 CONTEXT AND LEARNING RESULTS

  • The context handoff table covers 120 episodes and reports input per call, peak input, and total tokens; the accompanying description says it provides the numerical context comparison.Source: §A.6 · p. 14, Table 3 · p. 15
  • The revision records include 40 seed-0 episodes per round, evaluation usage, and representative playbook additions, removals, and modifications.Source: Table 4 · p. 15, Table 5 · p. 15, Table 6 · p. 16

A.7 ADDITIONAL RESOURCE AND DIFFICULTY BREAKDOWNS

  • RobotUse has the highest or tied-highest success among language/code agents on nine of ten attributes; reorientation is the exception and contains two tasks, while Cosmos 3 leads on relational attributes, semantics, and affordance.Source: §A.7 · p. 15
  • RobotUse uses 10.62 main requests per episode versus CaP-X's 6.48, with average input contexts of 9.12k versus 7.37k tokens per request and peaks of 12.58k versus 11.90k.The comparison covers the main agent and code agent on the same 40 seed-0 tasks; CaP-X's VDM and Molmo calls are excluded.Source: §A.7 · p. 15
  • RobotUse has higher task completion with higher reported API cost than CaP-X and lower reported cost than Open Robot Skill.The figure averages API cost over 120 episodes per method and excludes local perception and GPU costs.Source: Figure 10 · p. 16

B ROBOTUSE IMPLEMENTATION

RobotUse keeps physical choices available to the agent while delegating their geometric realization and execution to backend tools, and uses subgoal-local contexts and versioned playbook guidance.

B ROBOTUSE IMPLEMENTATION

  • The tool interface registers selected targets against the current observation and uses references for later tools; state checks prevent stale candidates from being treated as fresh executable selections.Source: B ROBOTUSE IMPLEMENTATION · pp. 16–17
  • Removing the grasp tool decreases success by 12.50 percentage points for RobotUse and 27.50 points for CaP-X; RobotUse specifies no-grasp motions through clicked locations, height, and pose adjustments.Each condition contains 40 seed-1 tasks.Source: Table 10 · p. 17
  • Pose inspection combines measured points and gripper views; edits specify translations in robot-base XYZ and rotations about the gripper contact center, while the backend resolves poses and plans motion.Placement keeps the gripper closed, with release requested separately after observation; the task verifier determines completion.Source: B ROBOTUSE IMPLEMENTATION · p. 17
  • Each subagent accumulates its own tool interactions, while the parent receives the selection, status, reason, and relevant feedback rather than the full child transcript.Source: B ROBOTUSE IMPLEMENTATION · p. 17
  • The playbook is versioned Markdown with common and role-specific guidance; episode records preserve its hash, selected version, and configuration across revisions.Source: B ROBOTUSE IMPLEMENTATION · p. 18

Related papers

Loading 2610.04929v1…