PaperlayerView exact revision on arXiv

Source-linked AI summary

Democratizing Robot-Use Agents: A Unified Harness with Open Models

Jiarui Yuan, Yuxin Zuo, Xiaoyu Huang, Wenbo Zhang, Kwanyoung Park, Junlin Yang, Tianxiang Zhang, Ruoqu Chen, Yutao Ouyang, Bingxiang He, Xingtai Lv, Hongjin Wu, Jiajun Liu, Jiahao Zhao, Tianwei Luo, Haozhan Li, Cheng Qian, Jiaxin Ge, Huan-ang Gao, Kaisen Yang, Jiaqi Deng, Qixin Xu, Siqiao Huang, Bowen Zhou, Mengdi Xu, Zhiyuan Liu, Pieter Abbeel, Ning Ding

arXiv:2611.01560v1cs.RO

TL;DR

Research on robot-use agents has been limited by proprietary models, restricting access to further training and model-level optimization. Uni-robot-agent provides a unified harness for open-weight models; without privileged inputs, Qwen3.8-Flash-Next achieves a 31.39 RoboDojo Score, exceeding the published GPT-6 Astra-RoboProbe result of 28.97.

  • Problem

    Proprietary frontier models limit further-training research on robot-use agents, while open-weight models often lack spatial understanding and agentic capabilities.

    Source: §1 · p. 4

  • Method

    Uni-robot-agent provides unified tools and feedback across models, embodiments, tasks, and environments, with efficient execution and modular components.

    Source: §1 · p. 4

  • Results

    Without privileged inputs, Qwen3.8-Flash-Next achieves a RoboDojo Score of 31.39, exceeding GPT-6 Astra’s RoboProbe result of 28.97.

    Source: §1 · p. 4, §8 · p. 21

  • Takeaways & Limitations

    Analysis identifies visual understanding, spatial grounding, geometric coding, and embodied reasoning as key areas for future RUA training.

    Source: §1 · pp. 4–5

    The evaluation uses RGB images, camera calibration, robot proprioception and descriptions, and task instructions, excluding privileged simulator information and external web access.

    Source: §3.1.1 · p. 9

Abstract

from arXiv · show

Robot-use Agents (RUAs) have shown promising zero-shot manipulation capabilities, but reliance on frontier, proprietary models limits research into further training. To democratize RUA research, we introduce uni-robot-agent, a unified harness for competitive open-weight robot use that supports further training. Without privileged inputs, an out-of-the-box Qwen3.8-Flash-Next achieves a RoboDojo Score of 31.39, surpassing the published GPT-6 Astra-RoboProbe result of 28.97 and many VLA/WAM baselines, and performs competitively on 19 real-world tasks. Our analysis across model generations, families, and sizes links effective robot use to four general capabilities, suggesting that future RUA training should develop them jointly.

1. Introduction

The paper introduces uni-robot-agent to make robot-use research with open-weight models more accessible to further training. It reports competitive simulation and real-world performance and identifies four capabilities associated with effective robot use.

1. Introduction

  • Proprietary frontier models limit further-training research on robot-use agents, while API dependence raises interaction costs and restricts model-level optimization.Source: §1 · p. 4
  • Open-weight models often lack spatial understanding and agentic capabilities, motivating a generalizable harness before attributing limitations to models or adding ad-hoc improvements.Prior approaches include privileged simulator information, oracle task-success feedback, or task-specific skills.Source: §1 · p. 4
  • Uni-robot-agent provides unified interfaces and feedback across models, embodiments, tasks, and environments, with efficient execution and replaceable components supporting future training.Source: §1 · p. 4
  • 31.39 RoboDojo Score: without privileged inputs, Qwen3.8-Flash-Next exceeds GPT-6 Astra’s RoboProbe result of 28.97.The paper also reports that this is the first open-weight system to outperform closed-weight foundation models on RoboDojo without privileged information or distillation.Source: §1 · p. 4
  • 49.50 Score across 19 real-world tabletop tasks: Qwen3.8-Flash-Next outperforms Dream-1’s 35.50 and matches fine-tuned π0.5’s 47.75.Source: §1 · p. 4
  • Trajectory analysis identifies visual understanding, spatial grounding, geometric coding, and embodied reasoning as capabilities for future RUA training.Source: §1 · pp. 4–5

2. uni-robot-agent: A Unified and Modular Harness

Uni-robot-agent connects agents and robot environments through shared tools, adapters, and replaceable components. Its design combines reusable interfaces, overlapping reasoning and execution, realistic inputs, and practical safeguards for reliable interaction.

2. uni-robot-agent: A Unified and Modular Harness

  • The harness connects diverse agents, environments, and embodiments through shared tools, supporting interaction based on information available in real-world deployments.Source: §2 · p. 5, §2.1 · p. 5

2.1. System Overview

  • Four principles organize the system: unified operations and feedback, efficient execution, realistic inputs, and modular runtimes, tools, and perception models.Interaction records can be combined for subsequent training; explicit interfaces allow components to be replaced independently.Source: §2.1 · p. 6

2.2. Unified Interaction Interface

  • Shared tools use common arguments, units, and feedback formats, while environment adapters translate operations into native commands and return state.This lets N tools share logic across M adapters rather than requiring N×M tool–environment implementations.Source: §2.2 · p. 6, Figure 3 · p. 6

2.3. Efficient Execution

  • Agent reasoning overlaps tool execution, while calls that share an exclusive resource remain ordered and independent calls can run in parallel.The pipeline allows successive calls to be submitted during execution; queued calls not yet started can be cancelled with flush_queue.Source: §2.3 · p. 7, Figure 4 · p. 7

2.4. Interaction Without Privileged Inputs

  • Physical transfer motivates testing whether agents can infer spatial relationships and assess progress from limited observations, without relying on simulator depth or task-verifier feedback.Source: §2.1 · p. 6, §2.4 · p. 7

2.5. Key Design Choices

  • Design choices bind images, calibration, and robot state to immutable snapshots, reject duplicate requests, and allow coordinated arm and gripper movement.Source: §2.5 · p. 8, Figure 5 · p. 8
  • Motion adapts to measured progress and returns control when stalled or limited; numerical motion feedback and outcome images help diagnose manipulation results.A sliding window replaces older images with text placeholders while retaining recent images and surrounding text; batched eviction supports a stable prompt prefix.Source: §2.5 · pp. 8–9

3. Can Open Models Use Robots in Simulation and the Real World?

The paper evaluates whether open-weight robot-use agents can compete in simulation and on physical robots using a shared harness and realistic inputs. Qwen3.8-Flash-Next is competitive on RoboDojo and matches fine-tuned π0.5 on the real-world evaluation.

3. Can Open Models Use Robots in Simulation and the Real World?

  • Open-weight agents achieve competitive performance in both simulation and physical-robot evaluation, and the harness also benefits closed-weight models.GPT-6 Astra with uni-robot-agent performs best in the RoboDojo comparisons.Source: §3 · p. 9, §3.1.2 · p. 10

3.1.1. Setup

  • The RoboDojo evaluation covers 42 tasks across five dimensions, averages six episodes per task, and excludes privileged inputs such as simulator depth and task-verifier feedback.The primary results are averaged over three seeds and two layouts; the 12 generalization tasks balance standard and randomized layouts.Source: §3.1.1 · p. 9, Table 1 · p. 10

3.1.2. Results and Analysis

  • 25.00% SR and 31.39 Score let Qwen3.8-Flash-Next with uni-robot-agent surpass the listed VLA/WAM baselines and GPT-6 Astra with RoboProbe (22.48% SR, 28.97 Score).DeepSeek-V4.1-Flash also exceeds the VLA/WAM baselines; switching Qwen to the strongest alternative harness, GAP, loses 8.89% SR and 10.38 Score.Source: Table 1 · p. 10, §3.1.2 · p. 10

3.2.1. Setup

  • The real-world benchmark spans spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks, with each dimension equally weighted.Qwen3.8-Flash-Next and GPT-6 Astra are evaluated zero-shot alongside Dream-1; π0.5 is fine-tuned on 10 hours of demonstrations per task.Source: §3.2.1 · p. 11

3.2.2. Results and Analysis

  • Qwen3.8-Flash-Next with uni-robot-agent matches fine-tuned π0.5 and outperforms zero-shot Dream-1 in the real-world evaluation.The agents are evaluated on 19 tasks across five dimensions using a dual-arm platform; the RUAs additionally receive depth while the other methods use RGB only.Source: §3.2.2 · p. 11, §3.2.1 · p. 11

4. What Capabilities Enable Robot-Use Agents?

The paper compares robot-use performance across model generations, families, and sizes, then examines failures and general benchmarks to identify capabilities associated with success. Its analyses point to visual understanding, spatial grounding, geometric coding, and embodied reasoning rather than model size or recency alone.

4.2. Performance Across Generations, Families, and Model Sizes

  • 19.17% SR and 24.04 Score for GPT-5.6 Sol show robot-use performance predates the latest generation, while GPT-6.1 Sol reaches 37.50% SR and 45.63 Score.Only the evaluated GPT models show successful robot use before the latest generation; Qwen task success appears in generation 3.8, and GLM-5.3 Flash has 0% SR.Source: §4.2 · p. 12, Figure 6 · p. 12
  • Qwen3.8-27B succeeds where larger, earlier Qwen models fail, suggesting that training-distribution differences matter more than model size alone.Earlier Qwen3.5–3.7 Flash, Qwen3.6–3.7 Plus, and Qwen3.5-122B record zero SR; Qwen3.8-27B and Qwen3.8-Flash-Next reach 8.06% and 25.00%, respectively.Source: §4.2 · pp. 12–13

4.3. Four Major Failure Modes

  • Trajectory analysis identifies four failure types: visual misunderstanding, inconsistent spatial grounding, errors in geometric computation, and physically inappropriate embodied reasoning.The analysis evaluates additional models on RoboDojo’s 42 tasks, using one seed and one layout for the additional evaluations.Source: §4.1 · p. 11, §4.3 · p. 13
  • Visual errors occur in 12.2–23.8% of earlier Qwen trials, while geometry-calculation errors affect 47.3–81.8% of earlier Flash and Plus calculations.Spatial-grounding failures remain substantial even for GPT-6.1 Sol, whose rate is 31.5%; embodied-reasoning analysis is limited to grasp poses, and GPT-6 Astra also struggles with bimanual coordination.Source: §4.3 · p. 14

4. What Capabilities Enable Robot-Use Agents?

  • The authors conclude that effective robot use draws on a combination of general capabilities and argue that future RUA training should develop them jointly.The comparisons cover model generations, families, and sizes to investigate which capabilities underlie robot use.Source: §4 · p. 11, §4.3 · p. 14

4.4. Relationship to General Model Capabilities

  • Positive correlations link RoboDojo performance with all four corresponding foundation-model benchmarks, strongest for embodied reasoning and visual understanding and weaker for spatial grounding.The authors note that ScreenSpot-Pro measures 2D screen grounding, unlike the 3D grounding needed for robotic manipulation.Source: §4.4 · p. 15, Figure 8 · p. 15

5. Analysis on Privileged Information

The analysis tests whether depth and task-specific skills are necessary for robot-use performance. Agents retain much of their performance without depth when camera parameters are available, and disabling the skill library yields a small performance gap.

5.1. RQ1: How Does Depth Affect Robot Use?

  • Astra retains 98.6% and Qwen 87.5% of their depth-enabled scores when depth is removed but camera parameters remain available.Scores change from 55.83 to 55.04 for Astra and from 35.89 to 31.39 for Qwen; agents can triangulate corresponding points across views.Source: §5.1 · p. 16, Figure 9 · p. 16
  • Without camera parameters as well as depth, Qwen’s score drops to 9.13, while Astra’s decreases to 31.29.The authors conclude that privileged depth is not necessary for strong robot-use performance when camera parameters are available.Source: §5.1 · p. 16

5.2. RQ2: Is skill library important for uni-robot-agent?

  • Without the skill library and skill writing, Qwen reaches 30.31 Score and 24.58% SR, retaining 96.6% of the skill-enabled score and a 0.42-point SR gap.Task instructions, environment descriptions, and standard robot tools remain available in the no-skill configuration.Source: §5.2 · p. 17, Figure 10 · p. 17

6. Analysis on Cost and Speed

The paper examines model resource use and whether pipelining reduces execution time. Qwen offers a lower-cost performance trade-off than Astra, while pipelined execution shortens episodes and overlaps model requests with physical actions.

6.1. Token Usage and API Cost

  • Qwen reaches 31.39 Score at $1.09 per episode, while Astra achieves 51.90 Score at $21.29 per episode.Astra’s score is higher, but its per-episode cost is nearly 20 times Qwen’s.Source: §6.1 · p. 18, Figure 11 · p. 18
  • To reach a score of 25, Astra requires 7.4M tokens and 42.6 minutes per episode, compared with 23.5M tokens and 95.5 minutes for Qwen and 49.9M tokens and 99.2 minutes for DeepSeek.The progress curves track each episode’s latest observed score over token usage and time.Source: §6.1 · p. 18, Figure 12 · p. 18

6.2. Execution Efficiency

  • Pipelining shortens episode duration on all five tested tasks by 4.8%–73.7%, with model-request overlap covering 84.5%–93.8% of physical execution time.Overlap is zero in all five serial episodes; each task and configuration is evaluated once, excluding simulator initialization from duration.Source: §6.2 · p. 19, Figure 13 · p. 19, Figure 14 · p. 20

7. Related Work

Prior work explores learned action policies, tool-mediated robot control, and harnesses for robot-use agents. Uni-robot-agent builds on these efforts with reusable components that separate robot tools from model runtimes and environment adapters.

Related Work

  • VLA and world-action models broaden the environments and behaviors learned policies can handle, while reliable use of past observations and long-horizon failure recovery remain open concerns.The cited examples include 𝜋0.5 (Physical Intelligence et al., 2025), DreamZero (Ye et al., 2026), and Cosmos Policy (Kim et al., 2026).Source: §7 · p. 20
  • Tool-mediated control combines observation, planning, and feedback, but robotics must coordinate these while the physical world changes during execution.Prior work connects environment feedback to replanning (Huang et al., 2022) and spatial representations to physical motion (Huang et al., 2023).Source: §7 · p. 20
  • Robot-use harnesses provide interfaces for general-purpose models, and uni-robot-agent separates shared tools, model runtimes, and environment adapters while allowing reasoning to overlap physical execution.Earlier examples include RoboDawn and Robo-Harness K1 (Li et al., 2026; Zhang et al., 2026b), OpenRUA (Chu et al., 2026), Harness VLA and GAP (Zhang et al., 2026c), and ORA.Source: §7 · pp. 20–21

8. Conclusion

Uni-robot-agent is a unified, modular harness for robot-use agents. Its evaluations show that an open-weight model can exceed the published Astra–RoboProbe RoboDojo result, while precise manipulation and reliable long-horizon execution remain challenges.

8. Conclusion

  • 31.39 is Qwen3.8-Flash-Next’s RoboDojo Score, exceeding the published Astra–RoboProbe result of 28.97.The authors also report strong GPT-6 Astra performance in simulation and real-world manipulation.Source: §8 · p. 21
  • Precise manipulation and reliable long-horizon execution remain challenges in the evaluations.Source: §8 · p. 21
  • Shared interfaces, replaceable components, and structured execution feedback provide a foundation for collecting interaction data, developing training methods, and evaluating their effects.The authors identify local deployment and adaptation through training on robot experience as avenues for further investigation.Source: §8 · p. 21

A. Experimental Setup and Reproducibility Settings

The evaluation covers simulation and real-world manipulation, with specified baselines, hardware, and task catalogues. The Real Eval catalogue distinguishes test variations from training demonstrations, which uni-robot-agent does not use.

A. Experimental Setup and Reproducibility Settings

  • RoboDojo provides 42 Isaac Sim tasks, and the simulation comparison uses Qwen3.8-Flash-Next as planner alongside specified harnesses and published VLA/WAM results.The experimental settings also specify model-dependent reasoning effort.Source: A. Experimental Setup and… · p. 25

B. Real Eval Benchmark Details

  • Real Eval contains 19 entries across five capability dimensions, allocating 20 points to each dimension and 100 points total.Entries distinguish object, language, or task-structure variations even when they share a manipulation primitive.Source: B. Real Eval Benchmark Details · p. 25

B.1. Hardware and Baselines

  • Real-world comparisons use an ARX AC one dual-arm platform, with GPT-6 Astra and Qwen3.8-Flash-Next compared against 𝜋0.5 and Dream-1.The harness additionally uses D405 depth, while 𝜋0.5 and Dream-1 consume RGB images only.Source: §B.1 · p. 25

B.2. Training Task Catalogue and Collection Protocol

  • The training catalogue contains 30 templates and 1,550 trajectories, but uni-robot-agent uses none of these demonstrations and is evaluated zero-shot.The collection varies objects, placements, materials, colors, and arrangements; the test catalogue defines new objects, instructions, and action combinations.Source: §B.2 · pp. 25–26

B.3. Evaluation Dimensions

Real Eval tests five kinds of capability variation, while the accompanying sections describe its scoring, cost estimation, monitored motion, and agent-written spatial grounding. The supplied results passage identifies per-task scores but does not give their values.

B.3. Evaluation Dimensions

  • The five Real Eval dimensions test spatial understanding, object generalization, instruction following, compositional generalization, and transfer to unseen tasks.Examples include changing objects or directions, recombining familiar actions and constraints, and reversing or extending training operations.Source: §B.3 · p. 26

B.4. Per-Task Results

  • Table 4 reports individual task scores underlying Table 2, with capability subtotals out of 20 and a total out of 100.Source: §B.4 · p. 28, Table 4 · p. 29

B.5. Scoring Protocol

  • Real Eval is a rubric-based 0–100 capability score, not the fraction of episodes fully completed; successful grasping can earn partial credit despite failed insertion.Trials share task-level instructions and scoring criteria, without task-specific demonstrations, subtask plans, or hints.Source: §B.5 · p. 28

C. API Cost Estimation

  • API episode cost is estimated from recorded usage and public rates, with cached and uncached input billed separately and reasoning tokens counted once.Recorded retries are included; requests without recoverable usage remain unestimated, and DeepSeek rates depend on processing time.Source: C. API Cost Estimation · pp. 28–29

D. Monitored Motion Execution

  • Monitored motion advances through bounded waypoints and checks position and orientation errors, stopping on arrival, divergence, insufficient progress, abnormal motion, or execution limits.Only accepted steps count toward the step budget or update progress history.Source: D. Monitored Motion Execution · pp. 29, 31
  • After specified planning rejections, eligible backends can search once for an alternative target within the original tolerances; feedback reports the ending pose, errors, and stop reason.Reaching a pose alone does not establish that the task objective, such as grasping or placement, succeeded.Source: D. Monitored Motion Execution · p. 31

E. Spatial Grounding Through Agent-Written Code

  • In two successful no-depth episodes, the agent wrote geometry code using RGB observations and camera intrinsics and extrinsics to estimate spatial relationships.The examples use feature matching across views and testing geometric assumptions against image evidence; Figure 15 illustrates triangulation from matching pixels and reprojection errors.Source: E. Spatial Grounding Through… · p. 32, Figure 15 · p. 32

E.1. Recovering Tube Positions Across Views

Across two manipulation examples, agents use code to infer geometry from RGB observations, test scene hypotheses, and guide successful actions. Recorded execution timelines also show concrete planning calculations generated while physical tools are running.

E.1. Recovering Tube Positions Across Views

  • All three tubes end in the rack after the agent reconstructs cap positions from two views, estimates tube axes, and uses observations and action feedback to guide insertion.The selected cap correspondences reproject within 0.46 and 1.94 pixels of the points in the two views.Source: §E.1 · p. 32

E.2. Exploring Geometry Before Moving a Cup

  • The agent completes the cover-and-uncover sequence after testing candidate table heights and comparing inferred cup dimensions before acting.It projects candidate object shapes into images and refines scene estimates; the example uses geometric hypotheses rather than treating one image estimate as fixed.Source: §E.2 · p. 33, Figure 16 · p. 33

F. Planning During Physical Execution

  • Recorded trajectories show model requests preparing concrete calculations while physical tools run, linking concurrent preparation to later actions and feedback.The figures pair observations with execution timelines and enlarge an interval containing a calculation; their bars mark model requests, not internal reasoning time.Source: F. Planning During Physical Execution · pp. 33–34, Figure 17 · p. 34

F.1. From Regrasping to Supported Sliding

In physical execution, advance preparation can overlap with tool activity, but subsequent robot feedback still shapes which actions are taken. The examples show a revised supported-sliding strategy and prepared orientation commands subject to execution monitoring.

F.1. From Regrasping to Supported Sliding

  • The blocks finish aligned after the agent abandons a failed grasp-retention attempt and switches to supported sliding, revising its preparation when physical evidence contradicts it.The agent makes small rotations and translations while maintaining support, then inspects the scene and adjusts contact.Source: §F.1 · pp. 34–35, Figure 18 · p. 35

F.2. Preparing Reorientation During Transport

  • During digit transport, the agent prepares candidate rotations, but a later rotation stops for divergence and prompts a new observation before proceeding.The transport first exhausts its step budget; the sequence illustrates that prepared actions still depend on the state returned by the robot.Source: §F.2 · p. 35
Loading 2611.01560v1…