PaperlayerView exact revision on arXiv

Source-linked AI summary

RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers

Haitong Ma, Chenxiao Gao, Rushi Qiang, Bo Dai, Na Li

arXiv:2609.34210v2cs.RO

TL;DR

Existing robotics benchmarks mainly evaluate individual policies or controllers, leaving coding agents’ broader robot-learning engineering capabilities less covered. RLE-BENCH evaluates agents across representative robot-development workflows using an overall index and workflow profiles. Its reported results show GPT-6 Astra and Claude Fable 5.1 leading, while the benchmark’s simulated scope does not establish real-world reliability or safety.

  • Problem

    Existing robotics benchmarks primarily evaluate individual policies or controllers rather than coding agents’ broader capabilities across robotics development.

    Source: §1 · p. 2

  • Method

    RLE-BENCH evaluates coding agents on physics-grounded tasks across representative workflows, aggregating workflow scores into an RLE Index and reporting capability profiles.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    GPT-6 Astra and Claude Fable 5.1 lead the leaderboard, with their advantage most pronounced in Interactive Control and Perception and Estimation.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    RLE-BENCH provides a reproducible instrument for developing agents that build, test, and improve reliable robot-learning systems.

    Source: §5 · p. 9

    Simulation performance does not establish real-world reliability or safety, and the four workflows cover only part of robot-learning engineering.

    Source: §5 · p. 9

Abstract

from arXiv · show

Coding agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics. Existing robotics benchmarks, however, primarily focus on the performance of individual artifacts, such as policies or controllers, offering limited coverage of coding agents' broader engineering capabilities. Real-world robotics extends beyond control: agents must build, integrate, diagnose, and improve heterogeneous artifacts under resource constraints and reason from multimodal feedback. To evaluate these broader capabilities, we introduce RLE-Bench, a benchmark of robot-learning tasks spanning four representative robotics development workflows: interactive control, policy learning, perception and estimation, and mechanical design. We use diverse task-specific metrics to evaluate the artifacts submitted by the coding agents, from the success rate the agents achieved to the policy agents trained, the harness agent built, and the mechanical structures the agent designed. We aggregate these metrics into an overall RLE Index and report workflow-specific capability profiles, enabling systematic comparison of coding agents' capabilities across multiple capability dimensions. Beyond performance ranks, we also conduct in-depth case studies examining agent behavior on representative tasks, highlighting both current capabilities and limitations, and pointing to the opportunities robotics tasks have to offer for future agent training.

1 INTRODUCTION

Robotics requires coding agents to reason about physical consequences and multimodal feedback, while existing benchmarks mainly evaluate individual policies or controllers. RLE-BENCH instead evaluates agents across broader robot-learning workflows and reports overall and workflow-specific capability measures.

1 INTRODUCTION

  • Existing robotics benchmarks focus mainly on policies and controllers, while CaP-X evaluates code-as-policy but not the holistic robot-learning workflow.Robotics engineering also spans mechanical design, calibration, closed-loop control, failure diagnosis, and policy development.Source: §1 · p. 2
  • RLE-BENCH covers four workflows and evaluates executable artifacts under hidden scenes, dynamics, embodiments, random seeds, or held-out tasks.The workflows are Interactive Control, Policy Learning, Perception and Estimation, and Mechanical Design.Source: §1 · p. 2
  • The benchmark reports an equally weighted RLE Index across workflow scores, per-workflow capability profiles, and case studies of agent behavior.It evaluates 11 model–harness combinations and reports performance and cost trade-offs.Source: §1 · p. 2

2 RELATED WORK

Prior benchmarks assess coding and machine-learning engineering through executable tasks, while robotics benchmarks generally score individual policies or controllers. RLE-BENCH extends coding-agent evaluation to the broader robot-development stack.

Software Engineering (SWE) and Machine Learning Engineering (MLE) Benchmarks

  • Coding benchmarks progress from repository patches and interactive computer tasks to machine-learning experiments and Gym-style challenges with online verification.SWE-Bench evaluates patches against functional tests; MLE-Dojo spans more than 200 Kaggle challenges.Source: §2 · p. 3

Robotics Policy Evaluation Benchmarks

  • Robotics policy benchmarks provide simulators, task environments, and evaluation protocols, but primarily assess individual controllers or VLA policies.Examples span manipulation, locomotion, and broader platforms supporting whole-body control.Source: §2 · p. 3

Coding Agents for Robotics

  • Robotics coding-agent work includes controller generation, policy-repository improvement, system reproduction, physical experimentation, and reward-function generation; RLE-BENCH evaluates the broader development stack.Code-as-policy systems include Code as Policies and VoxPoser; CaP-X evaluates code-as-policy agents across abstraction, interaction, and perceptual grounding.Source: §2 · p. 3

3 RLE-BENCH

RLE-BENCH organizes robot-learning engineering tasks into workflows and families, with resource-bounded development followed by verifier-controlled evaluation. Its tasks cover interactive control, policy learning, perception and estimation, and mechanical design.

3 RLE-BENCH

  • The benchmark overview groups 51 tasks into four workflows and nine families, each sharing capability requirements, artifact contracts, and evaluation metrics.Figure 2 presents an overview of the tasks.Source: §3 · p. 3, Table 2 · p. 4, Figure 2 · p. 4

3.1 TASK DESCRIPTION AND EVALUATION SETUP

  • Each task specifies an instruction, development environment, resource budget, artifact contract, evaluation environment, and verifier; the 51 tasks draw on repurposed open-source tasks and hand-designed tasks.Agents build and submit artifacts during development, then the verifier evaluates them; valid scores generally require satisfying the instruction’s artifact contract.Source: §3.1 · pp. 4–5

3.2 A WALKTHROUGH OF THE WORKFLOWS AND TASK FAMILIES

  • Interactive Control tasks require direct world interaction, reusable tools or lessons, or information gathering; Policy Learning tasks train locomotion controllers or resource-constrained VLA pipelines.Family01 tests closed-loop control under three harness levels; Family04 evaluates motion tracking under hidden dynamics, latency, and sensor-noise variations.Source: §3.2 · p. 5
  • Perception tasks test pose estimation under occlusion and code-as-policy pick-and-place; Mechanical Design couples designed hardware with supporting control software.Family06 varies sensors and inference hardware; Families08–09 address mobile-manipulator design and gravity compensation under unseen physical instances, poses, and payloads.Source: §3.2 · pp. 5–6

3.3 ORCHESTRATION AND EVALUATION PROTOCOL

  • After development, artifacts run in a verifier image that keeps hidden seeds, privileged states, and reference assets private and produces the authoritative reward report.The verifier runs artifacts in a sandboxed or constrained subprocess and executes physical rollouts.Source: §3.3 · p. 6

4 EVALUATION RESULTS

The evaluation aggregates task scores into an equally weighted workflow-level RLE Index and reports workflow profiles. GPT-6 Astra and Claude Fable 5.1 lead overall, while case studies reveal model-dependent scaffolding benefits and persistent challenges in policy robustness, perception under occlusion, and mechanical design.

4.1 SETUP AND METRICS

  • The RLE Index averages task scores through families and workflows, giving each workflow equal weight in the overall ranking.Per-workflow profiles show capability and cost trade-offs.Source: §4.1 · p. 6, §4.2 · p. 6, Figure 3 · p. 7

4.2 LEADERBOARD, CAPABILITY PROFILES AND COST

  • GPT-6 Astra and Claude Fable 5.1 lead the leaderboard, with their strongest advantages in Interactive Control and Perception and Estimation.The authors suggest stronger visual grounding contributes substantially to their overall lead.Source: §4.2 · p. 7
  • Model performance is more closely matched in Policy Development than in the benchmark overall.This workflow more closely resembles traditional machine-learning engineering tasks.Source: §4.2 · p. 7
  • Mechanical Design remains difficult across models: performance gaps are smaller, and no model consistently makes sound decisions with delayed or indirect physical consequences.Source: §4.2 · p. 7

4.3 CASE STUDIES AND KEY INSIGHTS

  • Richer robotics scaffolding generally improves scores and reduces costs for most models, but GPT-6-Astra performs strongly with basic support and gains little or sometimes loses performance as support increases.The authors suggest external modules may sometimes interfere with a capable model’s own perception and reasoning.Source: §4.3 · p. 7, Figure 4 · p. 8
  • Development-time evaluation improves submitted policies in 29 of 36 sessions, but robustness-subtask official scores trail development scores by 19 (LIBERO) and 41 (RoboTwin) points.On open-design subtasks, development success predicts the official score within 5 points; agents’ perturbation proxies do not close the robustness gap.Source: §4.3 · p. 7, Figure 5 · p. 8
  • Most models improve perception with RGB-D, but reliable performance under occlusion is limited to flagship models, and stronger methods use depth to judge occlusion masks.GPT-6 Astra trains a small convolutional network as a fallback when depth is unreliable.Source: §4.3 · p. 8, Figure 6 · p. 9
  • All nine models receive zero worst-arm credit for static stability margin despite most passing structural-validity checks.The designs can meet visible objectives while missing coupled physical consequences, including instability under load and collisions during motion.Source: §4.3 · pp. 8–9, Figure 7 · p. 9

5 CONCLUSION

RLE-BENCH is a physics-grounded qualifying exam for coding agents, pairing an overall leaderboard with workflow profiles, hidden evaluation, and resource accounting. The authors identify limits in simulated coverage and workflow breadth, as well as a risk that public release could contaminate future evaluations.

5 CONCLUSION

  • RLE-BENCH combines executable tasks across four representative workflows with an overall leaderboard, capability profiles, trusted hidden evaluation, and explicit resource accounting.The authors hope it will provide a reproducible instrument for developing agents that build, test, and improve robot-learning systems.Source: §5 · p. 9
  • Simulation performance does not establish real-world reliability or safety, and the simulated scenes do not cover all deployment conditions.The authors also note that the four workflows capture only part of robot-learning engineering, which requires broader capabilities.Source: §5 · p. 9
  • The planned public release risks future training-data contamination that could compromise the validity of later evaluations.Source: §5 · p. 10

A COMPLETE TASK CARDS

The task cards define three interactive-control families, specifying their simulated tasks, available tools and observations, evaluation procedures, and resource limits.

Task Family 01: RoboCasa Speed-Run 15 instances

  • RoboCasa Speed-Run tests five kitchen tasks across three harness levels, with evaluation in unseen layouts.The task set includes opening a fridge or cabinet, turning on a burner, and two pick-and-place tasks; higher harness levels add geometry, motion and grasp utilities, or privileged world-state information. Success supplies 80% of reward, while interaction efficiency contributes up to 20%, scaled by success.Source: Task Family 01: RoboCasa Speed-Run 15… · p. 13
  • Speed-Run development is limited to eight hours, one GPU, and 50,000 charged steps, followed by five evaluation trials of up to 1,000 actions each.Resets and executed actions are metered during development; evaluation uses a trusted ledger, and missing, unsealed, or unverifiable ledgers score zero.Source: Task Family 01: RoboCasa Speed-Run 15… · p. 13

Task Family 02: RoboCasa Harness Transfer 15 instances

  • RoboCasa Harness Transfer asks agents to build reusable perception tools, controllers, and a manual for fresh agents attempting a held-out kitchen task.Three composite tasks supply practice, while five new agents independently attempt the held-out task in unseen kitchens; only the reusable harness and manual transfer.Source: Task Family 02: RoboCasa Harness… · p. 14
  • Transfer reward averages five agents, gives simulator-confirmed success a score of 1, and otherwise uses stage records; development is capped at 75,000 charged steps.Missing trials count as zero, and development efficiency adds no reward; each evaluation agent receives one hour and 5,000 steps.Source: Task Family 02: RoboCasa Harness… · p. 14

Task Family 03: Tabletop Physical Reasoning 5 instances

  • Tabletop Physical Reasoning spans tower building, cantilevering, hidden-mass identification, cube solving, and ballast localization through interaction with physical properties.The tasks vary in information access: for example, masses remain hidden in Balance, while Hidden COM supplies no force, contact, mass, or object-pose readings and gives no correctness feedback.Source: Task Family 03: Tabletop Physical… · p. 15
  • Scores are task-specific: tower and cantilever rewards compare outcomes with analytic optima, while Balance, Pocket Cube, and Hidden COM use distinct success criteria.The four physical manipulation tasks allow nine hours and 50,000 steps for one continuous attempt; Hidden COM has one hour and 12,000 steps per box.Source: Task Family 03: Tabletop Physical… · p. 15

Task Family 04: Whole-Body Motion Tracking 5 instances

This family asks agents to train and export humanoid motion-tracking policies, then evaluates them in a separate simulator under hidden perturbations.

Task Family 04: Whole-Body Motion Tracking 5 instances

  • Agents train Unitree G1 policies to reproduce five 20-second motions, including a LAFAN1 dance excerpt, and export the learned policy as ONNX.Training uses GPU-parallel MuJoCo-Warp and evaluation uses CPU MuJoCo-C; submitted policies are limited to 10 million parameters and 1–16 observation frames.Source: Task Family 04: Whole-Body Motion… · p. 16
  • Evaluation perturbs dynamics, sensors, terrain, latency, and pushes, then averages reward over eight hidden seeds using tracking, survival, jerk, and fall terms.Training and evaluation share a four-hour session budget; dance thresholds are calibrated, while the other four motion sets remain provisional.Source: Task Family 04: Whole-Body Motion… · p. 16

Task Family 05: nanoVLA Recipe Engineering 4 instances

The nanoVLA family evaluates policy-recipe engineering in LIBERO and RoboTwin, with open-design and robustness tracks.

Task Family 05: nanoVLA Recipe Engineering 4 instances

  • Agents maximize LIBERO-10 success using DINOv2-base as the only pretrained component, choosing the remaining training and policy design themselves.The task provides demonstrations, an encoder, and a socket protocol, but no policy, trainer, simulator, or evaluator implementation.Source: Task Family 05: nanoVLA Recipe… · p. 17
  • Submitted Python recipes are replayed offline and scored by success rate on fixed episode sets, with perturbations in robustness tracks.LIBERO uses 500 evaluation episodes per track and RoboTwin uses 300 open-track or 225 robustness episodes; failed replay, artifact validation, or completion thresholds yield zero.Source: Task Family 05: nanoVLA Recipe… · p. 17

Task Family 06: Blind Planar Pose Estimation 4 instances

Family 06 tests planar pose and shape estimation for moving, partly occluded blocks under four sensor and hardware configurations. Evaluation separates pose and shape accuracy from inference speed and interface validity.

Task Family 06: Blind Planar Pose Estimation 4 instances

  • A robot-mounted stick pushes one of three asymmetric red blocks, whose planar pose and shape must be estimated through rotation and occlusion from a fixed oblique camera.Mesh geometry and the private scene are withheld, so agents must learn geometry and orientation conventions from rendered examples.Source: Task Family 06: Blind Planar Pose… · p. 18
  • The four variants span RGB-only and RGB-D Python estimators, a learned RGB-D estimator, and a method-agnostic Python estimator, with different CPU/GPU allocations.All final inference runs on CPU; the learned model receives RGB, depth, and a validity mask and makes independent per-frame predictions.Source: Task Family 06: Blind Planar Pose… · p. 18
  • Pose scoring averages the five largest translation and wrapped-angle errors separately, while a failed load or interface gate scores zero and verifier-measured speed can reduce reward.Shape correctness conditions the pose score; the speed deduction is D = 0.2 clip[0,1][(10 −ν)/9].Source: Task Family 06: Blind Planar Pose… · p. 18
  • Evaluation uses 100 independent frames in ten groups followed by ten push trajectories of up to 60 frames, with causal state allowed for Python estimators but not the learned model.Development lasts two hours; design-set evaluation is limited to six calls, and final inference is CPU-only.Source: Task Family 06: Blind Planar Pose… · p. 18

Task Family 07: Contact-Rich Bin Clearing 1 instance

Family 07 asks for a policy that clears loose steel brackets from a bin into a conveyor nest. The task emphasizes one-at-a-time placement, clean completion, throughput, and safe handling.

Task Family 07: Contact-Rich Bin Clearing 1 instance

  • The policy must clear 12–16 loose U-brackets into a small conveyor placement nest, one part at a time, while avoiding drops, damage, and hard bin impacts.The task scene requires contact to pick up a bracket, and delivering multiple parts together blocks the nest until extras are removed.Source: Task Family 07: Contact-Rich Bin… · p. 19
  • The policy receives joint positions, velocities, torques, wrist force/torque, magnet state, time, and overhead and wrist RGB-D images, but not ground-truth part poses.The setup includes a Panda workcell, contact electromagnet, public geometry and calibration, and five public pile seeds.Source: Task Family 07: Contact-Rich Bin… · p. 19
  • Episode scores combine cleared fraction, throughput, and a clean complete-clear indicator, then are clipped to [0, 1] and averaged over the eight piles.Missing or unloadable policies score zero; a subsequent smoke failure caps reward at 0.1.Source: Task Family 07: Contact-Rich Bin… · p. 19
  • Evaluation uses eight hidden piles, each with 120 seconds of simulated time and a 360-second cumulative policy-computation limit.The policy runs CPU-only, with a 30-second hang cap per call; incomplete throughput uses 120 seconds as the clear makespan.Source: Task Family 07: Contact-Rich Bin… · p. 19

Task Family 08: Universal Mobile-Manipulator Base 1 instance

These mechanical-design tasks test whether agents can produce artifacts that work across multiple robot configurations. Family 08 evaluates a shared mobile base, while Family 09 combines passive gravity compensation with adaptive feedforward for three leader devices.

Task Family 08: Universal Mobile-Manipulator Base 1 instance

  • Family 08 requires one mecanum chassis, battery, wheels, and arm-mount site to work unchanged with Panda, UR5e, and xArm7 arms during shelf entry and loaded target holding.The verifier replaces the preview arm with each canonical model; all 12 shelf targets count for each arm, with a required 1 kg load and a 2 kg margin test.Source: Task Family 08: Universal… · p. 20
  • Family 08 evaluates validity, design/resources, static and dynamic stability, and shelf integration, taking the minimum across the three arms at each checkpoint.The checkpoint weights are 15%, 35%, 15%, 20%, and 15%; a failed load or structure checkpoint caps reward at 0.15.Source: Task Family 08: Universal… · p. 20
  • The base design must balance compactness, mass, and material use against reaching and holding targets without shelf collision, tipping, or sustained loss of wheel support.The task provides a fixed universal adapter, immutable mecanum wheels, battery, payload, aluminum-profile catalog, and shelf specification.Source: Task Family 08: Universal… · p. 20

Task Family 09: Multi-Robot GELLO Co-Design 1 instance

  • Family 09 asks for gravity-compensation designs for three GELLO lead arms, paired with Panda, UR5e, and xArm7 follower robots.Each design modifies the lead assembly with springs, masses, supports, and counterweights, and implements model-based feedforward with per-device adaptation.Source: Task Family 09: Multi-Robot GELLO… · p. 21
  • Family 09 supplies 16 settled-pose probes per device, including a home measurement and 15 additional poses selected as one batch.Hidden device copies vary link masses and handle payload; designs must preserve joint frames, geometry, mounts, and stock servos under mass and printability constraints.Source: Task Family 09: Multi-Robot GELLO… · p. 21
  • Family 09 scores ten equally weighted checkpoints per variant, covering clearance, passive behavior, adapted reference-arm behavior, and combined hold, backdrive, servo headroom, and push recovery.Final reward averages all three variants; load or structure failure skips the submitted-arm battery and unavailable metrics score zero.Source: Task Family 09: Multi-Robot GELLO… · p. 21

B FULL PER-TASK RESULTS

The appendix reports verifier rewards for all 51 tasks across nine families and 11 model–harness systems, with a workflow-weighted RLE Index and task-specific diagnostics. It also describes resource reporting, whose elapsed-time measure reflects full run duration rather than isolated agent compute.

B.1 COVERAGE AND READING GUIDE

  • The RLE Index averages family scores within each of four workflows, then gives the workflows equal weight; family scores average their constituent tasks.Scores are verifier rewards multiplied by 100, not uniformly success percentages, and missing values are distinguished from measured zeros.Source: §B.1 · p. 22, Figure 8 · p. 22
  • Figures 9–14 report task-level rewards for agentic control, harness transfer, tabletop physical reasoning, whole-body motion tracking, nanoVLA recipe engineering, and blind planar pose estimation, preserving task distinctions.Source: Figure 9 · p. 23, Figure 10 · p. 23, Figure 11 · p. 24
  • Table 4 supplements normalized rewards with native diagnostics, but those quantities do not reconstruct gated verifier rewards.Motion errors and survival are averaged over clips, while bin-clearance and throughput come from the joint task report; machine-readable records retain exported per-task metrics and gate fields.Source: §B.1 · p. 25, Table 4 · p. 25

B.2 RESOURCE USE

  • Resource summaries follow the task–family–workflow hierarchy, while total cost sums retained runs and elapsed time measures run start to finish rather than isolated agent compute.All failed-status runs are timeout failures in which the agent uses the full time budget without calling termination or submission; Table 5 also notes missing cost and token totals for two models due to Harbor incompatibility.Source: §B.2 · p. 28, Table 5 · p. 28
Loading 2609.34210v2…