PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbodiedLab: Benchmarking Embodied Agents That Act to Know

Hongcheng Gao, Zelin Zheng, Alex Zhou, Jay Zhu, Shijia Ge, Yazhe Wang, Xiping Li, Jianshu Zeng, Di Wu, Xuan Shangguan, Yexing Wan, Lingyu He, Yongli Wang, Xiaohan Yu, Zhiqi Jia, Yue Liu, Bryan Hooi, Yinpeng Dong, Wentao Zhang, Sihang Wu, Xiao He

arXiv:2610.35385v1cs.RO

TL;DR

Existing benchmarks separate reaching a known physical state from learning through nonphysical experiments, leaving agents’ physical interaction and information gathering coupled unevaluated. EmbodiedLab uses parameterized simulated laboratory experiments and evaluates six frontier models across levels of support. The strongest T0 agent succeeds on 43.3% of runs, and most failures arise in interacting with the apparatus.

  • Problem

    Manipulation benchmarks specify the target state, while scientific discovery benchmarks replace physical experimentation with symbolic actions or numerical queries, separating physical interaction from learning.

    Source: §1 · p. 2

  • Method

    EmbodiedLab evaluates agents on 30 parameterized quantitative experiments across five disciplines and 14 apparatus platforms, scoring results, episode readings, and required bench state.

    Source: §1 · pp. 2–3, §3.1 · p. 4, §3.3 · p. 5

  • Results

    43.3%: Claude Opus 5.5’s T0 success rate; across agents, most failures arise in the physical interaction loop, and manipulation skills help weaker agents more than perception and planning.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The benchmark assesses whether agents can uncover information through physical interaction while grounding results in episode readings and satisfying required final states.

    Source: §1 · p. 3

Abstract

from arXiv · show

Embodied agents are expected to accomplish real-world goals by deciding for themselves how and when to interact with their surroundings. For many such goals, changing the physical state of the world is the means rather than the end: the agent changes the state of a system through interaction, observes the feedback, and explores until it either achieves the goal itself (e.g., discovering physical laws) or learns what the goal requires; it acts to know. Manipulation benchmarks stop at acting to change, with the target state given; scientific discovery benchmarks evaluate knowing but replace physical experimentation with symbolic actions or numerical queries. We introduce EmbodiedLab, a benchmark in which a robot must act to know. Quantitative science experiments make this setting controllable: the parameters of the system are resampled for every instance, so the result cannot be known in advance, yet it can be scored exactly. Across 30 experiments in five disciplines on 14 apparatus platforms, the agent receives an objective and the apparatus, chooses its own procedure, and operates a physics-based simulated laboratory, reading the instruments through its cameras. Evaluation checks what the agent came to know, grounded in readings recorded during the episode, as well as the physical state the task requires; every experiment is verified to be solvable. We evaluate six frontier models in their developers' coding agents with native observation and control, with added perception and motion planning, and with manipulation skills distilled from the strongest agents. With native control, even the strongest agent succeeds less than half the time; for every agent, most failures arise in interacting with the apparatus, and manipulation skills help weaker agents more than perception and planning. The benchmark and its evaluation code are available at https://github.com/HexaFuture/EmbodiedLab.

1 Introduction

EmbodiedLab addresses the gap between benchmarks that score reaching a known physical state and discovery benchmarks that omit physical experimentation. It introduces a controllable laboratory benchmark for agents that must interact to learn, and evaluates how they perform with different levels of support.

1 Introduction

  • Acting to know means changing a physical system to uncover information, with the resulting knowledge guiding what the agent does next.The objective is not necessarily to change the world to a known target; it may be to learn what the goal requires.Source: §1 · p. 2
  • Manipulation benchmarks provide a target state, while scientific discovery benchmarks use symbolic actions or numerical queries instead of physical experimentation.In real tasks, actions shape observations, and observations guide subsequent actions.Source: §1 · p. 2
  • EmbodiedLab comprises 30 experiments across five disciplines and 14 apparatus platforms, with objectives about a property, a prediction, or a configuration.Its tasks are solvable, with tolerances set by instruments, and are scored on results, recorded evidence, and required bench state.Source: §1 · pp. 2–3
  • The agent receives an objective and apparatus but no prescribed method, operates an open laboratory, and must ground its result in readings recorded during the episode.Success also requires leaving the bench in the state the task requires.Source: §1 · p. 3
  • Claude Opus 5.5 succeeds on 43.3% of T0 runs; perception and planning change success by at most 10 points, while manipulation skills raise it by 10 to 27 points.No agent receiving skills reaches the strongest agent at T1.Source: §1 · p. 3

2 Related Work

Related benchmarks separately assess reaching a specified physical state or learning through nonphysical experimental actions. EmbodiedLab evaluates agents on finding things out through physical interaction and measurements.

Embodied manipulation benchmarks

  • Manipulation and laboratory benchmarks score reaching a specified scene state or completing protocol steps, so they do not require finding out an unknown result.Examples include RoboCasa [13], LIBERO [12], BEHAVIOR-1K [31], RoboTwin 2.0 [14], RoboDojo [15], LabUtopia [21], AutoBio [22], SciHorizon-eLab [23], Pipette [24], and LabDex [25].Source: §2 · p. 3

Exploration and scientific discovery benchmarks

  • Scientific discovery benchmarks let agents experiment through text actions, instrument calls, or input choices, with results returned as text or exact values.Embodied question answering [33] [35] [36] and interactive perception [4] use action for information gathering but target views or a single object property.Source: §2 · p. 3

Agents that act through code

  • Coding-agent research has demonstrated composing perception and control tools, but evaluations use manipulation benchmarks or selected tasks rather than assessing the full measurement process.EmbodiedLab evaluates whether agents can move the robot and take, read, and analyze measurements required by the goal.Source: §2 · p. 4

3 The EmbodiedLab Benchmark

EmbodiedLab defines experiments in which agents choose how to interact with a parameterized physical system, then scores both what they learn and the required final state. Its open simulated laboratory, tiered interface, and validated experiments support evaluation of this setting.

3.1 Tasks

  • Each task hides a newly drawn system parameter and gives the agent an objective, public parameters, and rules but no method; the agent submits a result with citations to episode records.Objectives cover measuring a property, predicting a response, or configuring an apparatus.Source: §3.1 · p. 4

3.2 An open laboratory

  • The simulated teaching laboratory spans 30 experiments on 14 apparatus platforms, with physical operations and instrument readings visible through camera images.The lens experiment illustration shows a hidden parameter, camera-based interaction and observation, and success gates tied to the hidden target.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
  • The interface ranges from camera images and joint control at T0 to added perception and motion planning at T1, then skills distilled from successful runs at T2.T2 is diagnostic rather than part of the released protocol because its skills partly reuse known solutions; agents still read, record, and analyze at every level.Source: §3.2 · p. 5, Figure 4 · p. 6

3.3 Evaluation

  • Success requires cited readings from the episode, valid measurement conditions, tolerance compliance, and required bench restoration; milestones locate where failed runs stop.The equation expresses success as the conjunction of evaluation gates and an error within tolerance; restoration includes returning unused parts and leaving apparatus safe.Source: §3.3 · pp. 5–6

3.4 Construction and validation

  • Every experiment is verified solvable through a reference solution using the public interface, and its tolerance is set by instrument capability.Experiments are classified by basic versus advanced design demands and by interaction capabilities such as fine reading and multi-stage work.Source: §3.4 · p. 6

4 Experiments

The experiments test whether frontier agents can act to know, where their runs stop, how interface support changes outcomes, and how they fail.

4.1 Setup

  • Agents run in their developers’ coding agents at middle reasoning settings; all are evaluated at T0 and T1, while three also run with T2 skills.Runs share the same task card, budget, and wall-clock limit at every level; success is the headline metric and experiments count equally.Source: §4.1 · pp. 6–7

4.2 Can frontier agents act to know?

  • Claude Opus 5.5 reaches 43.3% success at T0, GPT-6 Astra 40.0%, and GPT-6.1 Sol 28.3%.Across the 30 experiments, 27 are solved by at least one agent, but no agent solves the black-box resistor, focal length of a lens, or two-lens relay.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Where runs stop

  • The three strongest agents usually reach sufficient readings, but knowing the result and restoring the bench are their lowest-pass stages.They know the result at similar rates (51.7%–55.0%); restoration separates success rates, and GPT-6.1 Sol leaves the bench unrestored after an accurate result in 23% of runs.Source: §4.3 · pp. 7–8, Figure 6 · p. 8

4.4 What each level changes

  • T1 raises success by 1.7–10.0 percentage points, while T2 manipulation skills raise it by 10.0–26.7 points, but experiment design remains with the agent.The skills help most in fine manipulation and multi-point acquisition and analysis; the three T2 agents complete readings in 47%–77% of runs, versus 90% for Claude Opus 5.5 at T1.Source: Table 3 · p. 9, §4.4 · p. 9

4.5 How agents fail

  • Acting is the largest failure group for every agent; acting, observing, and restoring account for 75%–100% of failed runs, while knowing accounts for at most 25%.For weaker agents, acting accounts for 64%–69% of failed runs and contact alone for 45%–62%; for the strongest agents, failures spread to later stages.Source: §4.5 · p. 9

5 Conclusion

EmbodiedLab measures whether agents can learn through physical interaction, and results show that reliable interaction remains a major barrier. Manipulation skills help weaker agents, but long-horizon success remains low.

5 Conclusion

  • 43.3% is the strongest agent’s T0 success rate, while three agents succeed on at most 10% of runs.Acting to know remains far from solved.Source: §5 · p. 9
  • 75% to 100% of failed runs involve the physical loop, while at most 25% fail in reasoning from readings already obtained.The decisive errors include slipping grasps, misread digits, and parts not restored.Source: §5 · p. 9
  • Perception and motion planning change success by at most 10 points, while manipulation skills raise weaker agents by 10 to 27 points.Even with those skills, weaker agents reach at most 14.3% in long-horizon work.Source: §5 · p. 9
  • EmbodiedLab scores what agents learn using episode readings and checks the physical state they leave behind.The authors release the benchmark and its evaluation code to track progress toward agents that act to know.Source: §5 · p. 9

A Experiment list

The experiment list covers 30 experiments grouped by primary capability dimension and tier. Its tables summarize coverage and include the cantilever experiment as a full example.

A Experiment list

  • 30 experiments are grouped by primary capability dimension, with the tier tagged in the camera-view overview.Table 4 summarizes coverage by capability dimension and tier.Source: Figure 7 · p. 13, Table 4 · p. 14
  • The cantilever experiment (#15) is presented in full as a condensed task-card and specification example.It is the experiment shown in Fig. 5.Source: A Experiment list · p. 14, Table 5 · p. 14
  • Task cards and evaluator specifications for all 30 experiments are included in the released benchmark.Source: A Experiment list · p. 14

B Benchmark details

The benchmark varies experiment complexity, resamples hidden parameters, and models apparatus behavior and instrument limits. It also documents experiment budgets and the interface levels and tools.

B Benchmark details

  • Physical models cover rigid-body mechanics and simulated circuits, optics, heat, dye spectra, and titrations.Instrument effects include quantization, noise, lag, stray light, and baseline drift in the relevant models.Source: B Benchmark details · p. 14
  • Every experiment is solved within tolerance; the reference solution stays within half the tolerance in all but one experiment, where its error reaches 0.55.Tolerances reflect instrument-supported precision, with steps from 0.2% to 3% or instrument-set absolute steps; static friction has 5%.Source: Figure 8 · p. 16, B Benchmark details · p. 16
  • Simulated-time budgets range from 200 to 1500 seconds per experiment, and every run also has a wall-clock limit.Source: B Benchmark details · p. 16
  • Advanced experiments require at least one scientific decision, such as choosing conditions, sequential design, model identification, or acting on a result.Basic experiments can meet tolerance with a fixed standard design and estimator; four apparatus have experiments in both tiers where only the procedure changes.Source: B Benchmark details · p. 16
  • Each experiment has a declarative specification, and its public-seed instances draw hidden parameters anew.Specifications include the scene, prior, instruments, gates, milestone checks, tolerance, budget, and submission schema.Source: B Benchmark details · p. 16

C Interface reference

  • The interface reference lists the tools available at each level and notes that T2 adds skills implemented in code that calls T1 tools.Tools outside a level are neither listed nor callable.Source: C Interface reference · p. 16

D Protocol

All coding agents use the same public interface, while the tools available vary by interface level. Runs fail when they do not succeed, including when budgets expire or submissions miss tolerance or gates.

D Protocol

  • Every coding agent writes and runs code against the same public interface, leaving decisions to the model.Models are evaluated together with their developers’ coding agents.Source: D Protocol · p. 16
  • A run fails if it does not succeed, including stopping without submission, exhausting either budget, missing tolerance, or failing a gate.Source: D Protocol · p. 16
  • Interface levels add layers, and tools outside a level are unavailable; T2 manipulation skills are code that calls T1 tools.Source: Table 7 · p. 17, Table 8 · p. 17

E Per-experiment results

The appendix presents per-experiment outcomes at T0 and the lowest support level at which each experiment is solved.

E Per-experiment results

  • Table 9 reports each agent’s success rate and process score for every T0 experiment, while Figure 9 identifies the lowest level at which each agent solves each experiment.Source: E Per-experiment results · p. 17

F Failure causes

Failure analysis attributes each failed run to its first decisive, uncorrected error, then reports cause shares and representative failures by stage. The section also presents per-experiment success at T0 and the lowest solving level for each agent.

F Failure causes

  • A run’s cause is its earliest uncorrected error after which success is impossible; when causes overlap, the earlier sufficient cause is used.Corrected errors and the last visible symptom do not determine attribution; failure only during L5 bench restoration is attributed to restoring.Source: F Failure causes · p. 17
  • Table 9 reports per-experiment success at T0, with process scores in grey; both are averaged over runs of that experiment.Source: Table 9 · p. 18
  • Figure 9 marks each agent’s lowest solving level per experiment; dots indicate solving at one level but not the next level run.GPT and Claude agents were not run at T2.Source: Figure 9 · p. 18
  • Figure 10 groups attributed failure causes by stage and shows five camera-frame examples per stage, with the decisive spot ringed and cause named.Examples include collisions, slipping parts, misread measurements, misjudged board height, procedures cut short, missing measurements, and untidy benches.Source: F Failure causes · p. 17, Figure 10 · p. 19

Related papers

Loading 2610.35385v1…