PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-Agent: Evaluating General-Purpose Agents for Direct Embodied Manipulation

Zijie Diao, Yitong Chen, Sicheng Xie, Tianyi Lu, Wujian Peng, Guojin Zhong, Houze Xu, Ziyi Ye, Zuxuan Wu, Yu-Gang Jiang

arXiv:2609.39507v1cs.RO

TL;DR

It remains unclear whether general-purpose agents’ planning and tool-use capabilities transfer from digital environments to embodied manipulation, where benchmark scaffolding can shape observations and actions. LIBERO-Agent evaluates agents through autonomous observation choices and native robot control across a factorized task suite. Agents perform well on perception and easy short-horizon tasks but are less reliable on harder tasks, with GPT-6 Astra leading the evaluated agents.

  • Problem

    It remains unclear whether general-purpose agents’ planning and tool-use capabilities transfer to embodied manipulation, where benchmark-side observation and action choices can obscure which capabilities come from the agent.

    Source: §1 · pp. 1–2

  • Method

    LIBERO-Agent provides an agent-native manipulation benchmark with a 200-task pool and a 30-task primary suite spanning perception, short-horizon execution, and long-horizon composition.

    Source: §1 · pp. 2–3

  • Results

    Across seven agents, performance is strongest on perception and easy short-horizon tasks but declines on harder tasks, while GPT-6 Astra leads with a 45.0/100 Performance Score.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Correct target identification does not consistently translate into reliable physical execution, and Astra’s remaining failures include cross-stage interference and geometric errors.

    Source: §6 · p. 10, §5.3 · p. 10

    Astra can perform individual physical interactions effectively yet still struggles to preserve task-relevant state across stages and realize actions with task-valid geometry.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

General-purpose agents can plan, use tools, and revise their behavior from feedback, but it remains unclear whether these capabilities transfer from digital environments to embodied manipulation. To investigate this question, we introduce LIBERO-Agent, an agent-native benchmark for evaluating these agents in robot manipulation tasks. Rather than asking agents to submit task-level Python control programs or operate through high-level robot skills, LIBERO-Agent provides an interactive robotic environment where agents can select which observations to inspect, process them with their own tools, and issue native action commands. LIBERO-Agent integrates 200 tasks into a common interaction framework and provides a 30-task primary suite that separates perception, short-horizon execution, and long-horizon composition. Results reveal a pronounced reliability gap: while agents perform well on perception and easy short-horizon tasks, their performance degrades substantially on hard short-horizon and long-horizon tasks. Richer observations improve short-horizon manipulation, while demonstration benefits depend on the agent and format. Among these agents, GPT-6 Astra achieves the strongest overall performance. Further analysis shows its major advantage lies in mechanism interaction, especially when sustained physical contact is needed, while its remaining failures stem from cross-stage interference and geometric errors.

1 Introduction

LIBERO-Agent evaluates whether general-purpose agents’ planning and tool-use capabilities extend to embodied manipulation, where benchmark-side choices can obscure what capabilities come from the agent. It gives agents responsibility for observation and action choices, tests three levels of competence, and finds a marked gap between easy and harder tasks.

1 Introduction

  • Existing manipulation benchmarks may preselect observations or abstract physical execution, making it difficult to distinguish agent capabilities from benchmark scaffolding and diagnose failures across competence types.The motivation is to test whether agents can organize perception, reasoning, and action throughout interactive tasks, including amid occlusion, execution errors, and changing task state.Source: §1 · pp. 1–2
  • In LIBERO-Agent’s agent-native interface, agents choose and process observations, decompose tasks, and compose native actions within fixed action and time budgets.The benchmark specifies available observations and the robot-control interface rather than prescribing task-level action primitives.Source: §1 · pp. 2–3
  • GPT-6 Astra leads seven agents with a 45.0/100 Performance Score, succeeding on all perception and easy short-horizon tasks but reaching 40% on hard short-horizon and 22% stable stage completion on hard long-horizon tasks.Richer observations improve short-horizon manipulation, while demonstration benefits depend on the agent and format; Astra’s advantage is strongest in mechanism interaction associated with sustained physical contact.Source: §1 · pp. 2–3
  • The benchmark integrates 200 tasks and selects a 30-task primary suite spanning perception, short-horizon execution, and long-horizon composition.Long-horizon tasks combine abilities across stages and add instruction decomposition, scene exploration, cross-stage constraints, and recovery from intermediate failures.Source: §1 · pp. 2–3
  • Astra’s advantage is strongest in mechanism interaction associated with sustained contact, while its remaining failures involve cross-stage interference and geometric errors.Richer observations improve short-horizon manipulation across Astra, Opus, and Fable, whereas demonstration benefits vary by agent and representation format.Source: §1 · p. 3

2 Related Work

General-purpose-agent benchmarks test multi-step interaction and execution across digital environments, while embodied-manipulation work broadens robot-policy capabilities but often fixes the observation interface. LIBERO-Agent brings agent-native evaluation to manipulation and factorizes its primary suite to diagnose capabilities.

2.1 Benchmarks for General-Purpose Agents

  • General-purpose-agent benchmarks cover reasoning, tool use, web, computer, code, and application interaction, emphasizing multi-step interaction, feedback, and execution-based outcomes.Examples include AgentBench [18], GAIA [21], WebArena [36], OSWorld [30], SWE-bench, and AppWorld [29].Source: §2.1 · p. 3

2.2 Embodied Manipulation Benchmarks

  • Robot-manipulation benchmarks broaden evaluation to language-conditioned reasoning, memory, bimanual generalization, and long-horizon execution, but typically assume a fixed observation interface.Foundation-model approaches such as Code as Policies [16] and VoxPoser [10] generate programs or spatial value maps executed through separate perception and control modules.Source: §2.2 · p. 3
  • Unlike EmbodiedBench [32] and CaP-X [8], LIBERO-Agent lets agents select and process observations and compose bounded native actions, with a 30-task suite separating perception, short-horizon execution, and long-horizon composition.EmbodiedBench uses discretized actions and additional object-level perceptual information; CaP-X synthesizes programs over predefined perception and control primitives.Source: §2.2 · p. 3, §2.1 · p. 3

3 Benchmark

LIBERO-Agent uses iterative interaction with a simulated manipulation environment, leaving agents to choose observations and compose actions within a fixed budget. Its 30-task primary suite separates perception, short-horizon execution, and long-horizon composition.

3.1 Agent–Simulation Interaction Protocol

  • Agents receive an instruction, available observation modalities, and a fixed budget, then alternate decisions with bounded robot-action sequences and updated observations; task success is evaluated independently.No task-level manipulation primitives or privileged simulator state are exposed to agents; some tasks also use private evaluators for physical events and ordering constraints.Source: §3.1 · p. 4

3.2 Simulation Implementation

  • The benchmark implements its interaction protocol through three MCP tools and uses a MuJoCo simulation with a 7-DoF Franka Panda and native OSC_POSE actions.Available observations include third-person and wrist-view RGB images, depth, calibration, and robot state; the task pool integrates 200 tasks and assets from related benchmarks.Source: §3.2 · p. 4

3.3 Task Design

  • The 30-task primary suite has ten tasks each for perception-focused, short-horizon, and long-horizon manipulation, with short- and long-horizon tasks divided into easy and hard subsets.Perception tasks test identifying task-relevant properties through simple embodied execution; short-horizon tasks isolate physical state changes, while long-horizon tasks test planning and state tracking across stages.Source: §3.3 · pp. 4–5

4 Experiments

The experiments compare seven agents on the primary 30-task suite, then examine how observation inputs and demonstrations affect manipulation. Astra leads overall, but performance falls on harder and longer tasks, while observation and demonstration benefits vary across agents and settings.

4.1 Experimental Setup

  • The primary evaluation compares seven agents over three rollouts per task using a 1,800-second limit; its 100-point score weights harder and longer-horizon tasks more heavily.The reported metrics require reproducible outcomes across rollouts, and tasks are equally weighted within each subset.Source: §4.1 · pp. 5–6

4.2 Main Results

  • In the primary 30-task suite, Astra achieved the highest overall Performance Score at 45.0/100, ahead of Opus (19.3) and Fable (15.8); Sol, Qwen, Kimi, and DeepSeek scored 10.8, 10.5, 5.5, and 4.5, respectively.Source: Table 1 · p. 6, §4.2 · p. 6
  • High target-identification accuracy did not ensure reliable execution: several agents with 90% stable judgment accuracy had only 0–30% perception-task success and no stable hard short-horizon successes.Opus and Fable also fell from 100% and 90% judgment accuracy to 80% and 60% task success, respectively.Source: §4.2 · p. 6
  • Astra fell from 100% easy to 40% hard short-horizon success and achieved 50.0% easy versus 22.0% hard stable long-horizon stage completion; every other agent scored 0% on hard long-horizon tasks.The results indicate that reliable execution on simpler tasks does not necessarily extend to harder interactions or successive stages.Source: §4.2 · p. 6

4.3 Observation and Demonstration Context

  • Adding calibrated depth raised overall stable short-horizon success by 20 percentage points for Astra, Opus, and Fable; dynamic proprioception raised Astra from 30% to 50%.Astra improved on both easy and hard tasks with calibrated depth, while Opus and Fable’s gains occurred entirely on easy tasks.Source: Table 2 · p. 7, §4.3 · p. 7
  • Video raised Astra’s overall stable stage completion from 36.0% to 66.2% and Fable’s from 8.3% to 17.5%, with larger gains on easy than hard tasks.Difficult physical interaction may remain a bottleneck for video guidance on harder tasks.Source: Table 3 · p. 7, §4.3 · pp. 7–8
  • Adding trajectories was inconsistent: Opus rose from 5.8% to 17.5%, Astra fell from 66.2% to 58.0%, and Fable gained little over video alone.The authors suggest cross-episode trajectories may provide control cues while also encouraging actions mismatched to the current episode.Source: §4.3 · p. 8

5 Diagnosing the Strongest Agent

Across five hard long-horizon tasks, Astra’s advantage is concentrated in mechanism-level execution and is associated with sustained contact; its remaining failures involve cross-stage interference and invalid geometry.

5.1 Localizing Astra’s Long-Horizon Advantage

  • Stage and failure categories distinguish object manipulation from mechanism interaction, and local execution failures from cross-stage interference.Execution success rates exclude failures attributed to cross-stage interference.Source: §5.1 · p. 8, Figure 2 · p. 8
  • Astra reaches 65.0% of hard-task stages and completes 36.7%; its mechanism-interaction execution success is 94.1%, versus 7.1%–37.5% for other agents.On object manipulation, Astra and Sol both achieve 66.7%.Source: Table 4 · p. 8, §5.1 · pp. 8–9

5.2 Sustained Contact During Mechanism Interaction

  • Astra’s mean sustained-contact ratio is 20.63%, versus 4.23%–7.17% for other agents; across 64 openings, SCR correlates with local success (r=0.711).None of 19 zero-SCR operations succeed; success is 11/11 when SCR exceeds 25%.Source: Figure 3 · p. 9, §5.2 · p. 9

5.3 Remaining Failures: Cross-stage Interference and Geometry

  • Astra’s residual failures include placements blocking later access, subsequent actions disturbing established state, and operations performed with invalid geometry.The analysis characterizes these as a shift from local execution errors toward cross-stage interference and geometric errors.Source: §5.3 · pp. 9–10, Figure 4 · p. 10

6 Conclusion

LIBERO-Agent evaluates general-purpose agents through autonomous observation selection and native robot control, and Astra leads the seven-agent comparison while remaining less reliable on harder manipulation. The appendix also records which harness versions were used.

6 Conclusion

  • LIBERO-Agent evaluates agents through autonomous observation selection and native robot control; Astra leads seven agents but is less reliable on harder short- and long-horizon tasks.Richer observations improve manipulation, demonstration benefits vary by agent and format, and Astra’s remaining failures involve cross-stage interference and geometric errors.Source: §6 · p. 10

A Agent Harnesses

  • Table 5 lists each evaluated model’s harness version, taken from archived run manifests and session metadata.The table describes harnesses and versions used in the experiments.Source: A Agent Harnesses · p. 14, Table 5 · p. 14

B.1 Task construction and primary evaluation suite

The primary suite contains 30 tasks drawn from a broader collection of manipulation benchmarks, organized to evaluate perception and short- and long-horizon manipulation.

B.1 Task construction and primary evaluation suite

  • The task collection combines LIBERO and tasks or assets from RoboMemArena, LiLo-VLA, VLABench, and adapted mechanisms in a shared runtime.The full collection has 200 tasks; 30 representative tasks form the primary suite.Source: §B.1 · p. 14
  • The 30-task suite divides into perception, short-horizon manipulation, and long-horizon manipulation, with ten tasks in each group.Perception tasks primarily test target identification and use deliberately simple actions.Source: §B.1 · p. 14
  • Short-horizon tasks split elementary grasping and actuation from more constrained operations, while harder long-horizon tasks combine planning, contact constraints, or coupled mechanisms.Each difficulty subset contains five tasks and receives equal task weighting internally.Source: §B.1 · p. 14

B.2 Task-specific success semantics

The appendices specify how task outcomes are scored, how episodes and prompts are configured, what demonstration bundles contain, and how agents construct programs during successful executions.

B.2 Task-specific success semantics

  • Task evaluators combine terminal predicates with physical event histories where needed, and record full success separately from partial stage completion.Scoring also checks ordered progress, repeated events, persistent constraints, active sensing, contact, and insertion geometry for task-specific requirements.Source: §B.2 · pp. 14, 16

B.3 Episode conditions and outcome adjudication

  • Primary comparisons use three executions per task under fixed scene conditions; runs use high reasoning where supported, full observations, no demonstrations, and a 1,800-second wall-clock limit.The wall-clock limit is the operative execution budget.Source: §B.3 · pp. 16–17

C.1 Shared episode prompt

  • The shared episode prompt gives the task instruction and interaction contract, points to the current observation, and supplies no task-specific control program.The recorded T23 example asks agents to inspect drawers and place butter in the drawer already containing an object.Source: Figure 5 · p. 18, §C.1 · p. 18

C.2 Representative task instructions

  • Representative instructions include physically weighing visually identical mugs, inserting a tube vertically, rearranging bowls without tabletop contact, and operating a latched door.Some prompts leave strategy or the buffer choice to the agent rather than spelling it out.Source: §C.2 · pp. 18–19

C.3 Video and action-trajectory bundles

  • Demonstrations provide synchronized head- and wrist-view video; the video-plus-trajectory format adds normalized native action vectors, while current observations remain separate from historical data.Agents can inspect the video with their own tools, and the trajectory records issued actions rather than measured end-effector poses.Source: §C.3 · p. 19

D Programs Constructed During Successful Executions

  • Successful Astra examples connect agent-created programs and tool use to recorded observations, actions, and outcomes; the programs were not supplied as task-level primitives.The examples cover drawer inspection and placement (T23) and physical weighing (T08).Source: D Programs Constructed During… · p. 20

D.1 Computing spatial coordinates from selected image locations

  • In the T23 trace, Astra selects image pixels, back-projects depth using camera calibration, and chooses an end-effector target with an approach offset.The coordinates measure visible surfaces in the robot-base frame; object poses and grasp targets are not supplied by the evaluator.Source: Figure 6 · p. 20, §D.1 · p. 20

D.2 Constructing and reusing a pose-feedback helper

Astra builds a local pose-feedback helper that converts current end-effector state and command-line targets into native actions. Repeated batches refresh feedback and support a successful eight-stage T23 execution.

D.2 Constructing and reusing a pose-feedback helper

  • Astra’s local Python helper reads end-effector pose, computes position and rotation errors against command-line targets, clips normalized controls, and prints a JSON action sequence.Rotation error is represented in rotation-vector form.Source: §D.2 · p. 20
  • Across six batches of 60 control steps, angular error fell from 121.88° to 0.76° while the target orientation stayed fixed.The helper repeats one computed action within a batch and refreshes feedback between batches.Source: §D.2 · p. 20
  • All eight required T23 stages completed in 57 accepted control submissions and 895 native control steps.The episode also included an optional final drawer closure.Source: §D.2 · p. 21
  • Figure 7 shows the generated action branch connected to native OSC control, while a receipt verifies execution and task completion is checked separately.Source: Figure 7 · p. 21

D.3 Acquiring physical evidence through active probing

Astra uses repeated lift-and-hold interactions and sensor readings to identify the heaviest of three visually identical mugs. The evaluator confirms all four required events, while the force readings include gripper load.

D.3 Acquiring physical evidence through active probing

  • For each visually identical mug, Astra issued 20 upward commands, then 20 zero-increment commands with the gripper closed, before reading state, proprioception, and an RGB image.Figure 8 shows the repeated action pattern and corresponding force readings.Source: §D.3 · p. 21, Figure 8 · p. 22
  • The middle mug had the largest sensor-frame load magnitude, and Astra placed it in the green bin, completing all four required events.The episode used 28 accepted control submissions and 614 native control steps.Source: §D.3 · p. 21
  • The three sensor-frame z readings were −8.64, −12.19, and −6.86 N; these raw end-effector readings include gripper load.The trace demonstrates repeated physical probing followed by successful target selection.Source: §D.3 · p. 21

E Per-task Results Across Three Rollouts

The per-task results report rollout-based success or stage-completion measures, and the qualitative visualizations pair head- and wrist-camera views. The examples show both completed tasks and failures that leave task criteria unmet or stages unfinished.

E Per-task Results Across Three Rollouts

F Qualitative Rollout Visualizations

  • The rollout visualizations show synchronized head-camera and wrist-camera views at selected control steps.Perception examples use the same task, while manipulation examples show different tasks within their suites.Source: F Qualitative Rollout Visualizations · p. 25

F.1 Perception

  • On T04, the successful rollout places the striped object in the bin, while the unsuccessful rollout transports it there but fails the final placement check.The wrist view shows texture cues and nearby objects; the head view shows positions relative to the bin.Source: Figure 9 · p. 25

F.2 Short-horizon manipulation

  • The short-horizon examples contrast a successful microwave-door opening with a sponge-grasp attempt that leaves the spill unwiped when time expires.The paired views show gripper configuration relative to the handle or sponge.Source: Figure 10 · p. 26

F.3 Long-horizon execution

  • The successful long-horizon rollout completes all eight stages, while the unsuccessful rollout completes two of four before time expires, leaving chocolate outside the drawer.Source: Figure 11 · p. 27
Loading 2609.39507v1…