PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboQuest: Generalist Physical Agents that Search, Inspect and Test

Liu Renhang, Navonil Majumder, Tej Deep Pala, Soujanya Poria

arXiv:2610.10388v1cs.ROcs.AIcs.CV

TL;DR

Robots in unfamiliar environments may need physical interaction to uncover task-relevant information that is absent from observations. RoboQuest benchmarks this goal-directed exploration across ten tasks and evaluates frontier agents and a fine-tuned policy. The best frontier agent succeeds in only 23.2% of episodes, while failure analysis finds exploration and decision-making are major obstacles.

  • Problem

    Generalist robots may encounter goals requiring information absent from observations, but existing manipulation benchmarks do not demand open-ended physical information seeking.

    Source: §1 · p. 1

  • Method

    RoboQuest benchmarks ten tasks across search, inspection, and testing, evaluating five frontier agents and a demonstration-fine-tuned π0.5 policy through a common visuomotor interface.

    Source: §1 · p. 2, §3 · p. 4, §4 · p. 6

  • Results

    GPT-6 Astra achieves the highest frontier-agent success rate at 23.2%, while isolated execution succeeds in 72–81% of attempts and execution accounts for 18–21% of full-task failures.

    Source: §5.2 · p. 11, §6 · p. 12

  • Takeaways & Limitations

    The results point to evidence acquisition, persistent goal and evidence tracking, hypothesis testing, and monitoring and recovery as challenges beyond perception and control.

    Source: §6 · p. 13

Abstract

from arXiv · show

Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks. However, successful operation in an unfamiliar environment may require an agent to seek task-relevant information through interaction when it is absent from the observations: it may need to determine where a relevant object is, inspect an unobserved property, or discover the effect of an unfamiliar tool. We thus introduce RoboQuest, a benchmark for goal-directed embodied exploration, where agents must actively acquire task-relevant information through physical interaction, use the resulting evidence to adapt subsequent actions, and autonomously decide when to commit to task completion. RoboQuest comprises ten mobile manipulation tasks centered on three forms of uncertainty: search, manipulation-based inspection, and interactive testing. We evaluate five frontier multimodal agents through a common visuomotor interface, as well as a $π_{0.5}$ policy fine-tuned on the full-episode demonstrations we release. The best agent succeeds in only 23\% of the episodes, and the fine-tuned policy almost never succeeds. Isolated tests of the execution skills the tasks are built from, with the hidden information supplied, show that the agents can carry out most of the required actions, and our failure analysis attributes only a minority of the failures to execution. Our failure analysis further finds that the agents often stop exploring too early as they make decisions before observing the required evidence for task completion. We also find that agents rarely prevent or repair the disturbances caused by their exploration. Moreover, learning by trial and error remains difficult for most models.

1 INTRODUCTION

RoboQuest targets a gap between manipulation competence and autonomous information seeking: agents must gather evidence through physical interaction, adapt their actions, and decide when to commit. Its evaluations find that exploration and decision-making, rather than execution alone, remain major obstacles.

1 INTRODUCTION

  • Existing manipulation benchmarks do not require open-ended physical information seeking, a capability increasingly relevant to generalist agents facing unfamiliar environments.In search, inspection, and testing, interaction can reveal information while changing beliefs and available actions.Source: §1 · pp. 1–2
  • RoboQuest evaluates goal-directed embodied exploration across ten mobile-manipulation tasks grouped into search, inspection, and testing.Agents must gather task-relevant evidence physically, use it to guide later actions, and decide when they know enough to commit.Source: §1 · p. 2, Figure 1 · p. 2
  • The best frontier agent succeeds in only 23% of episodes, while the fine-tuned π0.5 seldom succeeds; most failures involve exploration and decision-making, not execution.Agents often stop exploring before observing required evidence, rarely prevent or repair disturbances, and most models struggle to learn by trial and error.Source: §1 · p. 2
  • The paper releases 5,000 scripted-oracle demonstrations and evaluates five frontier agents and a fine-tuned π0.5 policy, alongside isolated execution tests and failure analysis.The demonstrations total 366 hours at 20 Hz and include a subtask caption on every frame.Source: §1 · p. 3

2 RELATED WORK

RoboQuest differs from manipulation benchmarks that emphasize execution in fully observed scenes: it requires physical evidence acquisition before task completion. It extends active exploration across search, inspection, and testing, and can evaluate both planning agents and direct-control policies under uncertainty.

2 RELATED WORK

  • Unlike standard manipulation benchmarks that provide complete or immediately accessible task information, every RoboQuest task requires physical information acquisition.Aggregate success also reflects perception and motor execution, so the paper analyzes gathered, retained, and used evidence to assess exploration.Source: §2 · p. 3
  • RoboQuest extends active sensing beyond viewpoint changes or single-step un-occlusion to directed search, active inspection, and interactive testing.These interactions can irreversibly alter the environment, requiring agents to reason about their consequences.Source: §2 · p. 3
  • RoboQuest evaluates both high-level reasoning agents and low-level vision-language-action policies under physical uncertainty, unlike prior evaluations in fully observed scenes.The paper identifies premature commitment or hallucinated states in reasoning agents and difficulty with multi-step hypothesis testing in Markovian VLAs.Source: §2 · p. 4

3 ROBOQUEST BENCHMARK

RoboQuest comprises ten mobile-manipulation tasks in which physical interactions reveal hidden locations, object properties, or cues from test outcomes. Its kitchen setup, demonstrations, and capability framework support evaluating how agents acquire and use evidence, organize actions, and execute goals.

3 ROBOQUEST BENCHMARK

  • The ten tasks require search for hidden object locations, interaction-based inspection of hidden object properties, or trials that reveal cues from outcomes.The three families are Search, Inspect, and Test.Source: §3 · p. 4, Table 2 · p. 5

3.1 BENCHMARK SETUP

  • Tasks run in RoboCasa365 kitchens with a mobile Franka Panda; hidden information is withheld, and success requires every goal condition to hold when the robot presses SUBMIT.Instances vary layouts and task parameters; five tasks also vary whether objects are visible, out of view, or covered.Source: §3.1 · pp. 4–5

3.2 DEMONSTRATION DATASET

  • The released dataset contains 5,000 successful scripted-oracle episodes, with exploration scripted as if the agent lacked private task information.It totals 366 hours at 20 Hz, covers all ten tasks, and evaluation scenes and configurations are held out from the demonstrations.Source: §3.2 · p. 5

3.3 CAPABILITIES NEEDED FOR ROBOQUEST TASKS

  • Task actions either acquire evidence or advance the physical goal, and can serve both roles; success draws on evidence acquisition, evidence use, interactive inference, and action organization.The evaluation separates execution tests from failure analysis, which maps failures to these capability groups.Source: §3.3 · p. 6

4 EXPERIMENTS

RoboQuest evaluates frontier agents and a fine-tuned policy on embodied exploration, alongside isolated tests of the required execution skills. GPT-6 Astra leads the benchmark, but overall success remains low, and isolated skill performance does not explain the benchmark gap.

4.1 EXPERIMENTAL SETTINGS

  • Five frontier multimodal models were evaluated zero-shot through a shared visuomotor interface, alongside a supervised fine-tuned π0.5 policy.The frontier models received camera images and robot state and acted through ARM, BASE, WAIT, or STOP calls; π0.5 was trained on demonstrations and evaluated on held-out instances.Source: §4 · p. 6, §4.1 · p. 6

4.2 FRONTIER MODELS ON ROBOQUEST

  • 23.2% success rate (SR) makes GPT-6 Astra the overall leader, with the highest SR on 7 of 10 tasks.Table 3 reports SR and mean progress for 50 episodes per cell.Source: Table 3 · p. 7, §4.2 · p. 7
  • Across 500 episodes, evaluating five models cost $21,352 total, from $997 for GPT-6.1 Sol to $9,610 for Fable 5.1; GPT-6.1 Sol was cheapest per episode and per successful episode.Fable 5.1 and Gemini 3.8 Flash were least economical per success.Source: §4.2 · p. 7
  • No model stopped because it was done, and no stopped episode met its goal; the authors interpret stopping and timing out as both giving up.Only 27% of GPT-6 Astra submissions and 16% of GPT-6.1 Sol submissions were correct; Opus 5.5 had the highest submission precision at 30%.Source: §4.2 · pp. 7–8

4.3 EXECUTION SKILLS IN ISOLATION

  • Across isolated skills, the three tested models scored 80%, 81%, and 72%, so execution alone does not explain their benchmark performance differences.Drawer and cabinet opening reached 30–60%, and shim placement 35–60%; GPT-6 Astra opened 90% of compartments in full tasks versus 50% in isolation.Source: §4.3 · pp. 8–9

4.4 FINE-TUNED π0.5

  • Fine-tuned π0.5 performed far worse than frontier agents: it achieved 2.0% success on Puzzle Box and 0.0% success with near-zero progress on the remaining tasks.Evaluation instances used novel kitchen styles, materials, mechanisms, and target configurations absent from training.Source: §4.4 · p. 9

5 FAILURE ANALYSIS

Failure attribution shows that exploration and decision-making, rather than physical execution, account for most failures. Agents often stop before gathering needed evidence, commit prematurely, and rarely prevent or repair disturbances caused by exploration.

5.1 ATTRIBUTING FAILURES

  • Missing evidence accounts for 43 / 43 / 46% of failures and wrong decisions for 23 / 31 / 25%; together they make up 66–74%, versus 18–21% for execution failures.The values are ordered GPT-6 Astra / Opus 5.5 / GPT-6.1 Sol; Figure 5 reports failure attribution by family.Source: Figure 5 · p. 10, §5 · p. 10, §5.1 · p. 11
  • Each unmet requirement is labeled separately at its first unrepaired error, using task-specific deterministic rules and equal episode weighting split across a failed episode’s units.Evidence counts as observed if visible in at least one camera view received by the model.Source: §5.1 · pp. 10–11

5.2 FAILURE PATTERNS

  • An object needed by the task was never placed in 45 / 53 / 50% of failures, often because a hiding place repeatedly seen by the agent was never opened.In 27–29% of all failures, the object remained hidden in an unopened drawer, cabinet, box, or cover.Source: §5.2 · p. 11
  • Roughly 9–11% of failures involved committing objects before gathering necessary evidence, including selecting parcels before weighing or placing stamps without a test print.For Odd Parcel, 71–84% of wrongly selected parcels were boxed before weighing; in Stamp Composition, 20–23% of incorrect placements by two models followed no test print.Source: §5.2 · p. 11
  • Starting objects out of view or covered reduces overall success by 8–10 points, with the additional failures attributed entirely to missing evidence.Wrong decisions, execution failures, and side effects do not increase in these controlled occlusion experiments.Source: §5.2 · p. 11
  • Execution failures account for 18–21% of errors, while several skills work better in full tasks than in isolation, including opening attempted target compartments in 75–90% of cases.The full-task opening rate compares with 30–60% in isolated tests; the authors suggest prior adaptation to controls as one possible explanation.Source: §5.2 · p. 11
  • Side effects cause 7–13% of errors; agents rarely use provided safeguards for spilled balls and recovered only 4 of 38 loose-ball attempts.The bowl was used in only 2–8 of 50 Marked Mugs episodes and never in Wobbly Stand.Source: §5.2 · pp. 11–12
  • Interactive testing remains brittle: hiding Puzzle Box locks barely affects GPT-6 Astra, but Opus 5.5’s success falls from 94.4% to 36.8%.GPT-6.1 Sol falls from 72.2% to 60.1%; in Unfamiliar Containers, Opus 5.5 opens 45% of attempted containers when discovering the mechanism interactively.Source: §5.2 · p. 12

6 CONCLUSION

Across five frontier agents, RoboQuest exposes a gap between physical execution and successful embodied exploration: the agents achieve low task success, often stopping before gathering evidence. The conclusion identifies evidence acquisition, persistent tracking, hypothesis testing, and recovery as needed areas of progress.

6 CONCLUSION

  • Five frontier agents succeed on only a small fraction of episodes: GPT-6 Astra leads at 23.2%, while Gemini 3.8 Flash achieves 2%.The other three agents achieve 11–14%; GPT-6.1 Sol is the most economical per successful episode.Source: §6 · p. 12
  • Most failures arise from exploration and decision-making rather than execution, and the conclusion calls for systematic evidence acquisition, persistent tracking, hypothesis testing, and robust monitoring and recovery.The reported failure patterns include stopping exploration early, committing before acquiring necessary evidence, and disturbances in 7–13% of failures.Source: §6 · pp. 12–13

A SYSTEM PROMPT AND TOOLS

  • The Puzzle Box prompt requires one tool call per response and a physical SUBMIT press to commit; wrong submission, timeout, or stopping without submission fails.The action-tools caption states that positions use world metres and ticks represent 0.05 seconds of simulated time.Source: A SYSTEM PROMPT AND TOOLS · p. 15, Table 6 · p. 16
  • Tool feedback reports acceptance, simulation ticks, and errors, but does not expose simulator state, runtime exceptions, or task progress..Source: A SYSTEM PROMPT AND TOOLS · p. 15

B EVALUATION HARNESS

The evaluation harness provides turn-based access to a simulated mobile-manipulation robot, with observations, actions, memory, and termination handled under fixed episode rules.

B EVALUATION HARNESS

  • At each decision step, agents receive proprioception, remaining decision budget, and three 512 × 512 RGB views from fixed scene and wrist cameras.Source: B EVALUATION HARNESS · p. 16
  • Agents issue exactly one ARM, BASE, or WAIT action per response, specifying a duration in ticks; commands run for the requested ticks without settling time unless SUBMIT is pressed.ARM servos toward a target pose, BASE applies body-frame velocities while holding the arm relative to the base, and WAIT holds both.Source: B EVALUATION HARNESS · p. 16
  • Textual dialogue history persists within an episode, but image history is pruned to the two most recent observations; episodes start with fresh context.Source: B EVALUATION HARNESS · p. 16
  • Episodes end on SUBMIT, exhausted decision budget, STOP, or an invalid response, and are scored strictly at termination.Each episode allows 200 model decisions and up to 200 simulation ticks per command.Source: B EVALUATION HARNESS · p. 16

C π0.5 FINE-TUNING AND EVALUATION DETAILS

The π0.5 policy is fine-tuned separately for each task on task-specific demonstrations. It combines subtask prediction with low-level action generation and replans in closed loop during evaluation.

C π0.5 FINE-TUNING AND EVALUATION DETAILS

  • The policy is fine-tuned separately for each task using that task’s demonstrations, with full-parameter AdamW training, cosine learning-rate scheduling, and EMA weight tracking.Source: C π0.5 FINE-TUNING AND EVALUATION… · pp. 16–17
  • At each decision step, the policy receives three 224 × 224 RGB views, proprioception, and the task instruction, then predicts 20-step action chunks.Each chunk corresponds to 1.0 second at 20 Hz and contains end-effector, gripper, base-velocity, and torso actions.Source: C π0.5 FINE-TUNING AND EVALUATION… · p. 17
  • Training combines autoregressive language modeling for subtask prediction with conditional flow matching for low-level action chunks.The subtask loss is cross-entropy over causal subtask tokens; flow matching is conditioned on vision, state, prompt, and the teacher-forced subtask.Source: C π0.5 FINE-TUNING AND EVALUATION… · p. 17
  • During evaluation, the policy replans every 20 ticks until physical SUBMIT or the task horizon ends the episode.Source: C π0.5 FINE-TUNING AND EVALUATION… · p. 17

D BENCHMARK TASKS

RoboQuest groups ten mobile-manipulation tasks by how agents acquire missing information: searching environments, inspecting objects, or testing how the world responds. Across the tasks, exploration can affect later actions and task success, and demands include evidence integration, memory, planning, and deciding when to submit.

D BENCHMARK TASKS

D.1 SEARCH & EXPLORE

  • Search Room requires choosing and tracking searches; Locked Storage adds token-dependent access and a chain of compartments; Blackout Search requires searching with a carried light.A token can change which locations are searchable, and closing a compartment with a token inside can lock it away permanently.Source: §D.1 · pp. 17–18

D.2 OBJECT INSPECT

  • Marked Mugs requires reading underside labels while preserving and returning contents; Painted Cubes requires inspecting hidden faces; Unfamiliar Containers requires discovering opening mechanisms.Inspection can spill mug contents, while partial cube views may reject but cannot confirm an object and containers may be impossible to open.Source: §D.2 · p. 19

D.3 TESTING

  • Testing tasks require learning through interaction: Stamp Composition tests impressions, Odd Parcel uses balance weighings, Puzzle Box tests part movements, and Wobbly Stand uses a ball to reveal tilt.Some test actions can have lasting consequences: marks on the final stamp surface are permanent, and wrong Puzzle Box moves can jam parts until reversed.Source: §D.3 · pp. 19–20

E FINE-GRAINED CAPABILITIES FOR ROBOQUEST

  • The task set targets directed search, active perception, evidence integration, memory, affordance discovery, experimental identification, physical causal inference, consequential action, and long-horizon planning.Consequential actions can damage the goal, while long-horizon ordering is specifically marked for Locked Storage and Puzzle Box.Source: E FINE-GRAINED CAPABILITIES FOR… · p. 20

F TASK PROGRESS

Task progress is measured on the scene frozen at the first submission, a stop, or the end of the decision budget. The task rules award partial credit for intermediate states, with additional success conditions varying by task.

F TASK PROGRESS

  • Progress is computed on the scene frozen at first submission, a stop, or budget end; only a correct submission succeeds, and precision is the share of submissions that are correct.The episode-ending table reports percentages over 500 episodes per model.Source: F TASK PROGRESS · p. 20, Table 10 · p. 22
  • Search Room awards each target staged credit for opening its compartment, taking it out, and placing it upright on the tray; Locked Storage shares progress between opened locks and the target.Blackout Search uses the Search Room rule; non-target objects on the tray reduce Search Room and Locked Storage progress.Source: F TASK PROGRESS · pp. 20–21
  • Marked Mugs scores placement by overlap with the label-colored pad and separately scores whether each vessel retains its own two balls.Full success requires both scores to equal 1 for every vessel, with placement counted only while the vessel is upright, released, still, and supported by the pad.Source: F TASK PROGRESS · p. 21
  • Painted Cubes progress is the fraction of cubes captured in the correct bin, while Unfamiliar Containers awards staged credit as items are opened, removed, and placed in the bowl.Painted Cubes captures are irreversible.Source: F TASK PROGRESS · p. 21
  • Stamp Composition progress rewards correctly filled target cells and penalizes stray dots or excessive ink outside the grid.A cell receives full credit for clean, centered dots within its allowed count and partial credit for edge-crossing or excess dots; success also requires stamps released, still, and supported.Source: F TASK PROGRESS · p. 21
  • Odd Parcel progress compares the odd-parcel set with parcels released in the answer box; an empty box scores 0, while parcels left on the pans are not penalized.Moving the balance more than 2 cm fails the episode.Source: F TASK PROGRESS · p. 21
  • Puzzle Box progress is shared equally across the chain bolts and item, with staged item credit as the lid opens, the item leaves the box, and it reaches the tray.Success also requires that the robot touch no part when submitting.Source: F TASK PROGRESS · p. 22
  • Wobbly Stand progress combines final leveling with whether the ball is on the stand, and touching the stand sets the leveling component to 0.A level stand without the ball scores 0.5; an untouched scene scores 0.Source: F TASK PROGRESS · p. 22

G EXECUTION SKILL TEST DETAILS

The skill tests isolate selected execution abilities in scenes based on evaluation instances, while omitting task-level exploration and submission. Comparisons with full tasks account for differences in starting conditions and information available.

G EXECUTION SKILL TEST DETAILS

  • Each skill test keeps the evaluation kitchen but includes only one object and its target, omits SUBMIT, and checks the goal after each physics tick.Each model plays the same 20 scenes per skill; otherwise an episode ends after 50 decisions.Source: G EXECUTION SKILL TEST DETAILS · p. 22
  • Pick-and-place and driving are the general skills; drawer and cabinet tests use Search layouts, while several counter tasks have no dedicated skill test.The omitted dedicated tests are for Marked Mugs, Odd Parcel, Stamp Composition, and Puzzle Box because their tasks mostly involve counter manipulation covered by the general skills.Source: G EXECUTION SKILL TEST DETAILS · p. 22
  • The full-task comparisons distinguish reaching a target from completing a skill and limit opening comparisons to settings with matching starts.Full-task retrieval and shim placement also include finding objects, while isolated tests name the opening mechanism or leg and provide a single shim.Source: G EXECUTION SKILL TEST DETAILS · pp. 22–23, Table 11 · p. 22, Table 12 · p. 23

H DEMONSTRATION DATASET DETAILS

The dataset provides successful scripted demonstrations for all ten tasks, with language annotations at stage and subtask levels. The accompanying workload and occlusion analysis defines the varied difficulty parameters and reports success, progress, and failure families by level.

H DEMONSTRATION DATASET DETAILS

  • Each frame stores three camera views, robot state, action, and two levels of language annotation; hidden information is not named before it is seen.Stages contain finer subtasks, each control step belongs to exactly one subtask, and objects are named by their appearance and position from the robot's viewpoint.Source: H DEMONSTRATION DATASET DETAILS · p. 23
  • The release uses LeRobot v2.1, with one dataset per task and 256×256 video at 20 Hz, 16-dimensional robot state, and 12-dimensional actions.Release scripts can regenerate datasets at other resolutions from recorded states.Source: H DEMONSTRATION DATASET DETAILS · p. 23
  • Every task has 500 successful, verified episodes, and Table 13 summarizes oracle information gathering, episode length, interaction time, and mean subtask counts.The scripted oracles perform exploration despite knowing the hidden information.Source: H DEMONSTRATION DATASET DETAILS · p. 23, Table 13 · p. 24

I WORKLOAD AND OCCLUSION DETAILS

  • Occlusion levels distinguish objects visible initially, out of view until the robot moves, or hidden under a removable cover; workload counts required units such as cubes or lock-chain depth.Table 14 reports success and progress by level, Table 15 failure families by level, and Figure 7 Painted Cubes error rates by rule.Source: I WORKLOAD AND OCCLUSION DETAILS · p. 24, Table 14 · p. 24

J FAILURE ATTRIBUTION DETAILS

Failure attribution assigns unmet requirements to failure families using deterministic rules applied to recorded run logs. Missing evidence remains the largest family for every model, and missing evidence plus wrong decisions account for 66–78% of failures under both tested aggregation schemes.

J FAILURE ATTRIBUTION DETAILS

  • Figure 7 reports false-negative and false-positive rates by rule for committed cubes, in percent.Source: Figure 7 · p. 25
  • Each unmet requirement in a failed episode is a separate unit, and each episode’s weight is divided evenly across its units when computing family shares.The analysis includes 931 units for GPT-6 Astra, 1,143 for Opus 5.5, and 1,182 for GPT-6.1 Sol.Source: J FAILURE ATTRIBUTION DETAILS · pp. 25–26
  • Missing evidence is the largest failure family for every model, while missing evidence and wrong decisions together account for 66–78% of failures under both aggregation schemes.No family share shifts by more than 7 percentage points between schemes.Source: J FAILURE ATTRIBUTION DETAILS · p. 25
  • A deterministic, ordered rule set assigns each unit to the first matching failure family after tracing events to the first error that was not repaired.An unnoticed error is classified as a wrong decision if its deciding evidence was observed, and as missing evidence otherwise.Source: J FAILURE ATTRIBUTION DETAILS · p. 26
  • Premature stopping is missing evidence before the deciding observation and a wrong decision after it; unrepaired scene damage is a side effect.A failed physical attempt to uncover evidence counts as execution failure, while the logs cannot determine whether unrepaired damage was unforeseen or clumsy execution.Source: J FAILURE ATTRIBUTION DETAILS · p. 26
  • Observed-object status uses fixed pixel-count and brightness thresholds on delivered views, with a lower brightness threshold in Blackout Search.The thresholds are at least 150 visible pixels and mean brightness of 25, or 3 in Blackout Search; attribution also uses logged simulator states, camera views, and interaction events.Source: J FAILURE ATTRIBUTION DETAILS · pp. 25–26
  • Table 17 separates missing-evidence and wrong-decision failures by physical action, including whether an object was never placed or placed wrongly.An object is placed wrongly if committed before deciding evidence is uncovered or contrary to observed evidence.Source: J FAILURE ATTRIBUTION DETAILS · p. 27, Table 17 · p. 27
Loading 2610.10388v1…