PaperlayerView exact revision on arXiv

Source-linked AI summary

Spotter: Let the Embodied Model Lead, and the VLM Reflect for It

Long Li, Qichao Zhao, Yue Yang, Fan Xu, Zhe Wang, Alan Wee-Chung Liew, Chao Qu, Heng Tao Shen, Shirui Pan

arXiv:2609.36808v1cs.ROcs.AI

TL;DR

Embodied models rarely recover from their own failures, and scorers struggle to recognize a successful correction after failure. Spotter keeps the embodied model executing while a parallel VLM monitors and repairs confirmed errors. With GPT, it improves both tested embodied models on RoboCasa, while irreversible environmental damage remains beyond its recovery scope.

  • Problem

    Embodied models rarely correct failures, and after a failure their scorers cannot reliably identify the successful candidate, leaving recovery from mistakes unresolved.

    Source: §1 · p. 1

  • Method

    Spotter lets the embodied model execute while a parallel VLM uses a lightweight screener to detect errors, intervene with a repair, and check its outcome.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    With the GPT judge, Spotter improves Cosmos Policy and π0.5 by 5.6 and 7.5 points on RoboCasa, respectively.

    Source: §4.2 · p. 9

  • Takeaways & Limitations

    Spotter shows that a VLM can supervise and repair errors without taking control of routine execution or requiring policy training.

    Source: §1 · p. 2

    Spotter cannot recover when an action makes the task irreversible, such as dropping an object out of view.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Current embodied models do not respond to their own failures, although what just went wrong could inform a small adjustment on the next attempt, the kind of reflection behind the gains of thinking in language models. We test whether they can repair a known error, which requires producing a correction and judging whether it is right. Stopped at a failure and allowed to retry, they seldom repair it through their own randomness or from a language description of the error, and best-of-N selection cannot pick the successful candidate after a failure. We attribute this to training only on successful demonstrations and to inputs too narrow to show what went wrong, and conclude that reflection must come from a vision-language model (VLM), which takes in far more information, such as the episode history and text, and is more general. Prior VLM-led work has the VLM plan every step and invoke the embodied model as a tool, placing the VLM on the critical path. We propose Spotter, which reverses the roles: the embodied model leads and executes continuously, while the VLM runs in parallel, monitors through a lightweight local screener, intervenes only when an error is detected, reflects on and corrects it, and returns control. We run Spotter with Qwen and with GPT as the VLM, and both improve the embodied models; with GPT, Spotter improves Cosmos Policy and $\pi_{0.5}$ by 5.6 and 7.5 percentage points on RoboCasa, and raises $\pi_{0.5}$ from 47.2% to 57.0% on the Hard setting of RoboTwin 2.0 and from 53% to 83% on a real robot. Because the VLM steps in only when an error is confirmed, a successful episode with Qwen takes only 13 to 16 s longer than with the embodied model alone and about 70% less time than with a VLM-led baseline using the same model. Our code is available at https://github.com/zqc3117/Spotter.

1 INTRODUCTION

Embodied models often fail to notice and repair mistakes, while recognizing a good correction also proves difficult. Spotter keeps the embodied model in control and uses a VLM to monitor and intervene when errors arise, unlike prior VLM-led approaches that put the VLM on the critical path.

1 INTRODUCTION

  • Embodied models rarely recover from missed grasps or premature releases, even though manipulation errors often become visible only after execution.The policy may continue as if an object were grasped when the grasp missed.Source: §1 · p. 1
  • After a failure, embodied models seldom produce a correction or identify a successful candidate; recovery fine-tuning also fails to help.The paper attributes this to training on successful demonstrations and scarce recovery data, while current-state inputs omit the episode history needed to distinguish recovery from repeated failure.Source: §1 · pp. 1–2
  • After the illustrated missed grasp, Spotter notices only after the arm has left the pan, then returns it to its earlier pose and shifts the gripper toward the lime.Source: Figure 1 · p. 2
  • Prior VLM-led approaches put the VLM on the critical path, with the embodied model waiting during deliberation and no supervision while it acts.Fixed action stretches can either waste time on unnecessary checks or let a policy proceed far into a failure.Source: §1 · p. 2
  • Spotter runs the embodied model continuously while a lightweight screener monitors chunks and a VLM intervenes to repair confirmed errors.The VLM also checks whether the repair worked before returning control; execution pauses if the error is confirmed or the supervisor reaches its lead limit.Source: §1 · p. 2

2 RELATED WORK

Prior robot-agent systems generally put a language model in charge of planning or supervision, while failure-recovery work also includes methods that learn from failures. Spotter instead keeps the embodied model leading and has a VLM supervise and repair errors in parallel.

Robot Agents

  • Existing robot-agent systems use a VLM to issue commands or plans, invoke an embodied model as a tool, or evaluate actions, so execution waits on the language model.Spotter instead runs the VLM in parallel and pauses only for a confirmed error or the bounded lead limit.Source: §2 · p. 3

Failure Explanation and Recovery

  • Failure-recovery approaches either replan at the skill level or train multimodal models on failure data; Spotter requires neither failure data nor additional training.Spotter’s VLM repairs errors with motion primitives rather than returning an instruction for the embodied model to execute.Source: §2 · p. 3

Test-Time Scaling

  • Embodied test-time scaling ranks sampled candidates with learned or external scorers, but these methods assume the scorer can identify a good candidate.The paper tests that assumption after failure and finds it does not hold.Source: §2 · p. 3

3 METHOD

The section evaluates whether embodied models can recover from a known failure and judge candidate corrections, then presents Spotter, an embodied-led framework for parallel VLM supervision and repair. Spotter screens execution, bounds how far the policy can get ahead of the supervisor, and verifies repairs; its asynchronous operation cannot undo irreversible environmental damage.

3.1 ERROR-CORRECTION ABILITY OF CURRENT EMBODIED MODELS

  • The evaluation interrupts execution at failures and tests correction through resampling, language feedback, and recovery fine-tuning, allowing up to three attempts.For the scorer test, candidates are ranked in pairs at post-failure and normal states; a score of 0.5 indicates random ranking.Source: §3.1 · pp. 4–5
  • Fewer than one in ten failures is corrected with three attempts, and after failure test-time scorers cannot reliably select the successful candidate.Recovery fine-tuning yields 8.13% for Cosmos Policy and 8.44% for π0.5; scorers that show skill at normal states lose it after a failure.Source: §3.1 · pp. 4–5
  • The VLM judge given only images ranks successful candidates only 53% of the time with GPT-6 Astra and 51% with Qwen after failure.The paper argues that recovery assessment needs episode history, including prior attempts, telemetry, and actions, rather than only the current images.Source: §3.1 · p. 5

3.2 SPOTTER

  • Spotter keeps the embodied policy frozen and executing while a parallel VLM supervisor monitors camera images, robot state, and commanded actions.The supervisor does not receive privileged simulator state or the task’s success signal.Source: §3.2 · pp. 5–6
  • The screener checks completed chunks, while a lead limit K bounds the policy’s unchecked progress and each delayed check includes all unchecked completed chunks.A locally deployed Qwen screener flags chunks using a risk threshold; empty-gripper evidence and failed screening calls also trigger flags.Source: §3.2 · p. 6
  • The judge may return an ok, intervene, or give-up verdict; an intervention uses a short primitive plan and an expectation that the judge checks in the next supervised window.If the expectation fails, the next intervention must try a different kind of repair; the policy resumes from the repaired state without rollback.Source: §3.2 · pp. 6–7
  • Spotter can undo faulty actions or recover from repairable damage, but cannot continue when an action makes the task irreversible, such as dropping an object out of view.The paper notes that the compared paradigms cannot recover from such environmental damage either.Source: §3.2 · p. 7
  • At test time Spotter returns the robot arm to a recorded pose rather than resetting the scene or simulator.The simulator reset is used only while learning general lessons; nothing is reset at test time.Source: §3.2 · p. 7

4 EXPERIMENTS

Experiments compare Spotter with embodied-model baselines across RoboCasa, RoboTwin 2.0, and a real Franka robot, and measure supervision cost. Spotter improves success in the reported settings while adding relatively little time on successful episodes.

4.1 EXPERIMENTAL SETUP

  • RoboCasa covers 24 tasks with 50 episodes each, RoboTwin 2.0 covers 50 tasks with 10 episodes each, and real-robot tests use three tasks with matched scenes, instructions, and budgets.Spotter is evaluated zero-shot and with one learned demonstration; paired episodes use fixed seeds and matched budgets.Source: §4.1 · pp. 7–8, §4 · p. 7

4.2 MAIN RESULTS

  • On RoboCasa, GPT Spotter gains 5.6 points for Cosmos Policy and 7.5 for π0.5; on RoboTwin 2.0, π0.5 rises from 47.2% to 57.0%.The RoboCasa comparisons are against each embodied model under the same budget; Figure 4 compares success rates on RoboCasa and RoboTwin 2.0.Source: §4.2 · p. 9, Figure 4 · p. 9
  • Adding Spotter (GPT) raises real-robot average success from 53% (16/30) to 83% (25/30) across three Franka tasks.The largest gain is on the long-horizon task, from 30% to 70%.Source: §4.2 · pp. 9–10, Figure 5 · p. 9

4.4 COST OF SUPERVISION

  • On successful episodes, Qwen Spotter adds 13 s on Cosmos Policy and 16 s on π0.5; failed episodes take 4.2× and 5.0× as long as without Spotter.Figure 6 compares episode time for the embodied model alone, Spotter, and Harness VLA.Source: §4.4 · p. 10, Figure 6 · p. 10
  • With GPT-6 Astra, the screener cuts judge calls per episode by 37% and tokens by 66%, at a 2.0-point success-rate cost.Table 5 reports judge calls and executed repairs per episode across configurations.Source: Table 5 · p. 10, §4.4 · p. 11

5 ANALYSIS

Spotter directs VLM attention to detected errors, where it can correct some failures, while avoiding the fixed, frequent interruptions of VLM-led supervision. The analysis also shows that a single demonstration can shift success rates and reduce oversight costs.

Why VLM-led supervision is slow where it is not needed

  • 146 s on successful Cosmos Policy episodes for Harness VLA (Qwen) versus 47 to 62 s for Spotter (Qwen) and 34 s for the policy alone.Spotter episodes are only 13 to 28 s longer than policy-only episodes; the VLM-led approach makes fixed-duration calls and can waste time checking when nothing is wrong.Source: §5 · p. 11

How often does the judge correct an error?

  • About 85% of Qwen judge interventions target scenes the embodied model alone fails, and GPT-6 Astra as judge raises this further.Figure 7 defines detection as the share of interventions on scenes the embodied model alone fails.Source: Figure 7 · p. 11, §5 · p. 11
  • A single Spotter repair corrects about 10% to 20% of errors, versus 7.5% to 9.7% with three embodied-model attempts and an oracle that found the error.Figure 7 reports single-repair correction, unlike the three-attempt comparison in Section 3.1.Source: Figure 7 · p. 11, §5 · p. 11
  • With one learned example each, GPT corrects 19.2% of errors against 13.5% for Qwen.Source: §5 · p. 11

Why the single demonstration does not always show up in the average

  • On π0.5, one demonstration adds 2.0 points in all three columns; on Cosmos Policy, the aggregate shifts only from 71.7 to 71.8.For Cosmos Policy, the demonstration is worth +4.2 points on eight pick-and-place tasks and −1.9 on sixteen others, which carry two thirds of the weight.Source: §5 · p. 11
  • Despite 2.5× and 2.2× the tokens, the demonstration changes episode time by −4% and 0%, respectively, and reduces judge calls by 7% per episode on both models.The call reduction is mainly associated with π0.5’s 2.0-point success-rate rise and 15% fewer calls on failed Cosmos Policy episodes.Source: §5 · p. 12

6 CONCLUSION

Embodied models trained only on successful demonstrations rarely repair failures and their scorers lose their ability to choose after a failure. Spotter keeps the embodied model executing while a VLM monitors and intervenes on flagged errors, but first-occurrence errors that ruin the scene remain out of reach.

6 CONCLUSION

  • Embodied models trained only on successful demonstrations rarely repair a failure, even when it is caught for them, and their scorers lose their skill after policy failure.Source: §6 · p. 12
  • Spotter uses a general VLM to read episode history, telemetry, and past attempts, then adjust from failure and check whether a repair worked.Source: §6 · p. 12
  • The embodied model runs continuously while a lightweight screener watches; the VLM handles flagged windows and repair verification, limiting its intervention to cases where it is needed.Source: §6 · p. 12
  • Errors that ruin the scene the first time they occur remain out of reach; the paper identifies anticipating them, perhaps with a world model, as a next step.Source: §6 · p. 12

A IMPLEMENTATION DETAILS OF THE ERROR-CORRECTION EVALUATION

The error-correction experiments use official Cosmos Policy and π0.5 checkpoints in closed-loop RoboCasa. Privileged simulator quantities are reserved for the evaluation harness rather than the models or supervisor.

A IMPLEMENTATION DETAILS OF THE ERROR-CORRECTION EVALUATION

A.1 PRODUCING A CORRECTION

The evaluation detects attempted grasp failures at chunk boundaries, then tests recovery by resampling or language feedback from the failure state. Correction is measured by whether an attempt lifts the object.

A.1 PRODUCING A CORRECTION

  • A grasp attempt requires the gripper to approach within 5 cm of the target and change aperture by at least 0.008 within the chunk.Source: §A.1 · p. 16
  • A grasp fails if the object is not held at chunk end, or was held at midpoint but dropped by the end.Source: §A.1 · p. 16
  • Resampling restores the state at the start of the failed chunk, runs three new-seed rollouts to episode end, and counts correction if any lifts the object.Source: §A.1 · p. 16
  • Language feedback stops without rewinding, gives π0.5 a VLM explanation and imperative instruction, and repeats failed attempts for at most three rounds.Success in any round counts as corrected; the protocol is run with π0.5 because its language input accepts free-form text.Source: §A.1 · p. 16
  • The correction rate averages whether each oracle-detected failure is followed by an object lift across up to three attempts.Source: §A.1 · p. 17

A.2 JUDGING A CORRECTION

The experiment tests whether scorers can identify successful grasp candidates sampled from saved simulator states. It compares post-failure states with states just before a successful grasp, using the same candidate count, horizon, labeling rule, and scorers.

Saved states and candidates

  • Eight candidates are sampled from each saved state, run for 32 steps, and labeled by the benchmark grasp predicate at step 32, retaining earlier successes.The candidates use the same policy and checkpoint but different random seeds.Source: §A.2 · p. 17

Triggers

  • Post-failure states are saved immediately after an unsuccessful grasp, while normal-regime states are saved just before a successful grasp.Both regimes use the same candidate count, horizon, labeling rule, and scorers; only the trigger differs.Source: §A.2 · p. 17

Metric

  • Ranking accuracy is the fraction of successful-versus-failed candidate pairs scored in the correct order, with ties counting as one half.Only states containing both successful and failed candidates are included.Source: §A.2 · p. 17

Sample counts

  • The post-failure regime supplies 991 usable states and 10,974 pairs, versus 432 states and 4,853 pairs in the normal regime.Confidence intervals are 95% intervals from 1,000 bootstrap resamples over states; normal-regime states are rarer because candidates near grasp completion almost always succeed together.Source: §A.2 · p. 17

Scorers

  • Scorers include the policy value head, imagined-future consistency measures, geometric reprojection, re-noising consistency, and a VLM judge.The VLM judge scores each candidate's grasp likelihood from images of the lead-in chunk and imagined future, without episode history; Consistency-Exploring uses executed outcomes and serves only as a diagnostic upper bound.Source: §A.2 · p. 17

B DETAILED EXPERIMENTAL SETUP

The experimental setup evaluates Spotter on simulated manipulation benchmarks under specified task, episode, and step-budget protocols. It also describes supervision, prompts, learning resources, model serving, and a running-cost comparison with Harness VLA.

Tasks and seeds

  • RoboCasa evaluation uses 1,200 episodes across 24 official tasks; RoboTwin 2.0 uses 50 Hard-setting tasks with 10 episodes each.The RoboCasa example library is mined separately and remains read-only during evaluation; π0.5 is trained on clean data only for RoboTwin 2.0.Source: B DETAILED EXPERIMENTAL SETUP · pp. 17–18

Step budget

  • RoboCasa's main comparison gives both baseline and Spotter 1.8× the official step limit, counting repair motions, gripper actions, and calibration.Official limits vary by task; RoboTwin 2.0 limits range from 400 to 1,700 steps.Source: B DETAILED EXPERIMENTAL SETUP · p. 18

Chunks and screening

  • Cosmos Policy executes 16-step chunks and screens every two chunks, while π0.5 executes 25 of each 50-step output and screens every chunk.The supervisor's lead is bounded to four Cosmos Policy chunks or two π0.5 chunks.Source: B DETAILED EXPERIMENTAL SETUP · p. 18

Inputs to the supervisor

  • The supervisor receives time-ordered camera montages, a fresh checkpoint image, instructions, action history, telemetry, and any retrieved lessons or examples.It receives no object pose or success signal; an enabled learned example can contain up to 12 images.Source: B DETAILED EXPERIMENTAL SETUP · p. 18

Serving and timing

  • Qwen3.8-27B-FP8 runs on four NVIDIA H20 GPUs, while GPT-6 Astra uses the Responses API and Spotter's local Qwen screener.The timing table reports time and tokens per episode, including overall, successful, and failed episodes; GPT times are omitted because they include network latency.Source: B DETAILED EXPERIMENTAL SETUP · pp. 18–19, Table 6 · p. 19

Learning

  • The frozen general-lesson library contains 77 entries, while example retrieval prefers the same policy and task before broader task matches.The lessons are learned on training data or non-test seeds; task-specific privileged information is removed.Source: B DETAILED EXPERIMENTAL SETUP · p. 18

Compared methods

  • Figure 4 compares Spotter-Qwen and Spotter-GPT against other embodied models on RoboCasa and RoboTwin 2.0.Spotter uses one learned example under 1.8× the step limit, on Cosmos Policy for RoboCasa and π0.5 for RoboTwin 2.0.Source: B DETAILED EXPERIMENTAL SETUP · p. 19

Harness VLA

  • With the same step limit as other methods, Harness VLA's success rate falls to about 50%, using 1,802 steps per episode on average.Its interaction rounds spend many steps while tasks are underway, so the limit is reached halfway through.Source: B DETAILED EXPERIMENTAL SETUP · p. 19

C PRIMITIVE LIBRARY

The primitive library supports repair plans by translating selected move targets into executable motion targets.

Move targets

  • A move target is either a world-coordinate vector or a pixel in a selected camera view, optionally with a vertical offset.For pixel targets, the executor samples calibrated depth at that pixel and applies inverse camera projection.Source: C PRIMITIVE LIBRARY · p. 19

Pixel-target execution

  • Pixel-target execution uses a single depth sample rather than a median over a bounding box.Depth remains internal to the executor and is not shown to the VLM.Source: C PRIMITIVE LIBRARY · p. 19

Target conversion

  • The executor adds the optional vertical offset after projecting the pixel-derived target into world coordinates.Source: C PRIMITIVE LIBRARY · p. 19
Loading 2609.36808v1…