PaperlayerView exact revision on arXiv

Source-linked AI summary

Future Anchored Verification and Online Recovery for World Action Models

Zhibin Qin, Zhenxiong Tan, Xinchao Wang

arXiv:2610.06280v1cs.ROcs.AI

TL;DR

When execution drifts from a WAM’s predicted future, remaining actions may no longer be trustworthy, and existing monitors do not say what state to restore. FAVOR uses the predicted future to verify execution and guide recovery through corrective instructions. It raises base-WAM task success on LIBERO and LIBERO-Plus.

  • Problem

    When execution deviates from a WAM’s predicted future, the remaining actions are invalid, while existing monitors flag when to stop but not what state to restore.

    Source: §1 · p. 1

  • Method

    FAVOR keeps predicted frames as anchors: an Anchor Verifier detects task-breaking deviations, and Anchor-Guided Recovery translates an alarmed anchor into a corrective instruction executed with strengthened instruction guidance.

    Source: §1 · p. 2

  • Results

    97.85% to 98.10% on LIBERO and 72.60% to 72.98% on LIBERO-Plus: FAVOR raises base-WAM task success without modifying the policy.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The future a WAM predicts before acting can serve both as a reference for verifying execution and as the target for recovery.

    Source: §1 · p. 2

Abstract

from arXiv · show

World action models (WAMs) have emerged as a promising paradigm for robotic manipulation. They act by first predicting how a task should be performed and then decoding the actions from that future. However, the remaining actions are invalid once execution drifts from the prediction. Simply replanning from the already out of distribution state rarely restores what the task still requires; existing execution monitors decide when to stop, but not what to restore. We observe that the answer is already in hand: the future the WAM predicted before acting depicts exactly the states it intended to pass through. We introduce FAVOR (Future Anchored Verification and Online Recovery), a lightweight framework that keeps these predicted frames as anchors and uses them for verification and recovery. An Anchor Verifier compares each observation with its anchor, together with the executed actions, to flag deviations that break the task. Anchor-Guided Recovery uses a vision-language model to turn the flagged anchor into a short corrective instruction. Under strengthened instruction guidance, the WAM executes this instruction to return to the intended future. It then resumes the task. FAVOR raises the task success of the base WAM from 97.85% to 98.10% on LIBERO and from 72.60% to 72.98% on LIBERO-Plus without modifying the policy.

1 INTRODUCTION

FAVOR reuses a WAM’s predicted future as an anchor for both execution verification and recovery. Its verifier flags task-breaking deviations, while recovery turns an alarmed anchor into a corrective instruction for the frozen WAM.

Motivation

  • Existing monitors flag when actions are no longer trustworthy but do not specify what state to restore; replanning from a deviated state offers little help.The paper notes that WAMs are trained on successful demonstrations, making predictions from unfamiliar states unreliable.Source: §1 · p. 1
  • A deviation invalidates the remaining actions, but the states the WAM intended to establish remain useful recovery targets because they lie within the policy’s training region.The predicted future can show the condition that was lost, such as a mug held in the gripper.Source: §1 · p. 1

Method

  • The Anchor Verifier compares each observation with its anchor alongside executed actions, tracking mismatches to identify deviations that break the task rather than any visual difference.This addresses the possibility that generative anchors contain visual artifacts.Source: §1 · p. 2
  • Anchor-Guided Recovery gives a vision-language model the alarmed anchor, current observation, and task to produce a short corrective instruction, which the frozen WAM executes under strengthened instruction guidance.The instruction names the condition to restore rather than pixels to reproduce.Source: §1 · p. 2

Results

  • 97.85% to 98.10% on LIBERO and 72.60% to 72.98% on LIBERO-Plus: FAVOR improves base-WAM task success; its verifier reaches 0.992 AUROC.The evaluation compares FAVOR with the base WAM on both benchmarks.Source: §1 · p. 2

Contribution

  • FAVOR’s future-anchored view uses the WAM’s predicted future both to verify execution and as the target that recovery should restore.The method keeps predicted frames as anchors for both functions.Source: §1 · p. 2

2 RELATED WORK

Prior WAM work uses predicted futures to condition action generation, but typically does not reuse them after actions are produced. Related monitoring methods assess whether to continue, while recovery methods obtain targets externally; FAVOR instead derives its recovery objective from the plan already being executed.

2.1 WORLD ACTION MODELS AND FUTURE PREDICTION

  • Predicted futures condition action generation in WAMs, but generally play no further role after actions are produced; FFDC retains them to judge action trustworthiness and shorten execution or replan.Other approaches use future prediction as explicit conditions or couple video and action generation.Source: §2.1 · p. 3

2.2 EXECUTION VERIFICATION AND RECOVERY

  • Monitoring methods can flag that something went wrong but do not specify a return state; replanning then starts from the deviated state.Dense correspondence measures motion rather than task relevance, while HarnessWAM monitors subgoal completion rather than the prediction.Source: §2.1 · p. 3, §2.2 · p. 3
  • Existing recovery methods obtain objectives from external planners, human instructions, failure-recovery data, or auxiliary models, whereas FAVOR uses the WAM’s own predicted plan and strengthens corrective instructions at inference time.The VLM translates the future into an instruction rather than estimating a recovery target; strengthened guidance makes that instruction shape future generation without retraining.Source: §2.2 · p. 3

3 PRELIMINARIES

A WAM predicts the visual states a task should pass through, then decodes actions to realize them. The predicted frames align with action groups, allowing observations to be checked against intended states.

3 PRELIMINARIES

  • A WAM predicts task states before decoding actions; this work builds on the IDM variant of Fast-WAM.The model factorizes an action chunk through a sequence of visual states.Source: §3 · pp. 3–4
  • A diffusion transformer trained by flow matching predicts future-frame latents conditioned on the current latent and task instruction.At inference, integration proceeds backward from noise to a clean latent, yielding H/∆ predicted frames spaced ∆ control steps apart.Source: §3 · p. 4
  • The IDM infers actions from the current latent and predicted future, conditioned on the task instruction.The action tokens are denoised with flow matching.Source: §3 · p. 4
  • Each group of ∆ consecutive actions targets the transition to the next predicted frame, which can be checked against the observation at that horizon.The first anchor is the current observation.Source: §3 · p. 5

4 METHOD

FAVOR uses WAM-predicted frames as anchors for both execution verification and recovery. The Anchor Verifier tracks task-relevant deviations, while Anchor-Guided Recovery converts an alarmed anchor into a corrective instruction and returns control when the anchored condition is restored.

4 METHOD

  • FAVOR uses predicted frames as anchors for verification and recovery, with the verifier checking deviation and recovery guiding the robot back on course.The method uses anchors twice: to check whether execution has deviated and to guide the return to the intended state.Source: §4 · p. 5

4.1 ANCHOR VERIFIER

  • The Anchor Verifier evaluates observation–anchor mismatch alongside commanded actions and its evolution across a chunk, rather than relying on fixed image distance.It raises an alarm when the deviation score exceeds a threshold fixed before evaluation.Source: §4.1 · p. 5
  • Verifier training combines locally labeled clean-versus-perturbed executions with episode-labeled rollouts and demonstrations, adapting from local supervision to terminal outcomes.The loss encourages at least one high-scoring anchor in failed episodes, while suppressing the strongest and average scores in successful episodes.Source: §4.1 · pp. 5–6

4.2 ANCHOR-GUIDED RECOVERY

  • At an alarm, Anchor-Guided Recovery discards the chunk’s unexecuted actions, uses a VLM to turn the alarmed anchor into a corrective instruction, and applies strengthened instruction guidance.At the end of the recovery window, the VLM checks whether the anchored condition is restored; if so, control returns to the policy.Source: §4.2 · p. 6

5 EXPERIMENTS

FAVOR is evaluated on LIBERO and LIBERO-Plus, with native task success as the primary metric. Results cover end-to-end performance, qualitative recoveries, detector comparisons, and recovery-guidance ablations.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses 100 rollouts per LIBERO task and 1698 total LIBERO-Plus rollouts, with native task success computed over the full cohort.The metric includes episodes without intervention, crediting successful recoveries while penalizing disruptive interventions.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • FAVOR raises base-WAM success from 97.85% to 98.10% on LIBERO and from 72.60% to 72.98% on LIBERO-Plus.It cuts LIBERO failures by 11.6% relative; the gain is largest on Object and Long tasks.Source: §5.2 · p. 7
  • Two qualitative LIBERO cases show recovery restoring an alarmed anchor and completing tasks that fail under native continuation.The figure contrasts both branches from the same simulator state at the alarm.Source: §5.2 · p. 7, Figure 4 · p. 8

5.3 ABLATION STUDIES

  • The Anchor Verifier leads detector comparisons on all three metrics, while learned anchor comparisons improve AUPRC by 0.12 and failure coverage by 24 points.Fixed distances rank episodes well but have low AUPRC and coverage; the verifier separates successful and failed episodes with a threshold.Source: §5.3 · pp. 8–9, Table 2 · p. 9, Figure 5 · p. 9
  • Recovery guidance must be strong enough to make the corrective instruction prevail: γ = 1 ignores it, γ = 3 registers but does not follow it, and γ = 5 realizes it.At γ = 5, the predicted future shows the arm grasping the cup, and the IDM produces corresponding recovery actions.Source: §5.3 · p. 9

6 CONCLUSION

FAVOR adds verification and recovery to a frozen WAM by reusing its predicted future. It reduces failures and improves end-to-end success on LIBERO and LIBERO-Plus without modifying the policy.

6 CONCLUSION

  • FAVOR equips a frozen WAM to detect task-breaking deviations and use its predicted future to guide recovery to a state where the task can continue.On LIBERO and LIBERO-Plus, it reduces failures and improves end-to-end success without modifying the policy.Source: §6 · p. 9

A COMPLETE CONTROLLER

Algorithm 1 runs a frozen WAM with a verifier and VLM, checking predicted states during action chunks. On an alarm, it discards the remaining action suffix and attempts recovery before resuming the original task instruction.

A COMPLETE CONTROLLER

  • The controller predicts future states and an action chunk, executes actions, and periodically verifies observations against corresponding predicted states.Verification occurs at interval Δ within a chunk of horizon R.Source: A COMPLETE CONTROLLER · p. 17
  • When the verifier score exceeds threshold τ, the controller selects that predicted state as an anchor and asks the VLM for a corrective instruction.The VLM receives the current observation, anchor, and task instruction.Source: A COMPLETE CONTROLLER · p. 17
  • The WAM executes the corrective instruction with guidance scale γ for B steps, repeating VLM instruction updates until the anchor is restored or the budget is exhausted.After intervention, the task instruction becomes active again and the remaining action suffix is discarded.Source: A COMPLETE CONTROLLER · p. 17
  • If no alarm occurs, the controller checks for task success and otherwise updates the observation and replans after the chunk.It returns immediately when the task succeeds.Source: A COMPLETE CONTROLLER · p. 17

B IMPLEMENTATION DETAILS

The implementation appendix describes default settings, verifier data construction, and verifier training refinements. It distinguishes local deviation labels from terminal outcomes in complete rollouts.

B IMPLEMENTATION DETAILS

  • Table 3 lists the default FAVOR settings referenced symbolically in the main text.The supplied passage does not enumerate the settings themselves.Source: B IMPLEMENTATION DETAILS · p. 17, Table 3 · p. 17
  • Verifier training snippets pair clean and perturbed executions from the same simulator state against a shared imagined reference.Perturbations add Gaussian noise or scale arm-command magnitudes over the final eight steps, leaving gripper commands unchanged; local labels derive from object and end-effector displacements.Source: B IMPLEMENTATION DETAILS · p. 17
  • Complete rollouts record imagined frames, planned and executed actions, synchronized camera observations, robot and object states, and terminal outcomes.Observations align with imagined frames after actions 4, 8, 12, and 16; incomplete trailing chunks are excluded.Source: B IMPLEMENTATION DETAILS · p. 18
  • Episode-level training warm-starts from a locally supervised checkpoint and optimizes the episode loss for five epochs.Training batches up to seven successes with one cyclically sampled failure; the local-label refinement adds reviewed anchor labels without changing inference architecture.Source: B IMPLEMENTATION DETAILS · p. 18

C VLM PROMPTS FOR DIAGNOSIS AND RECOVERY INSTRUCTIONS

The prompts present the predicted anchor before the current observation and ask for a short command toward the nominal state. They also specify the VLM configuration and an empty-instruction control.

C VLM PROMPTS FOR DIAGNOSIS AND RECOVERY INSTRUCTIONS

  • Each image combines external and wrist views, with the predicted anchor shown before the current observation.The prompt uses this ordering to present the expected and observed states together.Source: C VLM PROMPTS FOR DIAGNOSIS AND… · p. 18
  • The recovery prompt requests one short robot command toward the nominal state, or “continue the task” if the images already match.Source: C VLM PROMPTS FOR DIAGNOSIS AND… · p. 18
  • Qwen3-VL-2B-Instruct runs with deterministic bfloat16 decoding, a 384-pixel longest image edge, and a 64-token output limit.Raw responses and input images are retained, and subsequent ground-truth outcomes are excluded from the prompt.Source: C VLM PROMPTS FOR DIAGNOSIS AND… · p. 18
  • The empty-instruction control retains one valid special token and the usual padding and attention interface, so it does not remove the text branch.It differs from zero video guidance, which still leaves task text in the action stage.Source: C VLM PROMPTS FOR DIAGNOSIS AND… · p. 18

Related papers

Loading 2610.06280v1…