PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA-Corrector: Stage-Aware Observable State Understanding for Prompt-Based Closed-Loop Recovery of Vision-Language-Action Policies

Chang Song, Bin Qian, Yan Feng, Zhijie Song

arXiv:2609.06508v1cs.RO

TL;DR

VLA policies can produce plausible actions yet lack a mechanism to verify whether execution is proceeding correctly, leaving long-horizon manipulation vulnerable to execution deviations. The paper adds an observable-history verifier and stage-aware prompt recovery around a frozen policy; on LIBERO, Prompt Recovery improves success while the Learned Verifier performs without privileged state access. The results support observable visual-proprioceptive-action history as a useful basis for estimating execution state and recovering existing policies.

  • Problem

    VLAs lack mechanisms to verify execution fidelity, and execution deviations can derail otherwise valid long-horizon manipulation.

    Source: Introduction · p. 1

  • Method

    A Learned Verifier estimates progress and execution risk from observable histories, then stage-aware recovery prompts let the same frozen VLA generate corrective actions.

    Source: Introduction · pp. 1–2, Learned Verifier · p. 4

  • Results

    On standard LIBERO, Prompt Recovery raises success from 70.9% to 82.6%, while the Learned Verifier reaches 80.2% without privileged state access and performs within 2.4 pp of the rule-based verifier.

    Source: Introduction · p. 2, Discussion · p. 7

  • Takeaways & Limitations

    Observable visual-proprioceptive-action history provides useful estimates of execution stage and risk for practical recovery with existing VLA policies.

    Source: Conclusion · p. 7

    The formulation uses only temporally ordered history, with no future-step data, and executes only a prefix of each action chunk before replanning.

    Source: Problem Formulation · p. 3

Abstract

from arXiv · show

Long-horizon robot manipulation with Vision-Language-Action (VLA) policies remains vulnerable to execution-time deviations, as final task success provides little information for diagnosing and correcting failures caused by action noise, object displacement, or goal misalignment. We introduce a stage-aware failure verification and Prompt Recovery framework that enables closed-loop correction of a fixed VLA policy without parameter updates or privileged simulator states. The framework introduces an observable-history-based Learned Verifier that jointly estimates manipulation progress and execution risk by temporally modeling multi-view visual observations, proprioceptive states, and executed actions. To provide interpretable task understanding, we represent manipulation execution through semantic progress stages, including approach, alignment, grasp, transport, and placement, and identify stage-specific failure patterns. Upon detecting abnormal execution, the framework preserves the original instruction and generates a stage-conditioned recovery prompt, allowing the same frozen VLA policy to produce corrective actions. Extensive multi-round evaluations on LIBERO and LIBERO Plus demonstrate that the proposed approach substantially improves closed-loop reliability under diverse perturbations. Without access to privileged object or goal coordinates, the Learned Verifier achieves recovery performance close to that of the privileged rule-based verifier in the evaluated settings. These results show that observable visual-proprioceptive-action history is sufficient to infer latent task states and enable practical failure recovery for existing VLA policies.

Introduction

The introduction frames execution-state understanding as a gap between VLA action generation and reliable manipulation: execution deviations can derail otherwise valid long-horizon tasks. It proposes observable-history verification and stage-aware prompt recovery around a frozen policy, evaluated under controlled perturbations.

Introduction

  • VLAs can generate instruction-compliant actions but lack mechanisms to verify execution fidelity, while accumulated action noise, object displacement, grasp imperfections, or goal shifts can derail long-horizon tasks.The paper asks whether an agent can infer its execution state from observable interaction history and improve reliability without modifying its policy or accessing privileged information.Source: Introduction · p. 1
  • The Learned Verifier estimates progress and execution risk from temporal visual, proprioceptive, and action histories without privileged environmental states.The verifier monitors trajectory validity while the VLA generates actions.Source: Introduction · pp. 1–2
  • Prompt Recovery preserves the task instruction and adds a stage-aware recovery prompt so the same frozen VLA can generate corrective actions without parameter updates or trajectory modification.It is presented as a lightweight interface between execution monitoring and action generation.Source: Introduction · pp. 1–2
  • A controlled LIBERO and LIBERO Plus protocol introduces reproducible action-noise, target-object-shift, and goal-region-shift perturbations to assess execution-risk recognition and recovery.The three matched settings are No Assistance, privileged Rule + Prompt, and Learned Binary + Prompt.Source: Introduction · pp. 1–2
  • Prompt Recovery raises standard LIBERO success from 70.9% to 82.6%; the Learned Verifier reaches 80.2% without privileged state access, and real-world target-shift success is 88.3% versus 82.5% for the frozen baseline.The evaluation also compares No Assistance, privileged Rule + Prompt, and Learned Binary + Prompt settings.Source: Introduction · p. 2

Related Work

Related work spans visuomotor and generalist manipulation policies, execution monitoring, recovery strategies, and robot-learning benchmarks. The paper distinguishes its approach by monitoring observable execution histories and using the same frozen policy for prompt-conditioned recovery.

Related Work

  • VLA and earlier visuomotor policies map visual inputs and language or demonstrations to manipulation actions, while generalist systems use multimodal prompting, affordances, or large-scale demonstrations to transfer across tasks.Action chunking can improve temporal coherence and reduce policy queries, but may leave outdated actions executing during replanning latency.Source: Related Work · p. 2
  • Prior execution-monitoring work uses geometric control or temporal visual representations, whereas this monitor combines visual evidence, proprioception, and historical actions to trigger recovery based on closed-loop task success.The paper evaluates the trigger by its impact on task success rather than detector metrics alone.Source: Related Work · p. 2
  • Recovery approaches include low-level control, replanning, demonstrations, and language-conditioned policies; Prompt Recovery retains the original instruction and uses the same frozen π0.5 policy to generate recovery actions.This avoids requiring a separately trained recovery controller.Source: Related Work · p. 2
  • The study uses LIBERO’s sequential task suites to isolate online verification and prompt-recovery contributions under controlled execution perturbations.The related-work discussion contrasts this setting with benchmarks such as Meta-World and RLBench and scalable robot-data collections.Source: Related Work · p. 2
  • Privileged simulator signals support the rule-based verifier and offline teacher labels, while the online Learned Verifier uses only observable visual, proprioceptive, and action sequences.Domain and dynamics randomization alone do not provide a deployable execution monitor.Source: Related Work · pp. 2–3

Problem Formulation

The formulation treats the VLA as a fixed controller that consumes instruction and recent observable history, while a verifier predicts execution risk and can prompt recovery. Evaluation distinguishes final episode outcomes from trigger quality and action cost.

Problem Formulation

  • The observable history contains agent- and wrist-view images, proprioceptive state, and the action actually executed in the environment.The history is temporally ordered, with no future-step observations or actions used.Source: Problem Formulation · p. 3
  • The fixed VLA maps the task instruction and recent history to an action chunk, executes only a prefix, and is queried again with updated history without modifying its parameters.This defines the policy and replanning boundary of the monitoring loop.Source: Problem Formulation · p. 3
  • The Learned Verifier estimates the probability of a teacher-defined failure event within a prediction horizon and thresholds that probability to produce a binary risk estimate.The target label is binary, with one indicating a failure event within the horizon.Source: Problem Formulation · p. 3
  • The online gate requires consecutive sampled windows above threshold outside cooldown before issuing a prompt, with its parameters fixed after validation.Final episode outcome, trigger quality, and action cost are evaluated separately because success rate alone does not establish binary detection accuracy.Source: Problem Formulation · p. 3
  • A recovery prompt is formed by concatenating the original task instruction with a stage-routed or generic recovery fragment.The original instruction remains present, and the same policy generates recovery actions.Source: Problem Formulation · p. 3

Method

The system overview describes a closed loop in which a frozen policy proposes action chunks and a verifier can redirect execution through a recovery prompt. Recovery uses the same policy, after which the original task resumes.

Method

  • At each cycle, the frozen policy outputs an action chunk, but only a fixed-length prefix is executed before replanning.The verifier then ingests the resulting observable history.Source: System Overview · p. 3
  • If the gate suppresses intervention, execution continues under the original task instruction; otherwise, the pending chunk is discarded and the same policy generates recovery actions from a recovery prompt.The architecture is shown in Fig. 1.Source: System Overview · p. 3
  • The original task resumes immediately after the recovery actions finish.This closes the correction cycle without replacing the base policy.Source: System Overview · p. 3

Base π0.5 Policy

VLA-Corrector keeps the pretrained VLA policy frozen and separates action generation from monitoring and recovery. An observable-history verifier estimates execution risk and stage, while privileged simulator state is used only by the rule-based verifier and for offline supervision.

Base π0.5 Policy

  • The pretrained OpenPI pi05_libero controller remains frozen, executing the first five actions of each returned action chunk before replanning.The visual encoder and action-generation head are also unchanged.Source: Base π0.5 Policy · p. 3

Privileged Rule-Based Verifier

  • The privileged rule-based verifier uses simulator ground truth to emit stage-and-context recovery records, serving as an efficacy upper bound rather than a deployable monitor.Recovery is invoked before terminal SUCCESS; the verifier can use object and region poses, end-effector kinematics, gripper status, motion predicates, and task flags.Source: Privileged Rule-Based Verifier · p. 3

Learned Verifier

  • The Learned Verifier processes two visual views, proprioception, and executed actions over time to estimate execution risk and the current stage from observable inputs.Its five learned stages are APPROACH, ALIGN, GRASP, MOVE, and PLACE; privileged simulator states provide offline supervision only.Source: Learned Verifier · pp. 3–4

Online Trigger Gate and Recovery Routing

  • Risk predictions trigger intervention through thresholding, consecutive-positive confirmation, and cooldown, while stage estimates route recovery prompts and uncertain stages default to a generic prompt.The stated risk threshold is 0.40, confirmation requires four frames, the cooldown is 10 steps, and stage confidence below 0.9625 defaults to a generic recovery prompt.Source: Online Trigger Gate and Recovery Routing · p. 5

Prompt Recovery

  • The shared Prompt Recovery interface preserves the original instruction, appends a stage directive, and uses the same frozen policy to generate corrective actions.Low-confidence cases default to UNKNOWN_FAILURE; restoring the original instruction after recovery helps isolate the verifier-trigger pathway.Source: Prompt Recovery · p. 5

Experimental Setup

The evaluation uses a frozen VLA policy across standard LIBERO, two LIBERO Plus difficulty settings, and controlled perturbations. It measures episode success, verifier performance, and real-world recovery across three manipulation tasks.

Benchmark and Evaluation Protocol

  • Three benchmark settings use the same frozen VLA policy: standard LIBERO and LIBERO Plus at difficulties 4 and 5, with assisted variants sharing a Prompt Recovery interface.Standard LIBERO includes Spatial, Goal, Object, LIBERO-10, and LIBERO-90; LIBERO Plus uses the corresponding Spatial, Goal, Object, and 10 suites.Source: Benchmark and Evaluation Protocol · p. 5, Table 1 · p. 5

Perturbation Purpose and Definition

  • Controlled perturbations add Gaussian action noise, shift target-object or goal-region XY coordinates, or combine all three perturbations; methods share initial states, seeds, and protocols.Action noise uses σ = 0.1 and spatial shifts are 0.1 m; normal execution is capped at 280 steps, with up to 100 actions per intervention.Source: Perturbation Purpose and Definition · p. 5

Metrics and Learned Verifier Configuration

  • Episode success is the primary metric, while verifier Precision, Recall, Positive F1, AUROC, and AUPRC assess risk or teacher-warning detection.Gate quality, recovery success, and action cost are reported separately; the verifier estimates execution risk from observable history rather than final task success.Source: Metrics and Learned Verifier… · p. 5

Real-World Evaluation

  • Real-world evaluation tests Pick and Open, Place, and Pour Water under target-object shifts, using 40 trials per task without real-world policy fine-tuning or verifier training.The same robot platform, sensors, observation interface, control frequency, and data-processing pipeline are used across methods.Source: Real-World Evaluation · pp. 5–6, Figure 2 · p. 6

Results

Prompt Recovery improves simulation success across LIBERO benchmarks and perturbations, while the Learned Verifier performs close to the privileged rule-based verifier. The method also shows sub-millisecond monitoring and improved success in three real-world tasks.

Robustness to Environmental Perturbations

  • Both Prompt Recovery methods outperform No Assistance across clean execution, action noise, target-object shifts, goal-region shifts, and combined perturbations.Gains are most pronounced under target-object and goal-region shifts.Source: Robustness to Environmental… · p. 6

Overall Performance in Simulation

  • On standard LIBERO, success rates are 70.9% for No Assistance, 82.6% for Rule + Prompt, and 80.2% for Learned Binary + Prompt.Across LIBERO Plus, the corresponding rates are 73.7%, 79.7%, and 79.0% at difficulty 4, and 66.7%, 73.6%, and 72.6% at difficulty 5.Source: Figure 3 · p. 6, Overall Performance in Simulation · p. 6

Failure Detection and Recovery Analysis

  • The Learned Verifier reaches 93.1% accuracy, 61.1% precision, 94.9% recall, 74.4% Positive F1, AUROC 0.902, and AUPRC 0.7123.Inference latency is 0.457 ms (p50) and 0.589 ms (p95); combined with gating, it achieves 79.2% trigger precision, while diagnostic logs report recovery success rates of 77.0% for Rule + Prompt and 75.0% for Learned Binary + Prompt.Source: Failure Detection and Recovery Analysis · p. 6

Ablation Studies

  • Temporal + Fusion provides the strongest execution-risk recall and Positive F1, while State-action only performs worst overall.Vision only is the strongest masked-input alternative, and all configurations retain sub-millisecond inference latency.Source: Ablation Studies · p. 7

Real-World Evaluation Results

  • Across three real-world manipulation tasks, the method reaches 88.3% average execution success versus 82.5% for the frozen pi05_libero baseline, a 5.8-point improvement.Task-level success rates are reported in Fig. 5.Source: Real-World Evaluation Results · p. 7

Discussion

The Learned Verifier operates without privileged state and performs close to the privileged rule-based verifier. Reported gains across perturbations and real-world tasks support the method’s recovery potential in those evaluated settings.

Discussion

  • The Learned Verifier achieves failure recovery without privileged state and performs within 2.4 pp of the privileged rule-based verifier.The recovery module separates state estimation from the control policy and avoids costly retraining.Source: Discussion · p. 7
  • Across action noise, target-object shifts, and goal-region shifts, the reported gains are +11.7 pp and +9.3 pp over No Assistance.Figure 5 compares the frozen π0.5 baseline and proposed method across three real-world tasks, with annotations for absolute improvements in percentage points.Source: Discussion · p. 7, Figure 5 · p. 7

Conclusion

VLA-Corrector monitors temporally ordered observable histories and triggers stage-aware Prompt Recovery while keeping the base policy frozen. The paper concludes that these histories provide useful estimates of execution stage and risk for recovery with existing VLA policies.

Conclusion

  • Observable visual-proprioceptive-action history provides useful estimates of execution stage and risk, enabling practical recovery for existing VLA policies.The detect-and-correct module monitors temporally ordered histories and keeps the base policy frozen.Source: Conclusion · p. 7

Related papers

Loading 2609.06508v1…