PaperlayerView exact revision on arXiv

Source-linked AI summary

RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors

David Ahmedt-Aristizabal, Abdelwahed Khamis, Mohammad Ali Armin, Lars Petersson

arXiv:2610.13516v1cs.ROcs.AI

TL;DR

Runtime monitors can predict failure yet alarm too late for recovery or interrupt executions that would succeed, so prediction quality alone does not determine whether intervention is worthwhile. RESCUE restores execution states to compare intervention-conditioned recoverability and cost. Its results show that recoverability depends on the intervention, while favourable detection scores do not establish lower intervention cost.

  • Problem

    Failure prediction and intervention choice are distinct: an alarm may arrive too late for recovery or unnecessarily interrupt an execution that would otherwise succeed.

    Source: §I · p. 1

  • Method

    RESCUE restores simulator and policy states at checkpoints to evaluate alternative interventions, their action-conditioned success, and cost-to-completion value.

    Source: §I · pp. 1–2

  • Results

    At τ = 0.5, 14 of 66 OpenVLA-OFT/LIBERO-Plus failures have a recoverable checkpoint under continuation, versus 34 of 66 under continuation or scripted recovery; favourable detection scores do not establish lower intervention cost.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    Monitor value depends jointly on the intervention available and the workload, including failure prevalence and intervention costs.

    Source: §V · p. 8, §VI · p. 8

    The empirical conclusions are bounded by MuJoCo and LIBERO/LIBERO-Plus experiments; branch perturbations and decision costs are declared rather than calibrated from deployment.

    Source: §V · p. 8

Abstract

from arXiv · show

Runtime failure monitors for robot policies are typically evaluated by how accurately they distinguish failed from successful executions. Yet an accurate alarm may arrive too late for recovery, while an unnecessary intervention may disrupt an execution that would otherwise succeed. We introduce RESCUE, a counterfactual framework for evaluating whether a monitor's alarms enable useful interventions. By restoring simulator and policy states at execution checkpoints, RESCUE compares alternative actions, identifies the latest checkpoint satisfying a specified recovery criterion, and evaluates whether acting on alarms reduces expected task-completion cost. We evaluate RESCUE with OpenVLA-OFT and $π_{0.5}$ over LIBERO-Spatial and LIBERO-Plus using 128,340 counterfactual branches. Recoverability depends on the state--intervention pair rather than on robot state alone: at $τ=0.5$, 14 of 66 OpenVLA-OFT/LIBERO-Plus failures contain a recoverable checkpoint under continuation, compared with 34 of 66 under continuation or scripted recovery, while recovery reduces recoverability in other conditions. On the primary strong-policy conditions, favourable detection scores do not establish lower intervention cost. Under the evaluated stop-and-reset model, even an idealised detector can afford only a 2.0--3.3% false-alarm rate before unnecessary resets outweigh the savings from stopping failures early. These results show that monitor value depends jointly on the intervention available and the workload in which it is deployed.

I. INTRODUCTION

RESCUE evaluates whether runtime-monitor alarms support useful interventions, addressing the gap between predicting failure and deciding whether to act. It combines counterfactual recoverability with intervention costs and finds that favourable detection metrics do not establish lower intervention cost.

I. INTRODUCTION

  • Failure prediction and intervention choice are distinct: alarms may arrive after recovery is lost, or false alarms may interrupt executions that would succeed.Interventions can also incur reset, delay, or damage costs.Source: §I · p. 1
  • Existing timing-aware measures address only part of the problem; actionability depends on available repairs, and intervention value can vary across a trajectory.The same robot-policy state can have different value under continuation, recovery, or stop-and-reset.Source: §I · p. 1
  • RESCUE restores simulator and policy states at action-chunk boundaries to evaluate alternative interventions and estimate action-conditioned task success.Across two VLA policies and two LIBERO benchmarks, the evaluation comprises 1,562 trunk episodes and 128,340 counterfactual branches.Source: §I · pp. 1–2
  • RESCUE evaluates interventions in a common cost-to-completion space and derives a break-even false-alarm rate for stop-and-reset.The criterion uses failure prevalence, alarm timing, false alarms, and intervention cost.Source: §I · p. 2
  • 14 of 66 OpenVLA-OFT/LIBERO-Plus failures have a recoverable checkpoint under continuation, versus 34 of 66 under continuation or scripted recovery at τ = 0.5.Under the evaluated stop-and-reset model, even an idealised detector can afford only a 2.78–3.32% false-alarm rate.Source: §I · p. 2

II. RELATED WORK

Prior work studies failure detection, recovery behaviour, and intervention timing, while RESCUE evaluates alternative interventions across checkpoints and their decision value. Its recoverability horizon is conditional on the intervention repertoire, execution model, and task budget.

II. RELATED WORK

  • Runtime-monitor studies primarily assess how reliably and early failures are identified; RESCUE instead compares counterfactual interventions and expected task-completion cost.CheckVLA tests fixed repairs, whereas RESCUE compares alternative interventions throughout execution.Source: §II · p. 2
  • Recovery and timing work designs responses or assesses actionable windows; RESCUE evaluates branchable interventions at saved action-chunk boundaries.Its horizon is the latest checkpoint meeting a recovery criterion, without assuming recoverability decreases monotonically over time.Source: §II · p. 2
  • On OpenVLA-OFT/LIBERO-Plus, 14/66 failures have a qualifying checkpoint under CONTINUE, 33/66 under RECOVER, and 34/66 under their union.The figure shows recoverability estimates and horizons alongside monitor alarm times, illustrating that recoverability depends on intervention.Source: Figure 2 · p. 3

III. METHOD

  • RESCUE restores complete robot-policy states at checkpoints and evaluates alternative interventions for task success probability and expected cost to completion.These quantities depend on the declared intervention repertoire, execution model, and remaining task budget.Source: §III · p. 3

A. State–Intervention Value and Recoverability

RESCUE characterizes each checkpoint through intervention-conditioned success probability and expected cost to eventual task completion. It defines recoverability by a success-probability threshold and the horizon as the latest checkpoint meeting that criterion.

A. State–Intervention Value and Recoverability

  • The expected cost to eventual task completion includes the current attempt and, if needed, subsequent reset-and-retry attempts.Success probability describes completion within the current attempt, while expected cost provides the decision-oriented view.Source: A. State–Intervention Value and… · p. 3
  • A checkpoint is recoverable when its best available success probability meets criterion τ; the horizon is the latest qualifying checkpoint and is undefined if none qualifies.The best success probability is taken over the available intervention repertoire.Source: A. State–Intervention Value and… · p. 3
  • Recoverability is a property of the state–intervention pair, not the robot state alone, and changes with the intervention repertoire, execution model, or remaining budget.The horizon does not assume recoverability varies monotonically with time.Source: A. State–Intervention Value and… · p. 3

B. Counterfactual Branching and Estimation

RESCUE estimates intervention outcomes by restoring saved checkpoints and generating seeded counterfactual branches. It uses a lower Jeffreys posterior quantile for success probability and increases branch counts near the recovery-criterion transition.

B. Counterfactual Branching and Estimation

  • For each checkpoint and branchable intervention, RESCUE restores the state and generates seeded counterfactual branches until task success or budget exhaustion.The same branches yield success outcomes and realised execution lengths.Source: B. Counterfactual Branching and… · pp. 3–4
  • Success probability is estimated with a lower Jeffreys posterior quantile rather than the raw fraction of successful branches.The lower quantile is q = 0.25; checkpoints start with K = 4 branches and transition-region checkpoints expand to K = 16.Source: B. Counterfactual Branching and… · p. 4
  • A replay-fidelity check requires deterministic restored executions to reproduce the corresponding trunk continuation before branch outcomes are interpreted.The figure illustrates seeded branches, raw success fractions, posterior estimates, and the resulting horizon.Source: B. Counterfactual Branching and… · p. 4

C. Stop-and-Reset Decision Value

RESCUE compares interventions in a shared cost-to-completion space, where resetting ends the current attempt and starts a fresh one. For stop-and-reset, a break-even false-alarm rate marks when stopping failures early outweighs the costs of unnecessary resets.

C. Stop-and-Reset Decision Value

  • A reset terminates the current attempt, incurs unit cost, and starts a fresh attempt drawn from the same workload.The framework evaluates expected cost per completed task under this renewal model.Source: C. Stop-and-Reset Decision Value · p. 4
  • Branchable interventions are compared by their additional execution steps after a checkpoint, while immediate stop-and-reset is evaluated on the same cost scale.The already-incurred cost is common to actions at the same checkpoint and cancels in comparisons.Source: C. Stop-and-Reset Decision Value · p. 4
  • The break-even false-alarm rate uses failure prevalence, detection coverage and timing, false-alarm frequency and timing, and successful-episode duration.These quantities and the expected cost to completion when never intervening can be obtained from ordinary execution logs.Source: C. Stop-and-Reset Decision Value · p. 4
  • A realised false-alarm rate below the break-even rate makes stopping preferable to never intervening under the declared cost model; a higher rate makes it decision-negative.The numerator captures execution cost saved by stopping detected failures early, while the denominator captures excess false-alarm cost, including the reset and replacement attempt.Source: C. Stop-and-Reset Decision Value · p. 4

A. Experimental Setup

RESCUE evaluates counterfactual interventions across robot-policy trajectories and finds that recoverability changes with the available intervention. The direction and size of that change vary across policy settings and conditions.

A. Experimental Setup

  • The corpus contains 1,562 trunk episodes and 128,340 counterfactual branches across OpenVLA-OFT and π0.5 on LIBERO-Spatial and LIBERO-Plus.The primary analysis uses two strong-policy LIBERO-Plus cells; other cells provide controlled contrasts rather than cross-policy comparisons.Source: A. Experimental Setup · p. 4
  • Branches restore simulator and policy snapshots at action-chunk boundaries, evaluate continuation and a fixed scripted recovery, and inherit the original episode’s remaining budget.The primary recoverability criterion is τ = 0.5; monitor thresholds target a nominal 5% false-alarm rate on validation data and are evaluated on held-out data.Source: A. Experimental Setup · pp. 4–5

B. Recoverability Is Intervention-Relative

  • At τ = 0.5, 14 of 66 OpenVLA-OFT/LIBERO-Plus failures have a qualifying checkpoint under continuation, versus 34 of 66 under continuation or scripted recovery.Under scripted recovery alone, 33 of 66 failures qualify; 32 remain unrecoverable under either intervention.Source: B. Recoverability Is… · p. 5
  • Across four policy settings, Fig. 3 compares alarm lateness and alarm-to-horizon lag for continuation and scripted recovery.Positive lag indicates late detection; the plotted results include cases with both an alarm and a defined recoverability horizon.Source: Figure 3 · p. 5
  • Recovery effects vary by condition: it leaves 25 of 55 π0.5/LIBERO-Plus failures qualifying, but reduces qualifying failures on both weakened OpenVLA-OFT cells.For the weakened cells, the clearing sets fall from 50/51 to 43/51 at σexec = 0.2 and from 51/87 to 41/87 at σexec = 0.3; these are within-cell contrasts.Source: B. Recoverability Is… · p. 5
  • With zero branch perturbation, continuation succeeds on none of 61 valid OpenVLA-OFT replays, while scripted recovery retains qualifying checkpoints on 31.This result shows the intervention contrast persists when the injected branch perturbation is removed.Source: B. Recoverability Is… · p. 5

C. Decision Value of Stop-and-Reset

Under the evaluated stop-and-reset model, the steps-since-progress monitor’s realised false-alarm rates exceed break-even rates on both primary LIBERO-Plus cells. Even an idealised detector has only a small affordable false-alarm margin.

C. Decision Value of Stop-and-Reset

  • 7.25% realised false alarms exceed the 1.01% break-even rate for steps-since-progress on OpenVLA-OFT/LIBERO-Plus, yielding +0.0835 reset-equivalents per completed task versus never intervening.The excess-cost 95% CI is [0.0294, 0.1392].Source: Table I · p. 6, C. Decision Value of Stop-and-Reset · p. 6
  • 7.04% realised false alarms exceed the 0.45% break-even rate for steps-since-progress on π0.5/LIBERO-Plus.The nominal false-alarm target used to select a detector need not be economically compatible with the intervention it triggers.Source: Table I · p. 6, C. Decision Value of Stop-and-Reset · p. 6
  • 2.78–3.32% and 2.02–2.39% are the affordable false-alarm rates for an idealised detector on OpenVLA-OFT/LIBERO-Plus and π0.5/LIBERO-Plus, respectively.The ranges reflect false-alarm timing and the natural duration of the successful episodes on which false alarms occur.Source: C. Decision Value of Stop-and-Reset · p. 6
  • 9.68% false alarms fall within the 9.59–10.30% idealised ceiling on the weakened σexec = 0.3 cell, yet the monitor incurs +0.2613 excess cost per completed task.The monitor does not detect every failure or alarm at step zero, so the idealised ceiling is necessary rather than sufficient.Source: C. Decision Value of Stop-and-Reset · p. 6

D. Runtime Monitors as Intervention Rules

Detection and timing metrics often favoured monitors that did not lower intervention cost against never intervening on the primary strong-policy conditions. Changing the intervention or workload altered outcomes, but did not establish a general benefit.

  • At nominal 5%, action-jerk excess costs are +0.0563 reset-equivalents per completed task on OpenVLA-OFT/LIBERO-Plus and +0.0698 on π0.5/LIBERO-Plus.Source: D. Runtime Monitors as Intervention… · p. 6
  • Detection and timing criteria admit 61/75 monitor verdicts, while decision value admits 0/15 against never intervening and 5/15 against rate-matched random alarms.The random-alarm comparison is not sufficient: the deliberately inverted steps-since-progress decoy also clears that reference on all four universes.Source: Table II · p. 6, D. Runtime Monitors as Intervention… · p. 7
  • The SAFE-style monitor beats never intervening on three of six higher-prevalence held-out evaluations, but no real monitor establishes lower cost on the primary strong-policy conditions.Its largest gain is ∆C = −0.0431 (95% CI [−0.0432, −0.0426]) on a cell with 34% failure prevalence.Source: D. Runtime Monitors as Intervention… · p. 6
  • On the two primary strong-policy conditions, no real monitor has a confidence interval establishing lower cost than never intervening under either stop-and-reset or RECOVER.Routing the same alarms to RECOVER instead of stop-and-reset reduces excess cost by factors of 3.1–13.4 across evaluated configurations.Source: D. Runtime Monitors as Intervention… · p. 6
  • Cost-minimising validation thresholds do not establish improvement: five of eight monitor–intervention configurations produce no held-out alarms, and the sole negative point estimate has an interval spanning zero.For steps-since-progress routed to RECOVER, ∆C = −0.0058 (95% interval [−0.0204, 0.0143]).Source: D. Runtime Monitors as Intervention… · p. 6
  • On OpenVLA-OFT/LIBERO-Plus, a fixed-step rule stopping at step 168 costs +0.0038 reset-equivalents per completed task, versus +0.0835 for steps-since-progress.The fixed-step rule's 95% CI is [−0.0085, 0.0348].Source: D. Runtime Monitors as Intervention… · p. 7
  • Monitor families fail differently: steps-since-progress alarms late, SAFE's independent head realises a 26.5% false-alarm rate at a 5% target, and π0.5 disagreement has positive cost despite a 2.3% rate.Source: D. Runtime Monitors as Intervention… · p. 7

E. Robustness and Evaluation Controls

Robustness checks support the replay and estimator procedures while showing that recoverability and monitor verdicts depend on evaluation choices. Timing comparisons require matched nulls, and episode-level detection can be confounded by timeout length.

  • All 3,000 seeded checkpoint replays reproduced recorded joint trajectories with maximum joint-position deviation 0.0; higher-sampling checks left episode-level recoverability labels unchanged for all 20 failures under both interventions.The fixed-K estimator disagreed with the reference on 1.61% of CONTINUE and 1.07% of RECOVER checkpoints; horizons shifted on 3/20 and 1/20 episodes, respectively.Source: E. Robustness and Evaluation Controls · p. 7
  • Recoverability need not be monotone: RECOVER was significantly non-monotone on 8/20 fixed-K episodes, versus 0/20 for CONTINUE, supporting a latest-qualifying-checkpoint horizon.Source: E. Robustness and Evaluation Controls · p. 7
  • Base-task resampling widens the steps-since-progress 95% interval on OpenVLA-OFT/LIBERO-Plus to [0.0234, 0.2793] from [0.0294, 0.1392], but the verdict remains above zero.The monitor also remains decision-negative across median-successful-episode time prices of 0.05, 0.1, 0.2, and 0.5 resets.Source: E. Robustness and Evaluation Controls · p. 7
  • The recoverability threshold and clock convention affect estimates: at τ = 0.3, 0.5, and 0.7, 18, 14, and 13 of 66 failures qualify under CONTINUE.A fresh-budget clock shifts the paired π0.5/LIBERO-Plus horizon by +30 steps on 27 failures and changes 29/55 episode labels; this ablation preceded the sensor-noise correction.Source: E. Robustness and Evaluation Controls · p. 7
  • Random alarms occur after the horizon on approximately 73% of recoverable failures in each OpenVLA-OFT cell, so timing needs a matched null; all real monitors alarm after the horizon in over half of defined CONTINUE episodes.Source: E. Robustness and Evaluation Controls · p. 7
  • All 262 corpus failures are 220-step timeouts, and action-chunk count alone attains AUROC 1.000 on every validation split and task-disjoint holdout.A per-task early-exposure control reduces steps-since-progress AUROC from 0.979 to 0.637, so detection metrics are treated as diagnostics rather than intervention-value substitutes.Source: E. Robustness and Evaluation Controls · pp. 7–8

V. DISCUSSION AND LIMITATIONS

RESCUE evaluates monitors together with their triggered interventions: recoverability and intervention value depend on the declared repertoire, execution model, budget, and workload. The empirical conclusions are bounded by the simulated benchmark and evaluated monitor and policy scope.

V. DISCUSSION AND LIMITATIONS

  • Recoverability is conditional on intervention repertoire, execution model, and remaining budget, not an intrinsic property of robot state or policy.The evaluation covers CONTINUE and one fixed scripted RECOVER, not richer or learned recovery repertoires.Source: §V · p. 8
  • Experiments use MuJoCo and LIBERO/LIBERO-Plus, and all 262 observed failures are timeouts; hardware dynamics, damage, latency, and recovery costs are absent.The study also evaluates two VLA policies and four monitor families without training a new detector.Source: §V · p. 8

VI. CONCLUSION

  • Across 1,562 VLA episodes and 128,340 counterfactual branches, recoverability varies with intervention, and favourable detection or timing metrics can coexist with decision-negative intervention costs.Source: §VI · p. 8
Loading 2610.13516v1…