Source-linked AI summary
Do VLAs Know They Will Fail? Interpretable Success and Failure Patterns in Physical AI Systems
Christian A. Rogers, Qian Lin, Daniel S. Brown
TL;DR
VLA rollouts are difficult to diagnose, and warnings measured against episode end may arrive after recovery is no longer possible. The paper analyzes OpenVLA SAE features to predict outcomes and identify associated behaviors, finding that failures can be distinguished while still recoverable and that localized behavior descriptions correspond to feature activations on held-out clips.
Problem
Failure warnings measured against episode end may arrive after a robot has reached a state from which recovery is no longer possible, while it remains unclear which sparse features track rollout outcomes and what behaviors they correspond to.
Source: §1 · pp. 1–2Method
The authors use OpenVLA SAE features to predict trajectory outcomes, identify outcome-associated features and critical intervals, then interpret those intervals with a VLM and validate descriptions on held-out clips.
Source: §2.1 · p. 2, §2.2 · p. 2, §2.3 · p. 3Results
SAE features preserve outcome-predictive information in OpenVLA activations, and the predictor distinguishes eventual failures from successes before the estimated point of no return.
Source: §1 · p. 2, §2.2 · p. 2Takeaways & Limitations
Localized VLM descriptions validated against held-out clips connect early failure warning with interpretable behaviors associated with outcomes, providing a basis for investigating potential failure mechanisms and targeted safety interventions.
Source: §3 · p. 5Recoverability estimates use four sampled branches per probe and are coarse; temperature sampling can alter estimated recoverability across rollout phases.
Source: §E.4 · p. 12
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have emerged as powerful general-purpose robot policies, but their successes and failures remain difficult to diagnose and understand. In this work, we show that sparse autoencoder (SAE) representations of OpenVLA activations contain outcome-predictive information and can be used to predict task failure before the robot reaches an unrecoverable state. Moreover, to provide insights into what common behaviors may contribute to these outcomes, we identify individual SAE features whose strong activations are associated with specific stages of success or failure, and uncover recurring behavioral patterns by localizing temporal intervals where these features exhibit significant activation changes. We further develop a VLM-based interpretation pipeline to generate behavioral descriptions of these patterns and validate their correspondence to feature activations on held-out intervals.
1 Introduction
VLA rollouts can be hard to diagnose, and warnings measured against episode end may arrive after recovery is no longer possible. This work links early failure prediction to behavioral patterns associated with rollout outcomes.
1 Introduction
- Episode-end warning time may be too late to act, so the authors evaluate warnings against a point of no return after which branched policies reach only failures.This frames warning usefulness in terms of whether recovery remains possible, not just how long before an episode ends an alarm occurs.Source: §1 · p. 1
- The paper asks which sparse features track rollout success and what behaviors those features correspond to.This complements prior work characterizing VLA features by representation or control.Source: §1 · pp. 1–2
- SAE features preserve outcome-predictive information in OpenVLA activations, enabling the predictor to distinguish eventual failures from successes while failures remain recoverable.The authors also select outcome-associated features, localize critical intervals, and validate VLM-generated descriptions on held-out clips.Source: §1 · p. 2
2 Experimental Results
Across 500 OpenVLA-7B rollouts, the study tests outcome prediction from raw activations and SAE features, then examines outcome-associated features and behavioral intervals. The results connect early warning with feature-based behavioral patterns and held-out validation of VLM-generated descriptions.
2.1 Problem Setup
- The evaluation uses 500 greedy OpenVLA-7B rollouts on LIBERO-10, with 51.6% success and 95 trajectories held out.The model is fine-tuned on LIBERO-10 and evaluated from 50 benchmark initial states for each of ten tasks.Source: §2.1 · p. 2
2.2 Online Outcome Prediction
- Before the estimated PNR, SAE features distinguish failures from successes with AUROC 0.71 (95% CI [0.62, 0.79]), versus 0.65 (95% CI [0.57, 0.74]) for raw activations.PNR is defined from simulator probes of state recoverability; an alarm is actionable when it occurs before PNR.Source: §2.2 · pp. 2–3, Figure 1 · p. 3
- Across held-out frames, raw activations and SAE features have AUROCs of 0.80 and 0.81, and both outperform the majority baseline at early-to-mid timesteps.Both remain above chance from the first 80 steps (0.68) through steps 240–319 (0.75).Source: §2.2 · p. 2, Figure 1 · p. 3
2.3 Discovering Outcome-Associated SAE Features
- Among each feature’s top 30 trajectories, feature 1578 has 28 successes, while feature 480 has 29 failures across four tasks.These associations motivate selecting high-activation intervals to inspect feature-linked behavior.Source: Table 2 · p. 3, §2.3 · p. 3
2.2 Online Outcome Prediction
- As the alarm threshold rises from 0.5 to 0.9, false alarms fall from 64% to 14%, actionable detection from 88% to 33%, and median lead time from 190 to 70 steps.The table reports alarms relative to the point of no return on held-out trajectories.Source: Table 1 · p. 3, §2.2 · p. 3
2.4 Localizing Outcome-Associated Behavioral Patterns
- In three strong feature-480 failures, detected intervals show contact with the middle moka pot without transporting it onto the stove.The intervals target sustained activation rises, rather than merely selecting regions of high activation.Source: Figure 2 · p. 4, §2.4 · p. 4
2.5 Automatic Interpretation and Validation
- Held-out description detection scores a mean AUROC of 0.76 across 12 feature–task pairs, versus 0.58 for shuffled descriptions.The VLM describes critical intervals, and the descriptions are evaluated on held-out clips.Source: §2.5 · p. 4
3 Conclusion
The paper connects early failure prediction in OpenVLA with interpretable analysis of outcome-associated behaviors. It identifies outcome-associated SAE features, localizes critical intervals, and interprets those events with a VLM, complementing prior work on VLA failure prediction and SAE interpretability.
3 Conclusion
- SAE features preserve outcome-predictive information in OpenVLA activations and support failure prediction before the robot reaches an estimated unrecoverable state.The framework also identifies outcome-associated features, localizes critical activation intervals, and validates VLM-generated behavioral descriptions on held-out clips.Source: §3 · pp. 4–5
- The paper links early failure warning to interpretable behavior analysis, with a foundation for investigating potential failure mechanisms, safety interventions, and reward signals for future policy training.Source: §3 · p. 5
A Related Work
- Prior work uses internal VLA representations or external observations to anticipate failures; this work also measures warning time against a simulator-estimated point of no return.Source: A Related Work · p. 7
- Earlier VLA SAE studies identify motion and semantic features, including features that can steer behavior, while this work connects outcome-associated features to localized behavioral events.Source: A Related Work · p. 7
B.1 Dataset and Rollout Collection
The study evaluates OpenVLA-7B on 500 LIBERO-10 rollouts and uses a top-K SAE to represent its activations with sparse features. The resulting setup supports analysis of outcome-associated information and feature-level behavioral patterns.
B.1 Dataset and Rollout Collection
- Each rollout ends at task completion or a 520-step timeout, which is counted as failure.Source: §B.1 · pp. 7–8
- 500 OpenVLA-7B rollouts cover ten LIBERO-10 tasks from 50 fixed benchmark initial states per task, with a 51.6% overall success rate.Success rates range from 12% to 80% across tasks; 95 trajectories are held out for evaluation.Source: §B.1 · pp. 7–8
B.2 Sparse Autoencoder
- A top-K sparse autoencoder decomposes hi,t into sparse feature activations, using 2,048 features and K = 64, and is trained on mean-centered activations from all 500 rollouts.Source: §B.2 · p. 8
C.1 Full Automatic Interpretation Results
The automatic explanations associate selected SAE features with task behaviors, while also showing that a feature’s meaning can vary across tasks. The reported table covers highly outcome-associated features, and a figure tracks one success-related feature over time.
C.1 Full Automatic Interpretation Results
- Feature 480 is associated with a stalled moka-pot approach in Tasks 1 and 9, while feature 1671 is associated with transporting a grasped can into the basket.Source: §C.1 · p. 8
- Figure 3 tracks the temporal activation of success-related feature 1671 on Task 3.Source: Figure 3 · p. 9
- Table 4 reports success rates among top-k trajectories for features with the lowest and highest rates, corresponding to failure- and success-associated features.Source: Table 4 · p. 9
- The passage indicates that a feature’s meaning may depend on the task; such features are better interpreted as markers of successful or failed outcomes, rather than as causes.Source: §C.1 · p. 9
C.2 Explanation Validations
Descriptions correspond to feature activations on held-out clips in aggregate, but their specificity varies across features. The validation supports the methodology overall, while individual descriptions remain hypotheses.
C.2 Explanation Validations
- 0.76 mean AUROC and 0.46 mean Spearman correlation were obtained across 12 feature-task pairs and 261 clips.The mean AUROC has a 95% CI of [0.64, 0.87] and permutation p < 0.001.Source: §C.2 · p. 9
- Descriptions outperformed shuffled descriptions on 11 of 12 pairs, with mean ∆AUROC = 0.17 (95% CI over clips [0.10, 0.24]; sign test p = 0.003).Resampling pairs as well widens the interval to [−0.01, 0.35], indicating that specificity varies across features.Source: §C.2 · p. 9
- Individual descriptions were uneven: feature 850 was anti-correlated with activation (AUROC 0.42), while feature 132’s transport description scored 0.97 versus 0.92 shuffled.With 24 clips per pair, individual AUROCs have confidence intervals of roughly ±0.2, so the authors treat descriptions as hypotheses.Source: §C.2 · p. 10
D Selection of Critical Activation Intervals
The interval detector selects one critical interval per trajectory by finding a sustained activation rise and retaining context around it. Recoverability and the point of no return are estimated from sampled continuations and probes.
D Selection of Critical Activation Intervals
- A local contrast compares smoothed, normalized activation across a temporal horizon, so high values indicate a consistent rise rather than a single-frame spike.The horizon is 15 environment steps; normalization uses the 90th percentile of non-zero activations.Source: D Selection of Critical Activation… · p. 10
- The detector extends prominent local maxima while contrast exceeds a support threshold, adds 15 context steps on each side, merges events less than 25 steps apart, and keeps the most prominent group.It selects exactly one interval per trajectory using fixed parameters, without outcome labels.Source: D Selection of Critical Activation… · pp. 10–11
E.1 Recoverability Estimator
- Recoverability is estimated by restoring the recorded state and running four temperature-1 sampled continuations for the remaining steps.Sampling avoids simply reproducing the greedy rollout outcome; the appendix describes this estimate and the point of no return used in early-warning evaluation.Source: E Recoverability Estimation · p. 11, §E.1 · p. 12
E.2 Probe Placement and Point of No Return
- The point of no return is the earliest probe after which every subsequent probe has estimated recoverability at most 0.1.Probes span ±40 steps around alarms and abrupt rises in failure log-odds across 51 held-out failures.Source: §E.2 · p. 12
- Before-PNR evaluation uses only failures with an uncensored PNR before step 420, leaving 25 failures, and compares them with successful trajectories at matched times.Later PNRs are excluded because late probes can fail simply when the remaining step budget runs out.Source: §E.2 · p. 12
E.3 Short-Term Spikes in Predicted Failure Probability
Most short-term spikes in predicted failure probability occur after estimated recoverability is already low. Earlier spikes show no larger recoverability drop than matched non-spike intervals, and the spike-detection method may miss longer trends.
E.3 Short-Term Spikes in Predicted Failure Probability
- 83% of short-term failure-probability spikes occur after estimated recoverability has already fallen to at most 0.1.This makes most observed spikes late relative to the estimated point of no return.Source: §E.3 · p. 12
- Earlier spikes show no larger recoverability drop than matched non-spike intervals (difference −0.02, 95% CI [−0.12, 0.08]).With four branches per probe, smaller effects cannot be excluded.Source: §E.3 · p. 12
- Spike detection was based on qualitative human analysis and may exclude meso-trends over tens of timesteps that predict human-derived PNR values.The authors leave extending the pipeline to address this confound to future analysis.Source: §E.3 · p. 12
E.4 Limitations of the Recoverability Estimate
The recoverability estimate is an imperfect, policy-specific instrument: remaining step budget, temperature sampling, and only four branches limit the precision and interpretation of estimated failure times.
E.4 Limitations of the Recoverability Estimate
- Late branches can fail from timeout rather than state: success is 0.2% with fewer than 100 steps left versus 18–23% with 200, motivating exclusion of PNRs after step 420.The reported rates apply regardless of actual simulator state.Source: §E.4 · p. 12
- Temperature sampling can free a policy stuck against an object or cause object drops during fast motion and transport, shifting estimated recoverability relative to greedy behavior.The direction of the perturbation depends on the rollout phase.Source: §E.4 · p. 12
- With four branches, an estimate of recoverability 0.25 has a rough 95% interval of [0.05, 0.70], and recoverability describes this policy rather than physical recoverability.The authors therefore interpret results as supporting qualitative conclusions, not precise failure times.Source: §E.4 · p. 12