PaperlayerView exact revision on arXiv

Source-linked AI summary

Progress Probes $P^2$: Lightweight Runtime Monitoring for Vision-Language-Action Models

Atiksh Bhardwaj, Edward W. Duan, Junwon Seo, Prithwish Dan, Wei-Chiu Ma, Andrea Bajscy, Preston Culbertson

arXiv:2610.31613v1cs.RO

TL;DR

VLA policies remain prone to mistakes and unpredictable failures, while existing external-foundation-model monitors add complexity and computational cost at runtime. The method trains linear probes on VLA activations from successful demonstrations to estimate task progress, then monitors deviations from expected progress. Across 6 VLABench tasks and 6 real-world manipulation tasks, it achieved AUROC scores of 0.788 and 0.78, matching or exceeding SAFE and Robometer.

  • Problem

    VLA policies remain prone to mistakes and unpredictable failures, while existing external-foundation-model monitors add complexity and computational cost at runtime.

    Source: §I · p. 1

  • Method

    The method trains linear probes on VLA activations from successful demonstrations to estimate task progress, then monitors deviations from expected progress.

    Source: §I · p. 1

  • Results

    Across 6 VLABench tasks and 6 real-world manipulation tasks, it achieved AUROC scores of 0.788 and 0.78, matching or exceeding SAFE and Robometer.

    Source: §I · p. 2

  • Takeaways & Limitations

    VLA activations retain information useful for detecting task progress and failure, allowing a simple linear readout to support runtime monitoring.

    Source: §I · p. 1, §VIII · p. 8

    Experiments focus on π0.5 and primarily probe VLM-backbone activations, leaving transfer across VLA families and complementary signals from continuous action experts for future work.

    Source: §VIII · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) models can perform diverse manipulation tasks directly from visual observations and natural language instructions, but they remain prone to mistakes and unpredictable failures. Deploying these policies without continuous human supervision will require robust runtime monitors that detect when the policy has gone off-course. Existing monitors use external foundation models, such as vision-language models (VLMs) to supervise the policy, introducing significant complexity and computational cost for monitoring at runtime. In this work, we investigate an alternate approach: \textit{can a VLA's own internal activations provide the information needed to monitor its execution?} Our method trains linear probes on policy activations from demonstrations to estimate progress, then flags executions that deviate from expected progress. In experiments with the open-source VLA $\pi_{0.5}$ and across simulated and real-world manipulation tasks, this approach is competitive with or outperforms VLM-based monitors in task failure detection, as well as methods which require failure data for training. These findings suggest that simple probes of VLA internals can be used to develop reliable runtime monitors, opening a path toward building policies that can act reliably and monitor their execution without external foundation models.

I. INTRODUCTION

The introduction motivates runtime monitoring for VLA policies that can fail unpredictably, and proposes decoding task progress from policy activations using successful demonstrations. The resulting monitor detects deviations from expected progress and is reported to match or exceed competitive baselines across simulated and real-world manipulation tasks.

I. INTRODUCTION

  • VLA policies can perform diverse, multi-stage manipulation tasks but still experience sudden, difficult-to-predict failures.The introduction cites poor language grounding, deadlocks, and mistakes as examples.Source: §I · p. 1
  • Existing monitoring approaches include external VLM supervision and policy-internal failure prediction, while this work investigates progress decoding from policy internals.Examples include Sentinel, SAFE, Robometer, and FIPER.Source: §I · p. 1
  • Successful executions advance predictably through a task, while suboptimal or failed executions cause decoded progress to lag behind elapsed time.Progress can be learned from successful executions and labeled without human supervision.Source: §I · p. 1
  • Progress probes linearly decode task progress from VLA activations, and the monitor flags deviations from the progression expected during successful execution.The probe is trained using successful task executions, and monitoring reuses activations already computed during inference.Source: §I · pp. 1–2
  • AUROC scores of 0.788 on 6 VLABench tasks and 0.78 on 6 real-world manipulation tasks match or exceed competitive baselines including SAFE and Robometer.The monitor uses no failure labels for training.Source: §I · pp. 1–2

II. RELATED WORK

Prior work studies internal representations of robot policies and monitors failures using several signals, including external evaluators and learned predictors. This paper probes VLA activations for task progress, using labeled activations and a linear readout.

II. RELATED WORK

  • The work extends VLA interpretability research by testing whether probes trained only on successful demonstrations extract task progress and support failure detection without an external foundation model.Earlier work examined semantic concepts, motion primitives, and physical and relational states [8].Source: §II · p. 2
  • Runtime monitors use signals such as deviations from successful behavior, external VLM evaluators, learned failure predictors, and reward models.The paper frames whether monitoring needs an external validator, failure data, or useful policy representations as open design questions.Source: §II · p. 2

III. BACKGROUND: MECHANISTIC INTERPRETABILITY VIA LINEAR PROBING

  • For the studied transformer-based VLA class, observations combine an image, proprioceptive data, and a language prompt, and intermediate activations are extracted at each layer.The studied policy, π0.5, uses the PaliGemma vision-language backbone [21].Source: §III · pp. 2–3
  • The probe dataset pairs activations extracted from trajectories with a labeled target feature at each trajectory timestep.A linear probe maps activations from a chosen layer to a scalar target feature and is trained on the training split.Source: §III · p. 3

IV. PROBLEM FORMULATION: RUNTIME MONITORING

The runtime-monitoring problem is to detect during execution when a policy is unlikely to complete its task. A monitor scores deviations from successful behavior, balancing reliable discrimination against early detection.

IV. PROBLEM FORMULATION: RUNTIME MONITORING

  • The rollout outcome is successful if the policy reaches a successful state within H steps, and unsuccessful otherwise.The observation-action trajectory up to time t is denoted τ_t.Source: §IV · p. 3
  • A runtime monitor maps an observation-action trajectory to a scalar deviation score, with larger values indicating greater difference from successful, nominal data.A threshold determines when the monitor fires; it may never fire within the task horizon.Source: §IV · p. 3
  • The monitoring objective balances false positives on successful trajectories against mean detection time on failed trajectories.The formulation evaluates detections within the maximum task horizon and favors early detection under the closed-loop evaluation trajectory distribution.Source: §IV · p. 3

V. PROGRESS PROBES AS RUNTIME MONITORS

Progress Probes linearly decode task progress from VLA activations, using successful trajectories to train a lightweight runtime monitor. At deployment, the monitor compares predicted progress with the nominal trajectory and flags sufficiently delayed execution.

Progress Probes as Runtime Monitors

  • Progress probes use linear readouts of VLA activations because successful executions encode predictable, monotonically advancing task progress.Suboptimal or failed behaviors can violate that relationship, causing the probe to fall behind elapsed time.Source: §V · p. 3

Progress Probes

  • Successful trajectories supply normalized time-remaining labels, which decrease from 1 at the start to 0 at the end.The probe dataset pairs activations from demonstrations with these progress labels; linear regression trains the probe to predict progress from a layer’s activations.Source: §V · p. 3

Runtime Monitoring via Progress Probes

  • The monitor compares predicted progress with the expected progress at the current timestep, based on mean successful trajectory length.A positive score means estimated progress lags the nominal trajectory; failures are predicted when the score exceeds threshold δ, while faster execution is not penalized.Source: §V · pp. 3–4

Progress Probes as Runtime Monitors

  • The monitor requires only successful task executions for training and adds minimal runtime computation by reusing activations already computed during VLA inference.Task progress can be labeled from offline successful data without human supervision or explicit failure data.Source: §V · p. 4

VI. EXPERIMENTS: SIMULATION

The simulation studies assess probe efficiency, failure discrimination, generalization, and comparison with VLM-based monitoring. Experiments use a fine-tuned π0.5 policy and compare monitors with different inputs, features, and training supervision.

Simulation Study Goals

  • The simulation studies compare monitoring efficiency, failure discrimination, generalization to unseen tasks, and progress probes against VLM-based monitors.They also compare probes over VLA internals learned using robot actions with probes over VLM features trained only on images and text.Source: §VI · p. 4

Setup

  • The π0.5 policy was fine-tuned on 5,000 trajectories across 10 VLABench manipulation tasks; monitoring evaluation used six tasks with non-negligible policy success.The dataset contained 500 trajectories per task.Source: §VI · p. 4

Training

  • The probe training set comprised 300 trajectories, with 50 per task across six tasks, and an 80/20 train-validation split.The validation split calibrated the detection threshold following.Source: §VI · p. 4

Evaluation

  • Evaluation used 600 rollouts, evenly split between successful and failed executions, with failure onset manually labeled for frame-level evaluation.Each failed rollout’s later timesteps were marked as failures.Source: §VI · p. 4

Baselines

  • Baselines span input-, feature-, and output-level monitors, and include both unsupervised methods and SAFE methods trained with failure labels.Robometer provides a progress-estimation comparison using large-scale success-and-failure data, with zero-shot and fine-tuned evaluations; SAFE also uses additional failed trajectories for supervision and calibration.Source: §VI · p. 4

A. On the Quality and Efficiency of Our Progress Probe

In simulation, Progress Probe combines competitive failure discrimination with low inference latency, and remains competitive on unseen tasks and detection timing. The evaluation also compares its accuracy and speed with other runtime monitors.

Result #1: Efficiency

Result #2: Failure Discriminability

  • Episode AUROC 0.909 and State AUROC 0.788: Progress Probe achieves these scores without failure-label training and remains competitive with supervised SAFE baselines.Feature-level monitors achieve the strongest overall monitoring performance, suggesting VLA activations contain informative signals for distinguishing successful from failed executions.Source: A. On the Quality and Efficiency of… · p. 5, Table I · p. 5

Result #3: Generalization to Unseen Tasks

  • Episode AUROC 0.900 and State AUROC 0.794: Progress Probe remains competitive on unseen tasks.The comparison reports SAFE-MLP at 0.896 Episode AUROC and 0.866 State AUROC, and SAFE-LSTM at 0.859 Episode AUROC and 0.803 State AUROC.Source: A. On the Quality and Efficiency of… · p. 5

Result #4: Detection Time

  • TPR reaches 0.54 at 75% and 0.64 at 100% of the episode; average failure detection occurs after 0.60 of the episode.Detection timing is reported alongside Table II, and is described as comparable to SAFE-LSTM at 0.53 despite using only the current activation rather than recurrent temporal history.Source: A. On the Quality and Efficiency of… · p. 5, Table II · p. 5

B. VLA Progress Probes vs. VLMs as Runtime Monitors

The study compares Progress Probe with Robometer as runtime monitors, including Robometer’s progress and success heads. Progress Probe and fine-tuned Robometer have comparable success–failure discriminability, while the probe is substantially smaller and faster.

Baselines

  • Robometer’s Progress head estimates current task progress, while its Success head estimates success probability and is primarily intended as a terminal signal.The Progress head’s nominal trajectories are calibrated from its predictions on successful training trajectories.Source: B. VLA Progress Probes vs. VLMs as… · p. 6
  • The comparison includes zero-shot Robometer and a fine-tuned variant trained on the same success-and-failure dataset used to train SAFE.Robometer is treated as a task-progress reference because it was trained on diverse robot data, including successful, failed, and intermediate-progress examples.Source: B. VLA Progress Probes vs. VLMs as… · pp. 5–6

Results

  • Progress Probe and Robometer-FT achieve comparable success–failure discriminability, while Progress Probe fine-tunes 2K parameters versus 4B and runs over 10,000× faster at inference.Robometer is evaluated in zero-shot and fine-tuned variants, with its progress and success heads used in the monitoring framework.Source: B. VLA Progress Probes vs. VLMs as… · pp. 5–6

C. Does Robot Action Data Matter for Progress Probing?

The section tests whether robot action data contributes to progress probing by comparing probes trained on VLA and VLM activations. π0.5 leads the reported progress-prediction and failure-detection comparisons, while robot-visual training alone does not improve progress prediction over a standard VLM.

Methods

  • The comparison probes VLA activations against activations from zero-shot and fine-tuned Robometer and its 3B PaliGemma backbone.The study measures progress prediction with R² on successful evaluation rollouts and also measures success–failure AUROC.Source: C. Does Robot Action Data Matter for… · p. 6

Results

  • R² 0.83 for π0.5 progress prediction exceeds P² Robo-ZS (0.63), P² Robo-FT (0.76), and P² Pali (0.69) on successful evaluation rollouts.Models trained on robot visuals, such as Robometer, do not improve progress prediction over a standard VLM.Source: C. Does Robot Action Data Matter for… · p. 6
  • AUROC 0.93 for π0.5 exceeds P² Robo-ZS (0.61), P² Robo-FT (0.81), and P² Pali (0.88) on the simulation evaluation set.The authors hypothesize that the difference is related to the VLA training objective and argue that action data matters for assessing rollout success or failure.Source: C. Does Robot Action Data Matter for… · p. 6

VII. EXPERIMENTS: HARDWARE

The hardware experiments evaluate progress-probe monitoring with π0.5 on six manipulation tasks using a Franka Research 3 arm and DROID cameras. Probe training uses the policy’s activation data from successful task executions, and runtime evaluation includes both successful and failed rollouts.

Setup

  • The hardware setup uses a Franka Research 3 arm, DROID cameras, and a π0.5-DROID checkpoint fine-tuned on six tasks with 60 trajectories per task.The fine-tuning dataset contains 360 trajectories in total.Source: §VII · p. 6
  • The six tasks are Candy Bag, Coffee Bag, Coffee Bean, Knife Put, Block Stack, and Stacked Utensil.They involve pick-and-place, pouring, stacking, and utensil manipulation tasks.Source: §VII · p. 6

Training

  • Probe training uses activation data from the 360 hardware fine-tuning trajectories, labels them with the progress labels from Eq. 2, and uses an 80/20 training-validation split.Source: §VII · p. 6

Evaluation

  • Evaluation uses 120 policy rollouts, 20 per task, with an even success-failure split; failed episodes are manually labeled from the failure point onward for frame-level evaluation.The hardware figure reports episode-level and state-level AUROC for failure detection.Source: §VII · pp. 6–7, Figure 4 · p. 7

A. Discriminability by Failure Detection

On hardware, Progress Probe achieves higher episode- and state-level AUROC than all compared baselines, including methods trained with additional failure data. Qualitative trajectories show its predictions falling behind nominal progress near failure while remaining in the nominal range on successful runs.

Results

  • Progress Probe achieves higher episode- and state-level AUROC than all baselines, including Robometer-FT and SAFE, which use additional failure trajectories and labels.The hardware runtime-monitoring figure reports episode-level and state-level failure-detection AUROC.Source: A. Discriminability by Failure Detection · p. 7
  • On Knife Put, predicted progress falls behind the nominal trajectory and crosses the detection boundary near failure onset, while successful rollouts remain within the nominal range.The qualitative figure contrasts failed trajectories and predictions in red with successful-trajectory predictions in green.Source: A. Discriminability by Failure Detection · p. 7, Figure 5 · p. 7
  • Progress Probe has its highest reported hardware AUROC on Block Stack (0.87) and Stacked Utensil (0.81), but underperforms on challenging Coffee Bean failures that can be stochastic and visually subtle.The authors hypothesize that activations may not change enough to provide a strong signal in such cases, and note that internal features and visual task-progress cues may provide complementary signals.Source: A. Discriminability by Failure Detection · p. 7

B. How Well Do P 2 Monitors Generalize to Unseen Tasks?

P² achieved the highest held-out-task State AUROC across 20 seen/unseen splits, and the authors identify transfer to other VLA families and action-expert activations as open directions.

B. How Well Do P 2 Monitors Generalize to Unseen Tasks?

  • Each split held out three of six tasks; probes and baselines were trained on the other three and evaluated on unseen rollouts at the State level.Robometer was excluded because it is typically zero-shot, making all tasks unseen for it.Source: B. How Well Do P 2 Monitors Generalize… · pp. 7–8
  • 0.81 State AUROC was the highest averaged across all 20 seen/unseen task splits for P² on held-out tasks.Per-task results also showed consistent generalization across held-out tasks.Source: B. How Well Do P 2 Monitors Generalize… · p. 8

VIII. CONCLUSION AND LIMITATIONS

  • Across simulation and hardware experiments, P² achieved competitive failure-monitoring performance while being substantially more computationally efficient than existing VLM-based monitors.The authors conclude that task progress encoded in VLA activations provides a simple runtime-monitoring signal.Source: §VIII · p. 8
  • Experiments focused on π0.5 and primarily on VLM-backbone activations, leaving transfer across VLA families and complementary action-expert signals for future work.The paper names GR00T and MolmoAct as other VLA families for future transfer investigations.Source: §VIII · p. 8
Loading 2610.31613v1…