PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-hoc Epistemic Uncertainty for Vision-Language-Action Policies

Johannes Keller, Albrecht M. Bloße, Adrian Lindenmeyer, Thomas Neumuth, Daniel Schneider

arXiv:2610.29730v1cs.ROcs.LGstat.ML

TL;DR

VLA policies need to identify situations their training demonstrations do not support well so they can seek assistance or additional demonstrations. This paper fits post-hoc GP kernel-variance heads to embeddings from one trained policy, without changing the policy. On SmolVLA and LIBERO-10, the heads are competitive with two-policy ensembles, and controlled perturbations show modality-specific detection and stronger within-task referral by the decoder head.

  • Problem

    VLA policies encounter tasks, scenes, and instructions only partially covered by training demonstrations, creating a need to estimate how poorly the training data support a situation.

    Source: §1 · p. 1

  • Method

    Post-hoc GP kernel-variance heads estimate uncertainty from a single policy’s embeddings, using training-demonstration embeddings as support without regression targets or policy updates.

    Source: §1 · p. 2

  • Results

    On SmolVLA and LIBERO-10, kernel-variance heads perform competitively with two-policy ensemble scores, and under controlled perturbations KV-Dec improves within-task referral over VFD to a supervisor trained for those perturbations.

    Source: §1 · p. 3, §6 · p. 20

  • Takeaways & Limitations

    Useful uncertainty estimates for adaptive robot learning can be obtained from a single policy, while unfamiliarity and the benefit of assistance remain distinct questions.

    Source: §6 · p. 20

    The evaluation covers LIBERO-10 tasks and three training seeds, leaving generalization to other architectures, real robots, and real-world conditions untested.

    Source: §5.8 · p. 19

Abstract

from arXiv · show

Vision-language-action (VLA) policies need to recognize gaps in their training knowledge to request useful demonstrations or assistance in new situations, especially when failures are costly. Disagreement within an ensemble of independently trained policies can guide these decisions, but requires training and running multiple policies. We propose post-hoc kernel-variance heads that estimate epistemic uncertainty from the embeddings of a single policy. Each head computes the conditional variance of a Gaussian process conditioned on embeddings from the training demonstrations. Fitting requires no regression targets or gradient updates and leaves the policy unchanged. In experiments with SmolVLA on LIBERO-10, the kernel variance heads perform competitively with two-policy ensemble scores in task-level correlation with task success, in demonstration acquisition during active learning, and in referral to a stronger supervisor policy as a proxy for human assistance. For SmolVLA, the head on the vision-language backbone reduces uncertainty-scoring time by up to approximately a factor of 50 compared with ensemble scores, while also substantially reducing setup costs. We also show that the established evaluation protocol on LIBERO-10 primarily rewards recognition of coverage differences between tasks, while execution variability weakens assessment of within-task rankings. Thus, controlled perturbations provide a complementary evaluation: modality-specific heads identify unfamiliar camera, instruction, or state inputs, while fused heads can detect unfamiliar combinations of individually familiar inputs. With a supervisor proxy trained on the evaluated perturbation families, the decoder head improves within-task referral over ensemble disagreement. These findings show that useful uncertainty estimates for adaptive robot learning can be obtained from a single policy.

1 Introduction

VLA policies need to detect when their training demonstrations provide insufficient support, so they can request assistance or new demonstrations. This section introduces post-hoc kernel-variance heads that use a single policy’s embeddings, then outlines evaluations of their utility and cost.

1 Introduction

  • The motivation is adaptive response to situations that training demonstrations only partially cover, especially when failures may be costly.Epistemic uncertainty reflects how poorly training data support a situation and can shrink with relevant demonstrations.Source: §1 · p. 1
  • Prior LIBERO work used ensemble-based VFD to guide SAVE demonstration acquisition, but required at least two policies to be trained and run.SAVE with VFD needed at least 22% fewer demonstrations than compared acquisition strategies.Source: §1 · p. 2
  • Kernel-variance heads estimate uncertainty from a single policy’s embeddings using GP conditional variance, without regression targets or gradient updates and without changing predicted actions.Fitting requires one feature-extraction pass over support observations and one Cholesky factorization; separate heads can read modality-specific representations.Source: §1 · p. 2
  • The heads reduce setup time per fine-tuning round by up to about 44 times and scoring time by up to about 50 times versus ensemble scores.They require one support-observation feature pass and one matrix factorization rather than training and running a second policy.Source: §1 · p. 2
  • On LIBERO, the heads perform comparably to two-policy ensemble scores for task-failure ranking and demonstration selection, and match or exceed them in supervisor referral.These evaluations mainly reward recognizing task-level demonstration coverage; execution noise weakens evidence about within-task rankings.Source: §1 · p. 3
  • Controlled modality and combination perturbations test whether heads detect unfamiliar inputs and improve within-task referral to a stronger policy.Context-specific heads can indicate which input modality is unfamiliar.Source: §1 · p. 3

2 Related Work

Prior work spans uncertainty estimates based on model weights, feature-space distances, and generative-policy outputs, alongside methods for acquisition and deferral. The paper situates its post-hoc GP variance heads as coverage estimates on frozen policy embeddings, including modality-specific representations.

Epistemic uncertainty and distance awareness

  • Epistemic uncertainty concerns limited knowledge about the function or actions implied by demonstrations, whereas aleatoric uncertainty reflects inherent randomness that more data cannot reduce.For robot policies, aleatoric uncertainty can arise from multiple valid demonstrations or stochastic contact outcomes.Source: §2 · p. 3
  • Distance-aware single-model methods can match or improve on deep ensembles at detecting unfamiliar inputs, but rely on representations that keep unfamiliar inputs distinct.GP methods and deterministic feature-distance scores differ in whether they define a posterior over predictive functions.Source: §2 · p. 4
  • The kernel-variance head places an explicit GP prior over functions on frozen policy embeddings; its conditional variance measures coverage in that embedding geometry without output targets.The score vanishes at support inputs and approaches prior variance far from them; its interpretation depends on embeddings preserving relevant distinctions.Source: §2 · p. 4

Uncertainty for generative policies and VLAs

  • VLA uncertainty work includes execution-time failure detectors and confidence calibration, while these heads can assess modality-specific unfamiliarity before cross-modal mixing.The paper presents separate coverage estimates for individual VLA input modalities as a novel use.Source: §2 · p. 4

Active learning and interactive imitation

  • The paper builds on SAVE by replacing its VFD uncertainty score with kernel-head variance while keeping its demonstration-selection rule.Prior active-learning and interactive-imitation methods use uncertainty, diversity, or risk to select data or query experts.Source: §2 · p. 5

Selective prediction and deferral

  • Selective-prediction risk–coverage evaluations cannot distinguish epistemic from aleatoric uncertainty, and referral scores can receive credit for ambiguity unless evaluated against an imperfect expert’s actual gain.The paper’s fixed-budget referral metric uses a stronger policy’s possibly negative performance gain on referred conditions.Source: §2 · p. 5

3 Methods

The methods attach post-hoc kernel-variance estimates to embeddings from a frozen policy, then evaluate them through demonstration acquisition, calibration, referral, and perturbation protocols.

3.1 Kernel-variance heads

  • Kernel-variance heads score a frozen policy’s embeddings by conditioning a Gaussian-process variance on training-demonstration support inputs, without execution outcomes or regression targets.The score is computed with a Cholesky factorization; it is high far from support and low at an isolated support input.Source: §3.1 · pp. 5–6
  • Context-specific heads use raw or fused camera, instruction, and state representations to attribute unfamiliarity to individual inputs or their mixed representations.Fused representations have already integrated information across inputs.Source: Figure 1 · p. 6, §3.1 · p. 7

3.2 Demonstration acquisition with SAVE

  • SAVE scores candidate conditions, samples tasks in proportion to their uncertainty, requests each selected task’s most uncertain demonstration, and refits heads after each fine-tuning round.Acquired demonstrations remain eligible candidates, so a query can return a demonstration the policy already has.Source: §3.2 · p. 7

3.3 Evaluation metrics

  • Task-level calibration correlates task-mean uncertainty with task success, with a demonstration-count reference that uses task labels and is not a usable uncertainty score.The reported correlations are negative Spearman and Pearson, with higher values indicating better calibration.Source: §3.3 · pp. 7–8
  • Referral is evaluated by normalized gain over random referral relative to a hindsight oracle, both globally and within tasks, with automated fraction as an operating-point measure.Within-task referral gives zero to scores constant within each task; diagnostics also assess primary failures and reproducible condition-level outcome differences.Source: §3.3 · p. 8

3.4 Experimental protocol

  • The study compares acquisition arms and scores on common policies across three seeds, and tests post-hoc heads on controlled perturbations; individual perturbation comparisons are exploratory.The perturbation confidence intervals are not adjusted for multiple comparisons.Source: §3.4 · pp. 8–10

4 Results

The results compare single-policy kernel-variance heads with ensemble scores in calibration, demonstration acquisition, and referral, then examine what standard LIBERO-10 evaluations measure and how heads respond to controlled perturbations.

4.1 Task-level calibration

  • Kernel heads were on par with VFD in task-level calibration, while demonstration counts correlated with success at least as strongly as every learned score.The reference correlations were −ρS = 0.82 and −rP = 0.83.Source: §4.1 · pp. 10–11

4.2 Demonstration acquisition and cost

  • SAVE with KV-Ctx reached 65.7 ± 1.8% final success, level with GU at 64.4 ± 3.1% and above VFD at 61.9 ± 2.5%.Three seeds do not establish equivalence or a consistent advantage among these arms.Source: Table 2 · p. 11, §4.2 · p. 11
  • All 75 kernel-head-guided queries returned unique new demonstrations, compared with 69.3 for VFD, 62.7 for GU, and 46.3 for Random.After refitting, acquired observations fell to the variance floor set by ridge regularization, preventing repeat requests in these experiments.Source: §4.2 · p. 12
  • Across 15 rounds, KV-Ctx required about 15 GPU-minutes versus VFD's 15 GPU-hours, including second-policy pre-training.Per round, KV-Ctx took about one GPU-minute to embed, refit, and score; VFD required 9.0 minutes for ensemble scoring plus 34.8 minutes to fine-tune the second policy.Source: §4.2 · p. 12

4.3 Supervisor referral

  • At round 10, global referral nAUC was 0.50 ± 0.04 for KV-Ctx and 0.51 ± 0.07 for KV-Dec, matching or exceeding ensemble scores.Paired comparisons support both heads over GU; their differences from VFD include zero.Source: §4.3 · p. 12, Table 3 · p. 13

4.4 Task coverage versus within-task ranking

  • Task differences explained 71–76% of outcome variance at round 0, while execution variability weakened within-task assessment; KV-Dec's round-10 failure AUROC was 0.61 [0.56, 0.66].After ten rounds, 44% of variance reflected executions of the same condition; the cross-execution reference reached within-task nAUC 0.27 ± 0.04.Source: Figure 4 · p. 13, §4.4 · pp. 13–14

4.5 Context-specific heads under controlled perturbations

  • Raw modality heads rose selectively for perturbations to their inputs, while fused heads also responded to unfamiliar combinations of individually familiar inputs.An instruction naming objects absent from the scene left raw heads at chance but raised fused heads and whole-policy scores.Source: Table 4 · p. 14, §4.5 · p. 15

4.6 Referral under controlled within-task variation

  • Under trained perturbations, KV-Dec reached within-task nAUC 0.46 ± 0.03, exceeding VFD at 0.40 ± 0.03; KV-Ctx reached 0.40 ± 0.05.KV-Dec exceeded VFD by 0.06 [0.03, 0.10] in nAUC; KV-Ctx showed no significant nAUC difference from VFD.Source: Table 6 · p. 16, §4.6 · p. 17

5 Discussion

The kernel heads perform competitively with ensemble scores in task-level calibration and referral, and in SAVE they show similar acquisition performance. They also reduce setup and scoring computation, with costs depending on the head and support size.

5.1 Calibration and referral (Q1)

  • Both kernel heads perform competitively with ensemble scores in task-level calibration and referral, with advantages on particular metrics.Their task-success correlations are more linear than VFD and GU; in referral, both reach a higher automated fraction than VFD and GU on the supported comparisons.Source: §5.1 · p. 17

5.2 Demonstration acquisition (Q2)

  • SAVE with kernel heads shows similar acquisition performance, and every query guided by either head returns a new demonstration.Conditioning on acquired observations lowers their variance after refitting, encouraging later queries elsewhere; exact conditioning guarantees this drop.Source: §5.2 · p. 17

5.3 Computational cost (Q3)

  • Kernel heads are about 4–44 times faster to set up per acquisition round and about 4–50 times faster to use against VFD in evaluation scoring.Setup uses a frozen-policy support pass and kernel-matrix factorization; online, KV-Ctx adds a kernel evaluation, while KV-Dec reruns part of the action expert.Source: §5.3 · p. 17
  • For much larger supports, sparse variational GPs with inducing points reduce cost from cubic to linear in support size.Pivoted Cholesky bounds sparse variance inflation by a set tolerance at each support input, so acquired demonstrations still become certain.Source: §5.3 · pp. 17–18

5.4 What the established evaluations reward (Q4)

The established LIBERO evaluations mainly reward recognizing differences in demonstration coverage across tasks, while standard initial conditions provide a weak test of within-task ranking. The authors recommend reporting coverage references and separating global from within-task evaluation.

5.4 What the established evaluations reward (Q4)

  • The reproduced LIBERO evaluation mainly rewards recognizing how many demonstrations cover each task, so demonstration count reaches task-level calibration too.This coverage signal remains useful because demonstration counts are unavailable for new tasks and scenes at deployment.Source: §5.4 · p. 18
  • On standard LIBERO initial conditions, within-task ranking is largely inconclusive because task conditions differ little and execution variability exceeds reproducible outcome differences.High task-level calibration is necessary for prioritizing tasks but is a weak test of uncertainty quality.Source: §5.4 · p. 18
  • The authors recommend reporting a demonstration-count coverage reference, separating global from within-task metrics, measuring execution noise with repeated runs, and including meaningful within-task variation.They point to LIBERO-PRO, LIBERO-Plus, and their perturbed conditions as examples.Source: §5.4 · p. 18

5.5 Attributing unfamiliarity to input modalities (Q5)

Context-specific heads identify unfamiliar perturbations in the inputs they read, while fused heads detect some unfamiliar combinations of individually familiar inputs. Under meaningful within-task variation, referral can help when the supervisor is likely to succeed, though execution variability complicates gain estimates.

5.5 Attributing unfamiliarity to input modalities (Q5)

  • Raw context heads detect unfamiliar perturbations in the inputs they read, while fused-embedding heads detect unfamiliar combinations of individually familiar inputs.For example, an instruction naming objects absent from the scene leaves raw heads at chance but raises fused instruction and state heads; the raw state head also detects a position offset that whole-policy scores miss or only partly detect.Source: §5.5 · p. 18

5.6 When unfamiliarity supports within-task referral (Q5)

  • Within-task referral is possible when task conditions vary meaningfully; KV-Dec improves on VFD, while KV-Ctx performs comparably.The largest gains occur under perturbations the supervisor was successfully trained to handle, including wrist blackout, strong paraphrases, and object synonyms; unfamiliarity alone does not ensure useful referral.Source: §5.6 · p. 18
  • Execution variability makes gains measured in one execution noisy predictors of later gains, while input-based rankings support referral more effectively than a single-execution estimate.The single-execution estimate ranks conditions by the supervisor’s gain in one execution and evaluates on others.Source: §5.6 · pp. 18–19

5.7 Further findings

The kernel heads measure coverage in the policy’s embedding space, not whether the policy will fail. Across tasks, action-chunk entropy can correlate with demonstration coverage without representing epistemic uncertainty.

5.7 Further findings

  • Kernel-head unfamiliarity is not a measure of harm: perturbations detected almost perfectly can range from less than 20 points of success loss to complete failure.Detection strength correlates quite strongly with success loss overall, but scene changes detected only moderately cost about 40 points.Source: §5.7 · p. 19
  • Action-chunk entropy correlates with demonstration coverage across tasks despite not representing epistemic uncertainty, because wider action spread on tasks without demonstrations can make it track demonstration count.Before acquisition its Spearman correlation with demonstration count is 0.64; after ten rounds it drops to 0.21, while VFD reaches 0.82.Source: §5.7 · p. 19

5.8 Limitations

The evidence is limited to SmolVLA on a simulated ten-task benchmark, with several aspects of uncertainty estimation and referral evaluated only under specific supports, perturbations, and execution settings. The authors also identify representation, computational, and evaluation constraints that limit interpretation and generalization.

5.8 Limitations

  • The study covers one architecture, SmolVLA, on ten simulated target tasks with three training seeds; other architectures, physical robots, and real-world conditions remain untested.Three seeds estimate training variability coarsely, and three executions per condition leave success rates and supervisor gains noisy, so inconclusive comparisons do not establish equivalence.Source: §5.8 · p. 19
  • Kernel variance measures coverage under a chosen representation and GP prior, but its interpretation as policy epistemic uncertainty depends on embeddings preserving action-relevant distinctions.Feature collapse, token averaging, or dimensionality reduction can hide relevant differences; low variance does not establish successful action, and unfamiliar inputs may be harmless.Source: §5.8 · p. 19
  • Matched calibration and referral use only two random-acquisition checkpoints, correlate scores with task success rather than calibrated failure probabilities, and compare against a SAVE/VFD reproduction below its reported final success.Acquisition draws from a fixed recorded pool and permits repeated requests, so more unique demonstrations may reflect avoiding repeats rather than better selection among new demonstrations.Source: §5.8 · p. 19
  • Perturbation experiments use separate policies trained on all ten tasks and support observations sampled throughout demonstrations, under a different regime from the matched study.The perturbations are constructed, and the detection-to-success relationship is specific to their selection; the referral supervisor is trained on the same perturbation families, leaving unfamiliar failure mechanisms untested.Source: §5.8 · pp. 19–20
  • Referral tests pre-execution delegation to an automated supervisor proxy starting from the same initial state, not human assistance, in-execution handover, recovery, or intervention costs.Execution tests use two initial states per task and a fixed post-perturbation window; they do not establish warning before failure, episode-level false-alarm rates, or improved outcomes from responding to alarms.Source: §5.8 · p. 20
  • Context-specific heads identify unfamiliarity in individual representations but do not establish which input caused failure or which corrective intervention would help.Raw-head selectivity partly follows by construction, fused responses depend on cross-modal mixing and subsequent behaviour, and targeted assistance remains proposed.Source: §5.8 · p. 20
  • Head configurations came from limited exploratory studies, and comparisons with equally inexpensive distance or density scores are needed to isolate the contribution of exact GP conditioning.Reported savings apply to tested policies, hardware, and support sizes; exact conditioning has cubic factorization time and quadratic storage, supports above approximately 10,000 observations remain unevaluated, and sparse approximations are untested.Source: §5.8 · p. 20

6 Conclusion

The paper introduces post-hoc GP kernel-variance heads for estimating epistemic uncertainty from one VLA policy’s embeddings, without changing the policy. On SmolVLA and LIBERO-10 they are competitive with ensemble scores, while controlled perturbations and analysis clarify what these evaluations measure and where further testing is needed.

6 Conclusion

  • On SmolVLA and LIBERO-10, post-hoc kernel-variance heads performed competitively with two-policy ensemble scores across task-level calibration, demonstration acquisition, and supervisor referral, with lower computational overhead.KV-Ctx reduced scoring time by up to approximately 50 times and setup time per acquisition round by up to approximately 44 times compared with ensemble scores.Source: §6 · p. 20
  • The established protocol primarily reflects between-task demonstration-coverage differences, while execution variability weakens evidence about within-task rankings.The authors recommend coverage references, repeated executions, and separate reporting of global and within-task performance.Source: §6 · p. 20
  • Under controlled perturbations, context-specific heads identify unfamiliar modalities and fused heads detect unfamiliar combinations; KV-Dec improves within-task referral over VFD to a supervisor trained on those perturbations.Unfamiliarity supports useful referral only when the supervisor has a performance advantage; detection and the benefit of assistance are distinct.Source: §6 · p. 20
  • Future work should test other VLA architectures and physical robots, compare GP conditioning with simpler scores, and assess context-specific assistance during execution.Such tests should measure intervention timing, task success, and assistance costs alongside detection quality.Source: §6 · p. 20

A Head configurations and implementation

The appendix describes the implementation choices for the kernel-variance heads and notes that their configurations were selected informally rather than through systematic search.

A Head configurations and implementation

  • Configurations were selected from limited exploratory experiments, retaining variants that performed well rather than following a systematic search.KV-Ctx and context-specific heads remove duplicate support inputs before fitting, whereas KV-Dec retains them.Source: A Head configurations and implementation · p. 29
  • All main-experiment heads use exact GP conditioning on support observations and require no regression targets.Features are extracted in evaluation mode, one observation per call, to keep KV-Dec’s fixed initial noise independent of batch position.Source: Table 7 · p. 29, A Head configurations and implementation · p. 29

B Additional calibration and referral results

The appendix extends matched-study calibration and referral analyses with checkpoint-specific results and paired comparisons. Its intervals are interpreted for the evaluated tasks and population, with training-run variability estimated coarsely.

B Additional calibration and referral results

  • The appendix extends matched SAVE-study calibration and supervisor referral, scoring Random-arm primaries at rounds 0 and 10.Table 8 separates common-policy checkpoints, and Table 9 reports paired differences between kernel heads and ensemble scores with hierarchical bootstrap intervals.Source: B Additional calibration and referral… · p. 29, Table 8 · p. 29
  • A paired-difference interval excluding zero supports a difference for these tasks and this evaluation population, while three seeds estimate training-run variation only coarsely.Table 8 reports calibration and referral at each common-policy checkpoint as mean ± SD over three seeds.Source: B Additional calibration and referral… · p. 29, Table 8 · p. 29

C Context-head experiments

The appendix complements the context-head experiments with a perturbation library and paired comparisons between the paper’s heads and VFD.

C Context-head experiments

  • The appendix supplements context-head experiments in Sections 4.5 and 4.6 with perturbation and paired-comparison materials.The perturbation library is in Table 10 and paired comparisons with VFD are in Table 11.Source: C Context-head experiments · p. 29
  • Table 9 reports paired differences averaged over rounds 0 and 10, with 95% hierarchical bootstrap intervals from 2,000 draws across conditions within tasks and seeds.The draws are shared across scores.Source: Table 9 · p. 30
  • The perturbation library marks label-preserving changes, where demonstrated actions remain correct and the supervisor can be trained on the perturbation.P3 and P5 are released instruction paraphrases spanning close (P2) to distant (P5).Source: Table 10 · p. 30
  • Table 11 compares paper heads with VFD using paired differences, reporting referral intervals and counts of perturbations or events with significantly higher, similar, or lower AUROC.For starting observations, the comparison uses ten initial states per task; intervals are unadjusted for multiple comparisons, and perturbation-wise comparisons are exploratory.Source: Table 11 · p. 30

Related papers

Loading 2610.29730v1…