PaperlayerView exact revision on arXiv

Source-linked AI summary

Decision-Aware Uncertainty Evaluation of Vision-Language Model-Based Early Action Anticipation for Human-Robot Interaction

Zhaoda Du, Michael Bowman, Qiaojie Zheng, Xiaoli Zhang

arXiv:2603.10061v2cs.RO

TL;DR

Early VLM action predictions in HRI face a reliability gap: confidence under partial, temporal-prefix observations is not well characterized, despite its relevance to decisions. The paper evaluates that uncertainty through a decision-aware protocol and finds that aggregation reshapes confidence geometry, with trade-offs between calibration and decision-level separability. Its interaction analysis uses simulated confidence-gated policies rather than real robotic deployment.

  • Problem

    Whether VLM confidence reliably reflects uncertainty in temporal-prefix Top-K action prediction remains largely unexplored, although HRI evaluation must account for multi-intent ambiguity and confidence structure.

    Source: A. VLM-Based Early and Egocentric… · p. 2, §I · p. 2

  • Method

    The paper evaluates VLM uncertainty in temporal-prefix action prediction using stochastic decoding and a decision-aware framework covering correctness, reliability, selective utility, and confidence geometry.

    Source: §I · pp. 1–2, A. Uncertainty Generation: Stochastic… · p. 2, C. Evaluation Protocol · p. 3

  • Results

    Aggregation reshapes confidence geometry, trading off calibration fidelity against decision-level separability, and improved ranking performance does not necessarily imply improved uncertainty reliability.

    Source: §I · p. 2, §VII · p. 8

  • Takeaways & Limitations

    The findings support decision-aware uncertainty evaluation when assessing early multi-intent anticipation for confidence-gated HRI integration.

    Source: §VII · p. 8

    The interaction analysis uses simulated confidence-gated policies rather than real robotic deployment, which involves additional operational constraints and human feedback dynamics.

    Source: §VI · p. 8

Abstract

from arXiv · show

Robots in shared workspaces must interpret human actions from partial, ambiguous observations, where overconfident early predictions can lead to unsafe or disruptive interaction. This challenge is amplified in egocentric views, where viewpoint changes and occlusions increase perceptual noise and ambiguity. As a result, downstream human-robot interaction modules require not only an action hypothesis but also a trustworthy estimate of confidence under partial observation. Recent vision-language model-based approaches have been proposed for short-term action recognition due to their open-vocabulary and context-aware reasoning, but their uncertainty reliability in the temporal-prefix regime is largely uncharacterized. We present the first systematic evaluation of uncertainty in vision-language model-based short-term action recognition for human-robot interaction. We introduce a temporal-prefix evaluation protocol and metrics for calibration and selective prediction. We also characterize miscalibration patterns and failure modes under partial observations. Our study provides the missing reliability evidence needed to use vision-language model predictions in confidence-gated human-robot interaction modules.

I. INTRODUCTION

Early action anticipation in HRI must handle ambiguous partial observations without committing too soon or waiting indefinitely. The paper evaluates whether VLM confidence in temporal-prefix predictions is reliable enough to inform such decisions.

I. INTRODUCTION

  • In shared workspaces, ambiguous initial motion makes intent difficult to infer; committing too early can be unsafe or disruptive, while waiting for more evidence can also impede interaction.The challenge includes early, evolving observations and egocentric settings with additional noise.Source: §I · p. 1
  • The paper presents a systematic evaluation of uncertainty in VLM-based short-term action prediction for HRI, using a temporal-prefix protocol and calibration and selective-prediction metrics.It also characterizes miscalibration patterns and failure modes under partial observation.Source: §I · p. 1
  • The framework evaluates Top-K outputs through correctness, uncertainty reliability, selective decision utility, and confidence geometry rather than treating anticipation as only a ranking problem.It is intended to assess whether confidence signals suit confidence-gated interaction.Source: §I · p. 2
  • Aggregation strategies reshape uncertainty geometry and trade off calibration fidelity against decision-level separability; better ranking performance does not necessarily mean more reliable uncertainty.The introduction frames these findings as a reason to assess confidence as well as predictive ranking.Source: §I · p. 2

A. VLM-Based Early and Egocentric Action Understanding

  • VLM-based early action prediction has mainly been assessed by ranking accuracy, while whether its confidence reflects ambiguity and decision-level reliability remains largely unexplored.This gap matters because HRI decisions may involve multiple plausible intents under partial observation.Source: A. VLM-Based Early and Egocentric… · p. 2

B. Uncertainty-Aware Decision Making in HRI

The paper focuses on the uncertainty signals upstream modules need for HRI decisions, rather than proposing a new interactive planner. It reframes evaluation around the validity and structure of Top-K confidence in early, multi-intent settings.

B. Uncertainty-Aware Decision Making in HRI

  • Uncertainty-aware HRI methods often assume reliable upstream confidence, so this work evaluates the reliability and mechanisms behind VLM confidence in short-term action recognition instead of proposing a planner.The evaluation targets a precursor for downstream uncertainty-aware decision modules.Source: B. Uncertainty-Aware Decision Making… · p. 2

C. Uncertainty Evaluation

  • Early anticipation requires evaluating whether confidence reflects set-level validity and its distribution across ranked hypotheses, not only single-label correctness.Collaborative control may reason over multiple candidate actions before execution.Source: C. Uncertainty Evaluation · p. 2
  • The paper organizes evaluation around candidate validity, set-level calibration, selective execution behavior, and confidence geometry for interaction-aware reliability.This shifts the focus from purely probabilistic fidelity to decision-relevant properties.Source: C. Uncertainty Evaluation · p. 2

A. Uncertainty Generation: Stochastic Multi-Run Sampling

The method estimates uncertainty from variability across repeated stochastic Top-K predictions, then aggregates those outputs into a ranked prediction with confidence scores. It compares rank-wise voting approaches with a method that models global ranking structure.

A. Uncertainty Generation: Stochastic Multi-Run Sampling

  • Because VLM predictive distributions are typically unavailable, repeated stochastic decoding serves as an implicit sampling mechanism for approximating uncertainty without changing the model architecture.Variation across runs acts as a proxy: consistent outputs suggest confidence, while greater variation indicates increased uncertainty.Source: A. Uncertainty Generation: Stochastic… · p. 2

B. Aggregation of Stochastic Top-K Predictions

  • The pipeline aggregates stochastic Top-K outputs into one ranked prediction with associated confidence scores, with the strategies summarized in Table I.Fig. 2 describes the transformation from stochastic variability to structured confidence geometry.Source: B. Aggregation of Stochastic Top-K… · p. 2, Figure 2 · p. 3, Table I · p. 3
  • Consistency-based aggregation selects the action with majority support at each rank and uses agreement frequency as its confidence signal.The aggregated action at each position is determined from agreement across stochastic runs.Source: 1) Consistency-Based Aggregation · pp. 2–3
  • Confidence-weighted aggregation uses the model-reported confidence for each run’s action at a rank to compute a weighted vote and normalized support.Unlike consistency-based aggregation, it incorporates verbalized confidence.Source: 2) Confidence-Weighted Aggregation · p. 3
  • PairRank models global ranking structure by aggregating pairwise action preferences and fitting latent utility scores with a Bradley-Terry model.It differs from strategies that operate independently at each rank; the resulting scores rank actions and provide confidence.Source: 3) Pairwise Ranking Aggregation… · p. 3

C. Evaluation Protocol

The protocol evaluates Top-K action anticipation through correctness, confidence reliability, selective decision utility, and confidence geometry. It assesses whether confidence supports decisions under partial observation, not just whether predictions rank correctly.

C. Evaluation Protocol

  • The framework evaluates Top-K uncertainty across correctness, reliability, selective utility, and confidence geometry.These dimensions assess candidate validity, calibration fidelity, decision gating, and ambiguity modeling.Source: C. Evaluation Protocol · p. 3, Table II · p. 4

Correctness

  • Top-1 Accuracy checks whether the highest-ranked action matches ground truth, while Recall@K checks whether ground truth appears in the predicted set.Both assess ranking quality, not whether confidence scores are meaningful.Source: C. Evaluation Protocol · p. 3

Uncertainty Reliability

  • Top-1 ECE measures alignment between top-action confidence and empirical correctness, while set-level calibration concerns whether ground truth appears within Top-K.The protocol evaluates calibration at both single-hypothesis and set levels.Source: C. Evaluation Protocol · p. 4

Set-Level Calibration

  • Set-ECE compares mean Top-K confidence with empirical set correctness across confidence bins; lower values indicate better alignment.The protocol also reports Set-ECE as a function of K to assess robustness as ambiguity increases.Source: C. Evaluation Protocol · p. 4

Selective Decision Utility

  • The selective policy executes the top-ranked action above a confidence threshold and measures coverage and accuracy on retained predictions.As the threshold rises, useful confidence should reduce coverage while increasing selective accuracy.Source: C. Evaluation Protocol · p. 4

Diagnostic Analyses: Confidence Geometry

  • Confidence geometry analysis examines rank-wise confidence allocation and entropy to characterize ambiguity among competing action hypotheses.Structured decay without excessive concentration is desirable; low entropy indicates concentrated belief, while high entropy indicates distributed ambiguity, and either extreme may be undesirable.Source: 4) Diagnostic Analyses: Confidence… · pp. 4–5

IV. EXPERIMENTS SETUP

The experiments evaluate uncertainty aggregation for Top-K short-term egocentric action anticipation in a black-box VLM setting.

IV. EXPERIMENTS SETUP

  • The experimental focus is uncertainty aggregation for Top-K short-term egocentric action anticipation.The section covers dataset, model interface, sampling protocol, aggregation configuration, and evaluation implementation.Source: §IV · p. 5
  • The evaluation uses a black-box VLM setting.The passage identifies this as the setting for evaluating aggregation behavior.Source: §IV · p. 5
  • The setup includes a sampling protocol and an aggregation configuration for the evaluation.It also describes the model interface and implementation details.Source: §IV · p. 5

A. Datasets

The study evaluates immediate next-action anticipation on EGTEA Gaze+ and EPIC-KITCHENS-100 using visual observations before action onset.

A. Datasets

  • The benchmarks are EGTEA Gaze+ and EPIC-KITCHENS-100, both egocentric action anticipation datasets.EGTEA Gaze+ has approximately 10K annotated first-person action segments; EPIC-KITCHENS-100 has over 90K segments.Source: A. Datasets · p. 5
  • EGTEA Gaze+ contains approximately 10K annotated first-person action segments with verb-noun labels in kitchen environments.The dataset uses first-person action segments.Source: A. Datasets · p. 5
  • EPIC-KITCHENS-100 contains over 90K action segments collected across diverse subjects and environments.The evaluation uses official train/test splits and predicts the immediate next action from observations preceding action onset.Source: A. Datasets · p. 5

B. Vision-Language Models

The study uses Gemini 2.5 Flash-lite through a black-box API to generate ranked action predictions from visual observations and natural-language prompts.

B. Vision-Language Models

  • The evaluated model is Gemini 2.5 Flash-lite, accessed through a black-box API.The passage identifies it as a state-of-the-art VLM.Source: B. Vision-Language Models · p. 5
  • The model accepts visual observations and natural-language prompts and returns ranked action predictions in text form.Source: B. Vision-Language Models · p. 5
  • The lightweight model variant enables scalable uncertainty analysis with multi-run stochastic sampling and Top-K extraction over large-scale datasets.The passage frames this choice in terms of realistic deployment constraints.Source: B. Vision-Language Models · p. 5

C. Sampling Protocol

The protocol uses repeated stochastic VLM decoding to expose variability in ranked action predictions.

C. Sampling Protocol

  • Each input segment receives M = 5 stochastic decoding runs with temperature T = 0.8 unless otherwise specified.Runs use identical prompts; temperature-based decoding introduces controlled stochasticity while preserving semantically plausible predictions.Source: C. Sampling Protocol · p. 5
  • Each decoding run produces a ranked Top-K action list, with K = 10 as the default.Source: C. Sampling Protocol · p. 5
  • Repeated queries expose stochastic variability across predictions for each input segment.Source: C. Sampling Protocol · p. 5

D. Evaluation Implementation

Across the two datasets, aggregation has modest effects on ranking, while calibration and selective behavior vary by strategy and decision horizon.

D. Evaluation Implementation

A. Correctness

  • PairRank achieves the highest Recall@10, while Top-1 accuracy remains comparable across methods.Aggregation primarily reshapes candidate ordering rather than immediate prediction correctness.Source: A. Correctness · p. 5

B. Uncertainty Reliability

  • The single-run baseline has the lowest Top-1 ECE on both datasets, while PairRank has the largest error at K = 1.The baseline directly uses one stochastic decoding pass without aggregation.Source: B. Uncertainty Reliability · p. 5, §V · p. 5
  • PairRank’s Set-ECE falls faster as K increases, reaching its EGTEA Gaze+ minimum around K = 3–4 and continuing to improve on EPIC-KITCHENS-100.Source: B. Uncertainty Reliability · p. 5
  • Calibration should be judged at the intended Top-K decision horizon, where moderate K may improve PairRank reliability despite weaker Top-1 calibration.Source: B. Uncertainty Reliability · p. 5

C. Selective Decision Utility

  • PairRank maintains higher accuracy as confidence thresholds rise, while its sharper coverage decline separates retained predictions more decisively than averaging-based strategies.Source: C. Selective Decision Utility · p. 5
  • This selective decisiveness comes despite weaker Top-1 calibration, creating a trade-off between calibration and threshold-based abstention.Source: C. Selective Decision Utility · p. 5

D. Confidence Geometry

Aggregation strategies create distinct confidence distributions, with consequences for how robots may manage execution, clarification, and interaction burden.

D. Confidence Geometry

  • PairRank yields the lowest normalized Top-10 entropy on both datasets, concentrating confidence at rank 1 and suppressing lower-ranked candidates.Consistency and confidence-weighted methods distribute confidence more evenly, producing higher entropy and smoother decay.Source: Table III · p. 6, Figure 4 · p. 6, D. Confidence Geometry · p. 7
  • Confidence structure can inform whether a robot executes, seeks clarification, or gathers information, with low entropy indicating clearer intent and high entropy reflecting ambiguous multi-intent states.Source: D. Confidence Geometry · p. 7

E. Interaction-Level Implications

  • Interaction-level behavior depends on how Top-K predictions combine with confidence rules, including K and threshold τ, not ranking performance alone.These choices shape clarification scope, interaction burden, and risk of overconfident misprediction.Source: E. Interaction-Level Implications · p. 7
  • No aggregation strategy is universally optimal: sharper confidence may support targeted confirmation but risks severe misjudgment, while smoother outputs can increase clarification overhead.Low-confidence outputs may suppress interaction under threshold-based gating; strategy, K, and threshold should reflect domain requirements and interaction constraints.Source: E. Interaction-Level Implications · p. 7

F. Summary of Findings

Aggregation reshapes confidence geometry, creating trade-offs between calibration, decision-level separability, and interaction behavior. The study also identifies limits in model coverage, fixed evaluation settings, and simulated interaction analysis.

F. Summary of Findings

  • Aggregation reshapes confidence geometry: sharp distributions may cause overconfident misprediction, while smoother ones expand clarification and interaction burden.Under threshold-based control, these differences lead to distinct HRI behaviors even when Top-K rankings appear similar.Source: F. Summary of Findings · p. 7, Figure 5 · p. 7

VI. LIMITATIONS AND FUTURE WORK

  • The study evaluates uncertainty rather than architecture performance, using one representative black-box VLM; stronger or domain-adapted models remain applicable within the framework.Comparing multiple VLM architectures is outside the study’s primary focus.Source: §VI · pp. 7–8
  • Top-K size, stochastic temperature, and sampling count are fixed for controlled comparison, leaving adaptive or context-aware selection for future work.Source: §VI · p. 8
  • Interaction analysis uses simulated confidence-gated policies rather than real robotic deployment, where operational constraints and human-feedback dynamics also matter.Closed-loop robotic studies are identified as future work.Source: §VI · p. 8

VII. CONCLUSION

  • The evaluation provides empirical guidance for confidence-gated HRI integration and decision-aware uncertainty assessment in early multi-intent anticipation.Aggregation induces trade-offs between calibration fidelity and decision-level separability.Source: §VII · p. 8
Loading 2603.10061v2…