PaperlayerView exact revision on arXiv

Source-linked AI summary

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

arXiv:2609.25369v1cs.RO

TL;DR

Intent-confidence-based shared control can over-assist when a VLA is not reliable at executing the inferred task. Sem-Cap combines filtered VLM intent confidence with online VLA capability confidence in nonlinear authority allocation. It achieved the highest task success across evaluated methods and favorable subjective ratings among assisted methods.

  • Problem

    Intent-based arbitration can increase autonomous assistance with confidence in inferred intent, although that confidence does not establish that the robot can reliably execute the task.

    Source: §I · p. 1

  • Method

    Sem-Cap calibrates and filters VLM intent confidence, estimates VLA capability online from action uncertainty, and combines both signals in nonlinear authority arbitration.

    Source: §I · p. 2, D. Capability-Aware Arbitration · p. 3, B. VLM-Based Semantic Intent Inference… · p. 3

  • Results

    Sem-Cap achieved the highest task success across evaluated methods and the most favorable subjective ratings among assisted methods.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Authority allocation can account for both inferred intent and online execution uncertainty, addressing the gap left by intent-only arbitration.

    Source: §I · p. 1

    Evaluation was limited to a small VLA; larger models may produce different trajectory distributions and confidence behavior, potentially requiring recalibration of reference scales.

    Source: §VI · p. 8

Abstract

from arXiv · show

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping. We present a capability-aware shared-control framework in which a vision-language model (VLM) infers human intent and provides semantic-intent confidence, while a vision-language-action (VLA) policy generates autonomous actions. VLA capability confidence is estimated online from the dispersion and local instability of stochastic action trajectories. We design a nonlinear arbitration policy that combines Bayesian-filtered semantic-intent confidence with VLA capability confidence through a sigmoid mapping to adapt robot authority. Our evaluation combined VLM/VLA confidence assessment with a study involving 12 participants performing pick-and-place and bidirectional stacking under in-distribution and out-of-distribution conditions. The proposed method achieved the highest task success rate (92%), compared with manual teleoperation (83%), intent-only arbitration (44%), and fixed equal-weight blending (10%). It also achieved higher control friendliness and lower authority-weighted disagreement than both shared-control baselines. These results demonstrate the benefit of incorporating VLA capability into authority allocation to mitigate over-helping and improve shared-control performance.

I. INTRODUCTION

Intent-based arbitration can over-assist when confident intent is mistaken for evidence that the robot can execute reliably. Sem-Cap jointly accounts for intent and execution capability when allocating authority.

I. INTRODUCTION

  • Intent-confidence-driven arbitration assumes reliable autonomous execution, so high confidence can prompt undesirable assistance when robot capability varies across tasks and conditions.The introduction identifies this mismatch as a risk of over-helping.Source: §I · p. 1
  • Sem-Cap combines VLM semantic-intent confidence with online VLA capability confidence to allocate control authority.The framework calibrates and filters intent confidence, estimates capability from action-prediction uncertainty, and applies nonlinear arbitration.Source: §I · pp. 1–2

A. Intent-Based Shared Autonomy

  • Prior intent-based methods include goal-belief planning, learning from repeated interactions, and dynamic target identification, but intended task recognition does not establish execution reliability.Examples include BRACE [11], SARI, and VOSA.Source: A. Intent-Based Shared Autonomy · p. 2

B. VLM-Based Intent Inference in Shared Control

The paper positions semantic-intent inference and execution uncertainty as distinct inputs to shared-control arbitration. Its framework filters VLM intent predictions and uses the active instruction to condition VLA assistance.

B. VLM-Based Intent Inference in Shared Control

  • VLM top-K intent predictions can fluctuate, and prediction accuracy alone does not establish confidence calibration.The stated gap is how to use VLM semantic intent to guide action generation while continuously regulating authority.Source: B. VLM-Based Intent Inference in… · p. 2

C. Uncertainty in Autonomy

  • Prior uncertainty methods do not jointly use semantic-intent confidence and online VLA execution uncertainty to continuously allocate control authority.Sem-Cap combines intent confidence with a capability signal derived from VLA execution uncertainty.Source: C. Uncertainty in Autonomy · p. 2

A. Framework Overview

  • The VLM infers semantic intent and confidence, the instruction conditions the VLA command, and arbitration blends that command with human input.The framework uses intent confidence α and capability confidence β to determine robot authority.Source: A. Framework Overview · p. 2

B. VLM-Based Semantic Intent Inference and Confidence

  • The VLM evaluates supported complete-manipulation instructions, calibrates their scores, and Bayesian filtering combines current predictions with prior intent beliefs.A new instruction is accepted when its posterior is high enough and remains stable across consecutive updates; semantic-intent confidence is the filtered probability of the active instruction.Source: B. VLM-Based Semantic Intent Inference… · pp. 2–3
  • Filtered predictions switch 2 times, compared with 24 switches for unfiltered predictions.The figure also maps trajectory dispersion and local instability to capability confidence.Source: Figure 2 · p. 3

C. VLA Capability Confidence

The VLA capability signal reflects consistency and local stability across stochastic action trajectories. The arbitration mapping uses that signal together with semantic-intent confidence to set robot authority.

C. VLA Capability Confidence

  • Capability estimation draws stochastic action sequences under the active instruction and measures prediction consistency as mean dispersion across sampled trajectories.Each sampled sequence is mapped through forward kinematics to an end-effector trajectory.Source: C. VLA Capability Confidence · p. 3
  • Local trajectory instability is measured using the second waypoint difference, complementing trajectory dispersion in the capability assessment.The two measures are normalized by reference scales from held-out in-distribution data.Source: C. VLA Capability Confidence · p. 3
  • A high βt indicates consistent and locally stable VLA prediction; the sigmoid arbitration mapping combines capability and intent confidence to determine robot authority.The mapping uses wR = σ(κ(αβ − τ)).Source: C. VLA Capability Confidence · p. 3, Figure 3 · p. 3

D. Capability-Aware Arbitration

Robot authority is allocated using semantic-intent confidence, VLA capability confidence, and human–VLA command disagreement. A sigmoid mapping provides a smooth transition and blends the human and VLA commands.

D. Capability-Aware Arbitration

  • Robot authority depends on VLM semantic-intent confidence, VLA capability confidence, and disagreement between command increments relative to the previous blended command.Disagreement is computed with cosine dissimilarity.Source: D. Capability-Aware Arbitration · p. 3
  • A sigmoid mapping combines the confidence signals to provide a smooth authority transition, mitigating multiplicative attenuation in their product.Its threshold and slope are tunable.Source: D. Capability-Aware Arbitration · p. 3
  • The resulting robot authority weight ranges from 0 to 1 and blends the human and VLA commands.The logistic sigmoid uses a confidence midpoint τ and transition-slope parameter κ.Source: D. Capability-Aware Arbitration · p. 3

IV. EXPERIMENTS

The experiments used a six-joint arm and evaluated pick-and-place and stacking instructions with cubes and LEGOs under in-distribution and out-of-distribution conditions.

A. Experimental Setup and Tasks

  • Experiments used a six-joint LeRobot SO-101 arm, two Nintendo Switch Joy-Cons, and top- and wrist-view RGB cameras.Fig. 4 presents the human-study task design and experimental setup.Source: A. Experimental Setup and Tasks · p. 3, Figure 4 · p. 4
  • Five instructions covered three pick-and-place tasks and two stacking directions, evaluated with cubes and LEGOs in ID and OOD conditions.The listed task variants include placing colored cubes or LEGOs in bins and stacking cubes or LEGOs by color.Source: A. Experimental Setup and Tasks · pp. 3–4
  • Table I identifies the control methods compared in the human study.The supplied caption gives the table’s scope but not its entries.Source: Table I · p. 4

B. Model Training and Confidence Calibration

The study trained and calibrated VLM and VLA models, compared three arbitration baselines, and assessed intent-confidence reliability alongside a counterbalanced human study.

B. Model Training and Confidence Calibration

  • Moondream3 was adapted with rank-8 LoRA on 400 ID demonstrations, temperature-scaled on validation data, while SmolVLA was fine-tuned on the same dataset.Capability estimation sampled four stochastic trajectories of 50 waypoints, asynchronously with at least 0.3 seconds between submissions.Source: B. Model Training and Confidence… · p. 4

C. Baseline Arbitration Methods

  • The study compared manual teleoperation, fixed 50/50 blending, and semantic-intent-confidence-only arbitration.These baselines represent no assistance, constant assistance, and adaptive assistance, respectively.Source: C. Baseline Arbitration Methods · p. 4

D. Human Study Protocol

  • Twelve participants completed 192 formal trials, equally divided between ID and OOD conditions, with trials stopped at 120 seconds recorded as failures.Each participant performed 16 trials after a tutorial and practice session.Source: D. Human Study Protocol · p. 4
  • Manual teleoperation was always presented first, while assisted-method order, task order, and object layouts were counterbalanced.The three assisted methods followed all six possible orders, each used twice across participants.Source: D. Human Study Protocol · p. 4

1) VLM intent inference confidence evaluation:

  • VLM confidence reliability was assessed with reliability diagrams and ECE, alongside intent accuracy, wrong-switch rate, and robot authority.These measures examine calibration, whether filtering compromises intent correctness, erroneous intent changes, and control allocation.Source: 1) VLM intent inference confidence… · p. 4

2) VLA confidence evaluation:

The evaluation compared VLA capability confidence across training and distribution conditions and measured task performance and control quality using complementary metrics.

2) VLA confidence evaluation:

  • Pretrained and fine-tuned VLA models were compared on mean and progress-dependent capability confidence across ID and OOD pick-and-place and stacking tasks.The comparison used the same recorded observations and fixed reference scales.Source: 2) VLA confidence evaluation · p. 4

3) Framework objective measures:

  • Task performance was measured by success rate and completion time, while control quality included trajectory smoothness, control friendliness, and authority-weighted disagreement.Fig. 5 illustrates confidence and robot authority across normalized task progress for ID/OOD pick-and-place and ID stacking examples.Source: 3) Framework objective measures · p. 4, Figure 5 · p. 5
  • Trajectory smoothness was quantified by root-mean-square jerk of the end-effector trajectory, excluding stagnant-motion timepoints; lower JRMS indicates smoother motion.Control friendliness measures directional alignment between blended and human commands during active human input; higher values indicate more consistent support rather than opposition.Source: 3) Framework objective measures · pp. 4–5
  • Authority-weighted disagreement quantifies retained autonomous authority during human–VLA disagreement; lower CAWD indicates less authority is retained in those cases.Source: 3) Framework objective measures · p. 5

4) Framework subjective measures:

The study measured participants’ views of shared-control methods with a post-method questionnaire and analyzed questionnaire scores separately from objective control measures.

4) Framework subjective measures:

  • Q1–Q8 were adapted from a shared-control questionnaire derived from the USE usability questionnaire.The source questionnaire was used by Bowman et al. and derived from the USE questionnaire.Source: 4) Framework subjective measures · p. 5
  • Q9–Q10 were introduced to measure perceived understanding of the target object and intended task.These items asked whether the assistance understood which object the participant wanted to manipulate and which task they wanted to perform.Source: 4) Framework subjective measures · p. 5

5) Statistical analysis:

  • Questionnaire scores were compared with paired Wilcoxon signed-rank tests, while objective measures used ANOVA models with Method and Domain as factors.Post hoc comparisons used Holm-corrected paired t-tests; significance was set at p < .05.Source: 5) Statistical analysis · p. 5

A. Illustrative Confidence Dynamics

The confidence analyses show how calibrated and filtered intent estimates and task-dependent VLA capability estimates shape robot authority. Capability confidence varied with fine-tuning, task, domain, and manipulation phase.

A. Illustrative Confidence Dynamics

  • When intent confidence remained high, low capability confidence constrained authority; rising or falling β produced corresponding authority changes across task phases.The examples include β rising during transport and placement and declining after stacking placement and release.Source: A. Illustrative Confidence Dynamics · p. 5

B. VLM Intent Inference Confidence Evaluation

  • ECE fell from 39.67% to 7.62% after calibration, while intent accuracy remained 79.19%.With δt = 0, calibrated confidence also yielded greater offline-computed robot authority despite unchanged top-1 predictions.Source: B. VLM Intent Inference Confidence… · p. 5, Figure 6 · p. 6
  • Bayesian filtering reduced the wrong-switch rate from 3.84% to 1.05%, a 72.7% relative decrease, while largely preserving accuracy at 78.97%.The largest absolute reduction occurred at 10–15% video progress, from 8.30% to 1.81%; suppressing erroneous transitions can limit abrupt redirection of VLA actions.Source: B. VLM Intent Inference Confidence… · p. 5, Figure 6 · p. 6

C. VLA Confidence Evaluation

  • β generally increased with fine-tuning in both ID and OOD conditions, but remained lower for stacking than pick-and-place and was consistently higher in ID than OOD.At 10k steps, stacking β remained close to the pretrained baseline; the OOD LEGO pick-and-place example lacked a similarly sustained increase during placement.Source: C. VLA Confidence Evaluation · pp. 5–6

1) Task performance:

Sem-Cap improved task success over confidence-independent assistance and semantic-intent gating, with control-quality results also favoring it over the shared-control baselines on friendliness and authority-weighted disagreement.

1) Task performance:

  • Task success improved progressively from confidence-independent assistance to semantic-intent gating and then to Sem-Cap, with the improvement particularly evident under OOD conditions.Manual teleoperation had higher success in ID than OOD, where LEGO objects were harder to grasp and stack within 120 seconds.Source: 1) Task performance · p. 6
  • Sem-Cap had the lowest mean successful completion time, but the overall Method effect was not significant (F(2, 90) = 1.81, p = .170).The Semantic Intent–Sem-Cap comparison was significant (pHolm = .038), while the Domain effect did not reach significance (F(1, 90) = 3.76, p = .056).Source: 1) Task performance · p. 6

2) Control quality:

  • Sem-Cap’s mean JRMS was 8.68 m s−3, versus 12.13 m s−3 for Semantic Intent and 9.47 m s−3 for Fixed 50/50; Manual was smoothest at 6.66 m s−3.Sem-Cap was smoother than both shared-control baselines, but less smooth than Manual.Source: 2) Control quality · p. 6, Figure 8 · p. 7
  • Sem-Cap had the highest mean friendliness (F = 0.714) and lowest mean CAWD (0.035) among the assisted methods, outperforming both shared-control baselines on each measure.The CAWD reductions from Fixed 50/50 to Semantic Intent and from Semantic Intent to Sem-Cap were significant; no Method-by-Domain interaction was significant for the reported control-quality measures.Source: 2) Control quality · pp. 6–7, Figure 8 · p. 7

E. Framework Subjective Measures

Sem-Cap received more favorable ratings than both assisted baselines, though several ratings remained below manual teleoperation. The discussion highlights overall task success alongside limits involving user preferences, intent filtering, task duration, and model size.

E. Framework Subjective Measures

  • 3.66 was Sem-Cap’s mean favorable score for Q1–Q8, versus 1.95 for Semantic Intent and 1.23 for Fixed 50/50; its difference from Manual (3.96) was not significant.Sem-Cap rated significantly higher than both assisted baselines (W = 0, pHolm = .00293 for both), while its difference from Manual was not significant (W = 12, pHolm = .23438).Source: E. Framework Subjective Measures · p. 7
  • 3.00 was Sem-Cap’s combined mean for Q9–Q10, versus 1.67 for Semantic Intent and 1.17 for Fixed 50/50.Sem-Cap received higher target- and task-understanding ratings than both assisted baselines; participants perceived its assistance as more understandable and reliable.Source: E. Framework Subjective Measures · p. 7

VI. DISCUSSION

  • The proposed method achieved the highest task success rate across all evaluated methods, including manual teleoperation.It also received the most favorable subjective ratings among assisted methods, with advantages in perceived target and task understanding.Source: §VI · p. 7
  • Manual teleoperation received higher favorable ratings than Sem-Cap on several items, possibly because the VLA does not adapt to individual motion preferences.Even when VLA-generated and human-preferred actions both reach the goal, their movement strategies may differ.Source: §VI · p. 7, E. Framework Subjective Measures · p. 7
  • Bayesian filtering improves temporal stability but increases ECE and can delay responses to genuine intent changes.The discussion identifies a trade-off among stability, confidence calibration, and responsiveness.Source: §VI · pp. 7–8
  • Whether capability confidence β remains informative in longer-horizon tasks with more diverse or repeated phases remains unclear.The authors propose relating β variations to execution outcomes within individual phases.Source: §VI · p. 8
  • Computational constraints limited the study to a small VLA; larger models may yield different trajectory distributions and require recalibrated reference scales.The smaller increases in β from 20k to 60k steps suggest diminishing confidence gains, but confidence trends alone cannot establish a performance ceiling.Source: §VI · p. 8

Related papers

Loading 2609.25369v1…