Source-linked AI summary
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, Anirudha Majumdar
TL;DR
LLM planners can confidently hallucinate, creating a need for robots that know when they do not know and seek clarification. KnowNo combines multiple-choice prompting with conformal prediction to calibrate prediction sets and request help, achieving user-specified task completion while reducing human help by 10−24% versus baselines lacking formal assurances. Its guarantees depend on grounded environments, successful proposed actions, and faithful human assistance.
Problem
LLM planners can confidently generate plausible but incorrect predictions, while prior clarification approaches do not ensure a desired level of task success.
Method
KnowNo reformulates LLM planning as multiple-choice prediction and applies conformal prediction to calibrated candidate sets, requesting human help when the set is non-singleton.
Results
10−24% less human help than baseline approaches lacking formal assurances, while consistently achieving user-specified task completion levels across simulated and hardware setups.
Takeaways & Limitations
KnowNo provides a lightweight way to align LLM-planner uncertainty with statistical task-completion guarantees and minimized human assistance.
Takeaways & Limitations
The task-completion guarantee assumes objects are fully grounded in the LLM input, proposed actions execute successfully, and humans provide help faithfully when requested.
Abstract
from arXiv · showhide
Large language models (LLMs) exhibit a wide range of promising capabilities -- from step-by-step planning to commonsense reasoning -- that may provide utility for robots, but remain prone to confidently hallucinated predictions. In this work, we present KnowNo, which is a framework for measuring and aligning the uncertainty of LLM-based planners such that they know when they don't know and ask for help when needed. KnowNo builds on the theory of conformal prediction to provide statistical guarantees on task completion while minimizing human help in complex multi-step planning settings. Experiments across a variety of simulated and real robot setups that involve tasks with different modes of ambiguity (e.g., from spatial to numeric uncertainties, from human preferences to Winograd schemas) show that KnowNo performs favorably over modern baselines (which may involve ensembles or extensive prompt tuning) in terms of improving efficiency and autonomy, while providing formal assurances. KnowNo can be used with LLMs out of the box without model-finetuning, and suggests a promising lightweight approach to modeling uncertainty that can complement and scale with the growing capabilities of foundation models. Website: https://robot-help.github.io
1 Introduction
Language-instructed robots need calibrated uncertainty because LLM planners can confidently produce plausible but incorrect actions in ambiguous environments. KnowNo formalizes uncertainty alignment as achieving user-specified task success while minimizing human help, using conformal prediction to decide when clarification is needed.
- LLM planners generate temporally extended plans from natural-language instructions but may confidently hallucinate plausible, incorrect outputs.
- Ambiguous instructions can make acting directly unsafe, so robots should ask humans for clarification when multiple interpretations remain.The paper illustrates this with choosing between plastic and metal bowls for microwave use.
- KnowNo defines uncertainty alignment through calibrated confidence and minimal help: statistically guaranteed task success with as little assistance as possible.
- KnowNo uses conformal prediction to select candidate actions, executing a singleton prediction set and requesting help for non-singleton sets.
2 Overview: Robots that Ask for Help
KnowNo converts LLM planning into multiple-choice prediction, uses conformal prediction to form calibrated candidate sets, and asks for help when uncertainty remains. This design addresses variable-length scoring bias and supports uncertainty alignment in partially observable, potentially ambiguous robot tasks.
- Directly scoring variable-length language plans is unreliable because plan probabilities are sensitive to sequence length and can bias uncertainty estimates.
- Planning as multiple-choice Q&A: KnowNo reformulates next-step planning as multiple-choice Q&A by generating semantically distinct candidate plans and reducing selection to one next-token prediction.
- Planning as multiple-choice Q&A: Conformal prediction uses held-out calibration examples and LLM confidence scores to choose a reduced candidate set whose singleton status determines whether the robot acts or requests help.
- Robots that ask for help: The policy pipeline generates labeled candidate plans from observations, instructions, and few-shot examples, then applies conformal prediction before routing non-singleton cases to a human.
- Goal: uncertainty alignment: Uncertainty alignment targets a user-specified success probability and minimizes the average number of options presented to a human across scenarios.
3 Calibrating LLM Confidence with Conformal Prediction
KnowNo applies conformal prediction to LLM-generated candidate actions, producing calibrated prediction sets that determine when robots act or seek human help. The framework extends these guarantees from single-step to dependent multi-step planning through sequence-level calibration and causal reconstruction.
- Background: Conformal Prediction: Conformal prediction uses calibrated nonconformity scores and a quantile to construct prediction sets containing the true label with user-specified coverage 1−ϵ.The set includes labels whose LLM confidence is at least 1−q̂, and the coverage guarantee holds for unseen data.
- Background: Conformal Prediction: Dataset-conditional calibration supports desired 1−ϵ coverage for new test data without recalibrating after every calibration sample.The implementation uses N = 400 and δ = 0.01, providing probability 1−δ=0.99 over the calibration set for the adjusted coverage target.
- Single-Step Uncertainty Alignment: In single-step planning, the robot executes a singleton prediction set and requests human help when multiple candidate actions remain.The human selects the correct candidate when it is present; otherwise, the operation is halted or corrected according to the described setup.
- Single-Step Uncertainty Alignment: With probability 1−δ, single-step task completion is at least 1−ϵ, while average prediction-set size is minimized when the confidence model represents true conditional probabilities.The guarantee assumes accurate human help when the correct label is included in the prediction set.
- Multi-Step Uncertainty Alignment: Multi-step planning violates the original i.i.d. assumption because intermediate contexts depend on earlier contexts and actions, motivating a new CP extension.The extension lifts data to sequences and calibrates using a sequence-level nonconformity score.
- Multi-Step Uncertainty Alignment: Sequence-level prediction sets are causally reconstructed one timestep at a time using current and past information, preserving the calibrated quantile.The construction targets the sequence-level coverage bound P(y_test∈C(x_test))≥1−ϵ.
- Multi-Step Uncertainty Alignment: With probability 1−δ, multi-step task completion is at least 1−ϵ, and average prediction-set size is minimized when the confidence model matches true conditional probabilities.The result applies when CP constructs prediction sets causally and help is requested whenever a timestep’s set is not a singleton.
4 Experiments
Experiments evaluate KNOWNO across simulated and hardware robot tasks with diverse ambiguities, comparing calibrated uncertainty against prediction-set and prompting baselines. KNOWNO consistently targets specified success rates while reducing human help and prediction-set size.
- Experimental settings: Experiments span simulated tabletop rearrangement, hardware multi-step rearrangement, bimanual manipulation, and mobile manipulation with language ambiguities.Settings include attribute, numeric, spatial, preference, Winograd-schema, unsafe-action, and arm-choice ambiguities.
- Simulation results: KNOWNO achieves the target task success rate consistently, while Simple Set and Ensemble Set cannot maintain coverage across simulation settings.Prompt Set, Binary, and No Help show larger deviations because users cannot control their error rates.
- Simulation results: KNOWNO achieves higher success-to-help ratios than Simple Set across target error rates in simulation.Lower ϵ improves empirical success for KNOWNO, Simple Set, and Ensemble Set but requires more human help; KNOWNO reduces help by up to 24% at certain success levels.
- Hardware results: KNOWNO reduces step-wise human help by 14% and trial-wise help by 8% in hardware multi-step rearrangement.It also reduces average prediction-set size; Simple Set requires 87% step-wise help when assigned the same ϵ.
- Hardware results: KNOWNO reduces hardware human help by 14% and average prediction-set size while matching Simple Set’s plan success rate in mobile manipulation.The mobile-manipulation scenarios include Winograd-style and unsafe-action ambiguities.
- Robustness: The target success guarantee remains effective across LLM choices, including PaLM-2L-IF and GPT-3.5.For GPT-3.5, conformal coverage compensates for degraded performance by triggering more human intervention.
5 Related Work
Related work covers LLM planning, uncertainty quantification, conformal prediction in robotics, and dialogue-based interaction. KNOWNO distinguishes itself through multi-step sequence-level calibration for language-based planning.
- LLMs for robot planning and interaction: LLM planners support reasoning, high-level planning, language feedback, and robot-code generation, but remain prone to confidently hallucinating outputs.Examples include referring to unobserved objects or calling incorrect actions.
- Uncertainty quantification for LLMs: Uncertainty-quantification research increasingly calibrates LLM confidence, while conformal methods provide coverage guarantees for set-valued predictors.Few prior works had applied conformal prediction to quantifying uncertainty in LLMs.
- Conformal prediction in robotics: Prior robotic conformal-prediction methods provide average or conservative stepwise guarantees, whereas KNOWNO calibrates sequences across the entire planning horizon.The multi-step extension handles actions influencing future contexts.
- Human-robot dialogue and interaction: KNOWNO uses a simple dialogue setup in which humans clarify potentially ambiguous instructions when the robot requests help.It does not address human-provided observation information or possible human error.
6 Discussion
KNOWNO applies conformal prediction to align uncertainty in language-instructed robots, achieving user-specified task completion levels while reducing human help. Its guarantee assumes text-grounded environments and successful execution of proposed actions.
- KNOWNO applies conformal prediction to provide statistical guarantees of task completion while minimizing human help.The framework formalizes uncertainty alignment as both sufficiency and minimality.
- 10−24% less human help was required than with baseline approaches lacking formal assurances.This result was observed across simulated and hardware setups.
- The task-completion guarantee assumes objects are fully grounded in the LLM text input and proposed actions execute successfully.Future work considers incorporating perception and low-level action-policy uncertainty into calibration.
A1 Evaluating Semantic Uncertainty of the LLM with MCQA
The paper uses MCQA to represent semantically distinct plans and applies conformal prediction to select uncertainty-aligned action sets. This supports guaranteed task completion while requesting help when ambiguity remains, across multi-label and multi-step settings.
- MCQA setup: MCQA prompts the LLM to generate likely, semantically different candidate plans, reducing next-step prediction to a single choice among options.This avoids directly scoring variable-length natural-language plans, whose probabilities are sensitive to sequence length.
- MCQA setup: The planner constructs candidate labels from generated plans, then uses conformal prediction on uncalibrated confidence scores to form a prediction set.A singleton set is executed; otherwise, the robot asks for human help.
- Theoretical guarantees: Proposition 3 guarantees at least 1−ϵ task completion for multi-label actions with probability 1−δ over calibration-set sampling.The guarantee applies when help is requested whenever the prediction set is not a singleton.
- Theoretical guarantees: The method selects the highest-confidence true label when reducing multiple acceptable labels to a single branch for conformal calibration.This induces a transformed distribution on which standard conformalization can be applied.
- Theoretical guarantees: Proposition 4 extends the same at-least-1−ϵ completion guarantee to multi-step tasks with multiple acceptable labels at each step.The construction causally handles multiple true labels across successive timesteps.
- Empirical findings: In experiments, sentence-level scores performed worse in all three settings, while MCQA-based confidence showed no significant RawProb-versus-Perplexity difference.The degradation of sentence-level scoring correlated with variance in multiple-choice lengths.