PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners

Zhi Zheng, Qian Feng, Hang Li, Alois Knoll, Jianxiang Feng

arXiv:2406.00430v2cs.ROcs.AI

TL;DR

Open-loop LLM planners lack environmental feedback, while closed-loop failure detection may assume model predictions are trustworthy despite hallucinations. KnowLoop integrates uncertainty-based MLLM failure detection into LLM planning; token probability and entropy reflect predictive quality better than self-explained confidence, and filtering uncertain predictions improves task success rates.

  • Problem

    Open-loop LLM planners follow initial plans without environmental feedback, while some closed-loop detectors rely on task-specific heuristics or assume predictions are always trustworthy despite hallucinations.

    Source: §I · p. 1

  • Method

    KnowLoop integrates an uncertainty-based MLLM failure detector into a zero-shot LLM planner, filtering uncertain predictions and enabling requests for human help.

    Source: §I · pp. 1–2

  • Results

    Token probability and entropy reflect LLaVA’s predictive quality, whereas self-explained uncertainty has weak correlation with accuracy; KnowLoop improves task success rate by around 50% compared to ChatGPT-4V and around 40% compared to LLaVA.

    Source: 1) Uncertainty Estimation for MLLM… · p. 5, §V · p. 6, 2) Uncertainty for Closed-Loop LLM… · p. 6

  • Takeaways & Limitations

    Filtering uncertain predictions and seeking human help can enhance failure-detection accuracy and improve closed-loop planning task success.

    Source: 2) Uncertainty for Closed-Loop LLM… · p. 6, §V · p. 6

Abstract

from arXiv · show

Recently, Large Language Models (LLMs) have witnessed remarkable performance as zero-shot task planners for robotic manipulation tasks. However, the open-loop nature of previous works makes LLM-based planning error-prone and fragile. On the other hand, failure detection approaches for closed-loop planning are often limited by task-specific heuristics or following an unrealistic assumption that the prediction is trustworthy all the time. As a general-purpose reasoning machine, LLMs or Multimodal Large Language Models (MLLMs) are promising for detecting failures. However, However, the appropriateness of the aforementioned assumption diminishes due to the notorious hullucination problem. In this work, we attempt to mitigate these issues by introducing a framework for closed-loop LLM-based planning called KnowLoop, backed by an uncertainty-based MLLMs failure detector, which is agnostic to any used MLLMs or LLMs. Specifically, we evaluate three different ways for quantifying the uncertainty of MLLMs, namely token probability, entropy, and self-explained confidence as primary metrics based on three carefully designed representative prompting strategies. With a self-collected dataset including various manipulation tasks and an LLM-based robot system, our experiments demonstrate that token probability and entropy are more reflective compared to self-explained confidence. By setting an appropriate threshold to filter out uncertain predictions and seek human help actively, the accuracy of failure detection can be significantly enhanced. This improvement boosts the effectiveness of closed-loop planning and the overall success rate of tasks.

I. INTRODUCTION

LLM task planners can fail because they execute open-loop plans without environmental feedback, while closed-loop failure detectors may rely on task-specific heuristics or trust model predictions despite hallucinations. KnowLoop uses uncertainty-based failure detection to filter uncertain predictions and request human help, and is evaluated with manipulation tasks and real hardware.

  • Open-loop LLM planners rigidly follow initial plans without environmental feedback, leaving execution errors or plan deficiencies unaddressed and potentially causing task failure.Existing closed-loop detectors also tend to use task-specific heuristics or assume model predictions are always trustworthy, despite LLM/MLLM hallucinations.Source: §I · p. 1
  • KnowLoop uses an uncertainty-based, model-agnostic failure detector to filter uncertain predictions and solicit human help when confidence is low.The framework integrates the detector into a zero-shot LLM-based robot system for closed-loop planning.Source: §I · pp. 1–2
  • The detector estimates MLLM uncertainty using token probability, entropy, or self-explained confidence across three prompting strategies.The strategies include direct prompts asking about success or failure and indirect prompts without that question.Source: §I · p. 2
  • The evaluation uses a self-collected dataset spanning five manipulation tasks and experiments with an LLM-based robot system on both the dataset and real hardware.The experiments use LLaVA and ChatGPT-4V; the authors report promising results for uncertainty estimates in closed-loop planning.Source: §I · p. 2

II. RELATED WORK

Uncertainty estimation has been applied across robotics, including perception and control, and has gained attention for trustworthy robotics using deep learning and foundational models.

1) Uncertainty Estimation in Robotics:

  • Uncertainty estimation in robotics spans learning-based perception and control, with relevance growing as foundational models become more prominent.The passage traces this tradition to probabilistic robotics and robotic introspection.Source: 1) Uncertainty Estimation in Robotics · p. 2
  • KnowNo estimates uncertainty for an LLM planner by generating next-step options and analyzing token probabilities with conformal prediction.The cited work does not consider task-execution failure itself.Source: 1) Uncertainty Estimation in Robotics · p. 2
  • The passage frames uncertainty estimation as a way to support trustworthy learning-enabled robotics amid neural networks’ black-box nature.Source: 1) Uncertainty Estimation in Robotics · p. 2

2) Uncertainty Estimation in LLMs:

Prior work relates LLM uncertainty to hallucinations, while closed-loop robotic planning has used environmental feedback and dedicated failure detectors. KnowLoop integrates uncertainty derived from an MLLM into an LLM planner and requests human feedback for highly uncertain responses.

2) Uncertainty Estimation in LLMs:

  • Prior LLM uncertainty work includes prompted confidence reports, consistency-based estimates, token-level self-evaluation, and confidence elicitation, primarily for general question answering rather than robotics.The passage also relates model uncertainty to hallucination occurrence and describes zero-resource hallucination prediction as avoiding external knowledge resources.Source: 2) Uncertainty Estimation in LLMs · p. 2

3) Closed-loop planning with LLMs:

  • Related closed-loop planners use environmental feedback for self-reflection and correction, while one real-world approach relies on hand-crafted external failure detectors.The passage contrasts that approach with estimating uncertainty from an MLLM for more adaptable failure detection.Source: 3) Closed-loop planning with LLMs · p. 2

III. THE PROPOSED CLOSED-LOOP PLANNER

  • KnowLoop decomposes language instructions into subgoals, then activates an MLLM failure detector after each subgoal using environment images to assess achievement.The implementation uses LLaVA and ChatGPT-4V for detection with a ChatGPT-4-based planner; merging planner and detector models is left for future work.Source: §III · p. 2

A. Uncertainty-based Failure Detection

  • The detector estimates uncertainty for prompted responses and filters those above a predetermined threshold, requesting human feedback to assess the current state.Its failure-detection approach is described as agnostic to the planner’s LLM, unlike task-specific or heuristic approaches.Source: A. Uncertainty-based Failure Detection · pp. 2–3

1) Prompting Strategies:

The prompting strategies define failure detection through either direct success assessment or indirect next-action selection, using expected states and environmental images as context.

1) Prompting Strategies:

  • Subgoal State Comparison (SSC) asks the MLLM to compare an action’s expected state description with the current environmental image to judge subgoal success.Source: 1) Prompting Strategies · p. 3
  • Spatial Relationship Analysis (SRA) uses intermediate reasoning to compare current and expected states, including changes in gripper-object interaction and spatial arrangement.Source: 1) Prompting Strategies · p. 3
  • Next Action Prediction (NAC) presents plans, executable actions, and current-environment images; selecting the predefined plan’s next action counts as success, while another choice counts as failure.Source: 1) Prompting Strategies · p. 3

2) Uncertainty Quantification:

The section presents three prompting strategies for MLLM-based failure detection and describes token probability, entropy, and self-explained confidence as uncertainty measures. Because detector errors can disrupt closed-loop operation, it motivates gauging response confidence.

2) Uncertainty Quantification:

  • MLLM hallucinations make confidence important to gauge: missed errors can lead to task failure, while false negatives can disrupt operations.The section says uncertain responses should be met with “I don’t know” rather than a false answer.Source: 2) Uncertainty Quantification · p. 3
  • SSC asks whether the expected state is satisfied based on the image, while SRA first asks for spatial-relation analysis before that judgment.NAC instead asks which listed subtask should come next, using the expected state, plan, and image.Source: 2) Uncertainty Quantification · pp. 3–4
  • Token probability treats the selected option’s token probability as the likelihood that the affirmative or negative response is correct.The passage describes deriving it from multiple-choice questions and responses.Source: 2) Uncertainty Quantification · p. 4
  • Entropy uses output-token probabilities to estimate uncertainty about the MLLM’s response.The passage describes entropy as a measure of uncertainty or disorder.Source: 2) Uncertainty Quantification · p. 4
  • Self-explained confidence prompts the MLLM to state a confidence percentage, but may not produce precise estimates.Its reliability depends on meticulous prompt engineering and the model’s own confidence-related processes.Source: 2) Uncertainty Quantification · p. 4

B. Closed-loop Planning

The closed-loop framework plans and executes subtasks one by one, using a detector to assess execution and guide whether to continue or retry.

B. Closed-loop Planning

  • ChatGPT-4 decomposes instructions into subtasks, and GPT code generation with the robot-arm motion API produces Python code for each action.The arm executes the resulting task plan action by action.Source: B. Closed-loop Planning · p. 4
  • After each action, the detector assesses the environment; success advances to the next planned action, while failure stops the current action and re-executes all actions.The framework description says the detector is called after action execution to assess the environment and detect failure.Source: B. Closed-loop Planning · p. 4
  • Failure detection takes the current subtask and state image, and asks the user for help when the assessment is uncertain.Its output is whether the subtask was successfully executed.Source: B. Closed-loop Planning · p. 4
  • The planning loop tracks the subtask index and retry count, continuing while subtasks remain and retries stay below the maximum k.The loop initializes the index and retry count to zero.Source: B. Closed-loop Planning · p. 4

IV. EXPERIMENT

The experiments assess task completion under interference and evaluate uncertainty methods for selecting a combination for real-hardware closed-loop planning.

IV. EXPERIMENT

  • The experiments evaluate three uncertainty-estimation methods to select a best-performing combination for real-hardware closed-loop LLM planning.The test set includes slightly different tasks with similar items.Source: §IV · p. 4
  • The listed test tasks are picking up a sponge and placing it on a notebook, placing a smartphone in an upper drawer, and opening that drawer.These task examples are given in the experiment’s task footnote.Source: §IV · p. 4
  • 70–80% task success without interference fell to 0% under human interference with open-loop control.Interference included displacing objects from intended locations or impeding operations.Source: §IV · p. 5

A. Dataset

The dataset samples execution states from five manipulation tasks and focuses its failure examples on detection and operational errors.

A. Dataset

  • The collected dataset contains 142 samples representing different execution states across five tasks.Source: A. Dataset · p. 5
  • The primary failures are detection and operational errors; planning errors are explicitly excluded.Source: A. Dataset · p. 5
  • The tasks primarily involve grasping, placing, and pushing actions.Source: A. Dataset · p. 5

B. Evaluation Metrics

The section defines metrics for evaluating uncertainty, failure detection, human assistance, and confidence-score generation, alongside a self-collected dataset breakdown.

B. Evaluation Metrics

  • Calibration-AUC tracks accuracy as uncertain samples are excluded by threshold, while Selective-AUC tracks accuracy as the highest-uncertainty fraction is abstained from.Both curves assess whether selective abstention improves retained-sample accuracy.Source: B. Evaluation Metrics · p. 5
  • Detection Accuracy measures correct predictions among detector decisions, Success Rate measures successful experiments, and Human Involve Rate and Generation Rate capture assistance and confidence-generation frequency.Human assistance is excluded from Detection Accuracy; Generation Rate applies only to self-explained confidence.Source: B. Evaluation Metrics · p. 5

C. System Setup

  • The dataset breakdown covers five manipulation tasks, including short-term and long-term complex actions.Listed tasks include picking up and placing objects, opening a drawer, and closing it.Source: Table I · p. 5, B. Evaluation Metrics · p. 5

D. Results

KnowLoop uses uncertainty to filter failure-detector predictions and seek human help when predictions are uncertain. The experiments report improved task success and show that token probability and entropy better reflect predictive quality than self-explained uncertainty.

2) Uncertainty for Closed-Loop LLM Planners:

1) Uncertainty Estimation for MLLM Failure Detecor:

V. CONCLUSION

The conclusion finds that token probability and entropy reflect LLaVA predictive quality and applies uncertainty thresholds in KnowLoop to identify lower-quality responses. It points to more resource-efficient planning and stronger thresholding and failure reasoning as future directions.

V. CONCLUSION

  • Token probability and entropy reflect LLaVA’s predictive quality, and thresholding identifies lower-quality responses in KnowLoop’s uncertainty-based failure detector.The authors state that these improvements make the framework suitable for failure detection in closed-loop control systems.Source: §V · p. 6
  • Future work includes using an MLLM instead of the LLM in the task planner, applying conformal prediction to set thresholds, and investigating uncertainty for failure reasoning and correction.Source: §V · p. 6
Loading 2406.00430v2…