Source-linked AI summary
Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language
Zachary Ravichandran, Jonathan Diller, Fernando Cladera, Varun Murali, George J. Pappas, Vijay Kumar
TL;DR
Robots facing underspecified tasks in unfamiliar environments must infer what counts as success, which information matters, and where it can be found. CLUE grounds LLM-generated hypotheses in an online language-embedded map and refines plans through closed-loop interaction. Across real-world tasks, it rivals oracle performance and outperforms planners without feedback and map-then-query approaches.
Problem
Method
CLUE uses an LLM-enabled policy to generate hypotheses, grounds them in an online language-embedded map, and updates plans through active interaction and observations.
Source: §1 · p. 3, §4.1 · p. 7, §7 · p. 14Results
Across real-world tasks, CLUE rivals oracle performance (within 7 percentage points in task success) and outperforms no-feedback planning 4x and map-then-query 3x.
Source: §1 · p. 3, §7 · p. 15Takeaways & Limitations
Map construction and querying alone are insufficient for these complex contextual planning tasks; closed-loop interaction is needed to resolve them.
Source: §1 · p. 3The language-embedded map is memory-intensive, with outdoor maps reaching 100 GB.
Source: §6 · p. 14
Abstract
from arXiv · showhide
Foundation models provide robots with the ability to interpret natural language and reason about environmental context, yet most language-conditioned policies assume that goals are well-specified and that task-relevant information is provided upfront via a prior map. Operating in unfamiliar environments with underspecified tasks entails high contextual uncertainty: the robot must jointly infer what constitutes task success, what constitutes relevant information, and where (or whether) that information exists. We address these limitations via CLUE (Closed-Loop contextual Uncertainty rEsolution), a framework for actively resolving contextual uncertainty given underspecified tasks in natural language. CLUE uses an LLM-derived policy to hypothesize task-relevant concepts and potential plans. It then uses a language-embedded map, which is constructed online, to ground these hypotheses into actions. The policy sequentially evaluates hypotheses via closed-loop environment interaction and refines its plans as it gathers new information. We deploy CLUE on a Boston Dynamics Spot across three real indoor and outdoor environments spanning 15 tasks that require object disambiguation, functional inference, and occlusion reasoning. CLUE achieves a success rate within 7 percentage points of an oracle policy and outperforms an LLM-enabled planner without closed-loop feedback by a 4x margin. Supporting experiments demonstrate that simply building and then querying a language-enriched map is insufficient to resolve complex contextual planning tasks; these approaches achieve roughly one third the success rate of CLUE while requiring over 10x more VLM tokens. We provide additional information at https://zacravichandran.github.io/CLUE.
1 Introduction
Underspecified tasks in unfamiliar environments require robots to infer what success means, which entities matter, and where or how to inspect them. CLUE couples LLM-generated hypotheses with a language-embedded map and closed-loop interaction to resolve these uncertainties.
1 Introduction
- Contextual uncertainty couples unknown task success and relevant entities with unknown locations and functional interactions, such as removing occlusions.The hazardous-barrel example requires the robot to locate candidates, rule out a false positive, and remove tarps to inspect hidden signs.Source: §1 · p. 1, Figure 1 · p. 2
- Prior language-driven planners often assume explicit instructions and high-quality prior information, while map retrieval and task-agnostic captions lack verification or intervention mechanisms.Related uncertainty-planning methods also assume task-relevant semantics are known a priori.Source: §1 · pp. 2–3
- CLUE uses an LLM to hypothesize relevant information and actions, grounds hypotheses in a language-embedded map, and refines plans through interaction and new observations.The policy may broaden an overly specific hypothesis when the map cannot ground it.Source: §1 · p. 3
- Within 7 percentage points of an oracle, CLUE achieves over 4x the success rate of an LLM-enabled planner without feedback and over 3x that of map-build/query.The evaluation covers 15 tasks in three real-world environments.Source: §1 · p. 3
2 Related work
Related work spans foundation-model planning, active perception, and planning under uncertainty. The paper distinguishes CLUE by combining semantic grounding with closed-loop hypothesis refinement, targeted inspection, and physical intervention.
LLM-Driven Planning
- LLM planning divides between direct control-sequence generation and high-level translation into objectives for classical solvers.The passage describes direct approaches as semantically adaptive but often lacking formal safety guarantees and struggling with high-frequency reactive constraints.Source: §2 · p. 4
Language-Driven Planning with Active Perception
- Active-perception systems can resolve visual ambiguity, but many require targets in view or near-complete maps and do not reason over partial observability.The cited PDDL approaches resolve visual ambiguities or verify preconditions [36], while semantic-reconstruction methods rely on near-complete environmental understanding [11].Source: §2 · p. 4
Language-Driven Planning with Uncertainty
- Uncertainty-aware planners use belief spaces or POMDPs, but the surveyed methods do not dynamically infer task relevance or physical affordances as semantic understanding changes.CLUE instead reasons over semantically embedded maps to locate conceptually relevant items, then interprets objects and evaluates their physical appropriateness.Source: §2 · p. 4
3 Problem Definition
The problem formalizes underspecified tasks in unknown environments, where success conditions and feasible actions are uncertain. CLUE approximates belief over this uncertainty with a hypothesis state that is updated as observations arrive.
3 Problem Definition
- The robot operates without a predefined object catalog or scene graph and assumes reliable odometry, a static episode environment, and task information observable from RGB-D after inspection or occlusion removal.The robot has navigation, inspection, and manipulation behaviors, and the episode ends when it answers or exhausts its action budget.Source: §3 · pp. 4–5
- The objective maximizes the probability of terminating in a goal state while discouraging both unnecessary information gathering and premature answers.The policy chooses actions based on its belief state.Source: §3 · p. 5
- Contextual uncertainty jointly concerns task-relevant semantics and success conditions, entity locations and feasible actions, and functional interactions needed for progress.The exact belief over goal states, environment state, and feasible actions is intractable in continuous open-vocabulary settings, motivating a hypothesis-state approximation.Source: §3 · pp. 5–6
- CLUE approximates belief with a symbolic hypothesis state covering the task, discovered entities, and outstanding uncertainties, then replans as observations reduce uncertainty.The policy proposes behaviors over hypothesis entities, and environment grounding rejects proposals that cannot be realized.Source: §3 · p. 6
4 Actively Resolving Contextual Uncertainty with CLUE
CLUE pairs an LLM policy with an online language-embedded map to ground hypotheses about underspecified tasks and refine plans through closed-loop interaction. Its representation, hypothesis state, grounding process, and action policy connect contextual inference to inspection and task progress.
4 Actively Resolving Contextual Uncertainty with CLUE
- CLUE gives the policy two action categories—hypothesis grounding and active information gathering—to turn an underspecified task into grounded objectives and environment actions.The LLM policy uses manipulation, inspection, and navigation, then replans as new information arrives.Source: §4 · p. 6, Figure 2 · p. 7
4.1 Environment Representations
- The language-embedded map stores online voxel features, while a sparse observation graph exposes discrete objects and traversable regions with coordinates and natural-language attributes.The graph is updated through textual operations, and object descriptions can change as evidence is gathered.Source: §4.1 · pp. 7–8
4.2 Contextual Planning
- At each planning iteration, the LLM maintains a symbolic hypothesis state of confirmed entities, unverified or ruled-out hypotheses, and unresolved task-relevant uncertainties.Confirmed entities ground later decisions, ruled-out hypotheses prevent repeated effort, and open uncertainties guide information gathering.Source: §4.2 · p. 8
4.3 Hypothesis Grounding
- For grounding, CLUE queries the online map with a task-relevant concept, scores voxel relevance against background concepts, clusters retained voxels, and can broaden an overly specific query when results are empty.The query yields candidate entities subject to a confidence threshold.Source: §4.3 · pp. 8–9
- CLUE places candidate viewpoints in nearby freespace and supplies a Euclidean-distance-optimal inspection order, but the policy must still infer preconditions such as navigation or clearing occlusion.The visitation order is a hint, not a complete action plan.Source: §4.3 · p. 9
4.4 Active Information Gathering
- After map queries or direct observations, the policy constructs and executes actions, replans as it learns, and may stop once the task is resolved.Available behaviors include navigation, VLM-based inspection, manipulation, querying, and answering.Source: §4.4 · pp. 9–10
5 Experimental Evaluation
The evaluation tests CLUE on real-world tasks and compares it with oracle, open-loop, and map-retrieval approaches, then examines retrieval sufficiency and path planning.
5.1 Implementation details
- CLUE runs on Spot with a forward-facing ZED 2i camera and onboard Jetson AGX Thor; GPT-5.1 queries use a network connection.The mapping framework extends RayFronts with RadSeg; navigation uses Nav2, while other processes run onboard.Source: §5.1 · pp. 10–11
5.2 Experimental setup
- Fifteen tasks span object disambiguation, functional inference, and occlusion reasoning across two indoor settings and outdoors; each task is evaluated once against oracle and NLMaps baselines.The oracle receives step-by-step instructions; NLMaps uses the same map and action space but plans open-loop.Source: §5.2 · pp. 11–12, Table 1 · p. 12
5.3 Results
- 86.7% task success for CLUE compares with 93.3% for the oracle and 20% for NLMaps across fifteen experiments.Source: §5.3 · p. 12, Table 1 · p. 12
- CLUE uses more actions, time, and LLM/VLM resources while resolving map distractors; its failures include contextual inference and behavior execution.NLMaps often stops at a false-positive map result without exploring other candidates.Source: §5.3 · p. 12
5.4 Does Information Retrieval on a Map Suffice?
- 22.2% task success for DAAAM contrasts with CLUE; DAAAM uses over 2.8x as many VLM calls and 12x as many VLM tokens.DAAAM’s reported failure modes include contextual imprecision, missed object detections, and inability to resolve occlusions.Source: §5.4 · p. 13, Table 2 · p. 13
5.5 Can LLMs Provide Informative Paths?
- For six or more waypoints, CLUE without a TSP hint produces paths over 1.5 times as long; Path LLM rivals TSP.The distance ratio is averaged over 25 trials for two to ten waypoints.Source: §5.5 · p. 14, Figure 6 · p. 14
6 Limitations and Future Work
The language-embedded map is memory intensive, with outdoor maps reaching 100 GB. The evaluation uses only a cloud-based model; future work could investigate memory reduction and on-device policies.
6 Limitations and Future Work
- Outdoor language-embedded maps reach 100 GB, making map memory a stated limitation.Submap caching, which stores inactive map portions to disk, is suggested as a possible way to reduce memory requirements.Source: §6 · p. 14
- Submap caching is proposed as a possible memory-reduction approach, while semantic frontier exploration is identified as future work.The paper points to existing work on caching inactive map portions to disk and to semantic frontier exploration as a promising avenue.Source: §6 · p. 14
- The study considers only a cloud-based model, GPT-5.1; smaller open-source models and on-device policies remain future work.The authors also mention distillation as a possible route to on-device policies.Source: §6 · p. 14
7 Conclusion
CLUE addresses underspecified natural-language tasks by iteratively grounding and verifying hypotheses through interaction. In real-world experiments, it approaches oracle success and outperforms an LLM-enabled planner without closed-loop information gathering.
7 Conclusion
- CLUE uses an LLM-enabled policy to infer hypotheses, grounds them into concrete actions with a language-embedded map, and updates hypotheses through interaction.The policy uses observations from active interaction to iteratively replan.Source: §7 · p. 14
- CLUE rivals an oracle within 7 percentage points in task success, outperforms a planner without closed-loop information gathering by 4x, and a mapping-then-querying approach by 3x.The comparison is reported across real-world experiments in indoor and outdoor environments.Source: §7 · p. 15
- The real-world experiments span object disambiguation, functional-affordance reasoning, and occlusion handling in indoor and outdoor environments.The conclusion reports these settings as the scope of the experiments.Source: §7 · p. 15