PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Learned Visual Perception and Symbolic Belief-Space Planning

Guy Azran, Michael Navat, Sarah Keren

arXiv:2609.16884v1cs.AIcs.RO

TL;DR

Partially observable robots must plan from uncertain visual state estimates, but existing VLM-as-planner and VLM-as-grounder approaches treat outputs deterministically. RoVLaP turns predicate probabilities into beliefs for conformant planning, and achieves the highest task success across evaluated splits.

  • Problem

    VLM-planning approaches treat visual grounding deterministically, although household scenes can contain ambiguity and hidden information.

    Source: §1 · p. 2

  • Method

    RoVLaP uses VLM predicate probabilities to maintain symbolic beliefs and produces conformant plans over likely states.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    RoVLaP achieves the highest success rate in every split, improving over VLM-G by 116.8% on simple tasks and 166.8% on medium tasks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The belief representation supports robust task-plan synthesis using off-the-shelf planners in visually uncertain settings.

    Source: §6 · p. 10

    Conformant planning trades robustness against completeness, and users can adjust threshold θ if no plan is found.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

In partially observable settings, agents must act without full knowledge of the world state and rely on uncertain state-estimation pipelines. Obtaining grounded and verifiable symbolic plans under such uncertainty remains a key challenge. Recent work has integrated Vision-Language Models (VLMs) to bridge perception and symbolic reasoning, following two main paradigms. The first, VLM-as-planner, maps images directly to action sequences, and the second, VLM-as-grounder, grounds observations into symbolic predicates used as the initial state by off-the-shelf planners. Both approaches ignore uncertainty in the planning process, compromising robustness. We introduce a third paradigm, VLM-as-probabilistic-grounder, a novel approach that captures the uncertainty of VLM predicate groundings as a probability distribution over symbolic states. This enables planning in belief space and producing robust plans under uncertainty. Experiments in simulated household robot settings show improved robustness and task success over deterministic grounding, underscoring how our approach leverages foundation models for reliable planning under uncertainty.

1. Introduction

Existing VLM-planning approaches treat visual grounding as certain, despite ambiguity and missing information; RoVLaP instead uses predicate probabilities to plan over possible symbolic states.

1. Introduction

  • VLM-as-planner generates actions from images, while VLM-as-grounder supplies symbolic predicates to an off-the-shelf planner; both treat VLM outputs deterministically.The paper identifies these as the two established VLM-planning paradigms.Source: §1 · pp. 1–2
  • Visual ambiguity can make a robot plan as if it holds a nearby radio instead of the target book, or repeatedly seek a bowl hidden in a cabinet.Maintaining beliefs over whether the book is held or the bowl is in the cabinet can reduce replanning.Source: §1 · p. 2
  • RoVLaP uses VLM-produced predicate probabilities to maintain a belief over high-level states and formulate planning as conformant probabilistic planning.The approach aims to retain symbolic grounding's interpretability while producing more robust plans.Source: §1 · p. 2
  • The paper introduces VLM-as-probabilistic-grounder, formalizes Robust Visual Planning, and compiles symbolic beliefs derived from VLM probabilities into conformant planning problems.It also develops a planning-execution loop with correctness and safety guarantees under stated conditions on VLM accuracy and usefulness.Source: §1 · p. 2

2. Background and Related Work

Prior work uses VLMs to produce actions or symbolic states, while classical and conformant planning offer different ways to plan under uncertainty and partial observability.

2. Background and Related Work

  • End-to-end Vision-Language-Action policies map observations and language instructions to controls, whereas this paper focuses on VLMs used for task-level planning.Task-level approaches use VLMs as high-level planners or symbolic grounders.Source: §2 · p. 2
  • VLM-as-planner maps observations and task descriptions to action sequences, while VLM-as-grounder assigns Boolean fluents for a symbolic planner; both typically use one output.RoVLaP instead derives per-fluent probabilities, selects states whose cumulative probability reaches θ, and plans for all selected states.Source: §2 · p. 3
  • STRIPS represents a classical planning problem with fluents, an initial state, actions, and goals; grounder-based systems give a planner one grounded state.Planner-based systems generate action sequences in the same symbolic model, and replanning can be ineffective when it is costly or unavailable.Source: §2 · p. 3
  • Conformant planning seeks a plan that achieves the goal from every possible initial state, represented as a belief set.Conformant probabilistic planning replaces that set with a probability distribution over states and seeks a plan meeting threshold θ.Source: §2 · pp. 3–4

3. Problem Formulation

Robust Visual Planning models a robot that infers symbolic state from onboard observations and chooses high-level actions under partial observability and uncertain execution.

3. Problem Formulation

  • Robust Visual Planning targets satisficing task plans that remain effective despite sensor noise, occlusion, and partial observability.The robot perceives through onboard sensors, infers symbolic state from visual input, and selects high-level actions implemented as skills.Source: §3 · p. 4
  • The problem tuple ⟨F, A, G, W, O, M, ξ⟩ combines a symbolic task model with continuous workspace configurations, observations, and stochastic skill execution.F, A, and G define the symbolic task space following STRIPS.Source: §3 · p. 4
  • The workspace W contains feasible robot and object configurations, while O is the observation space, such as RGB-D or multi-view images.A workspace configuration includes robot joint positions, object poses, and environmental states.Source: §3 · p. 4
  • The observation model M maps workspace configurations to distributions over observations that reflect sensor noise, occlusion, and limited field of view.Given w, the robot receives an observation sampled from M(w).Source: §3 · p. 4
  • The STRIPS model specifies expected action effects deterministically, but the skill executor ξ handles actual transitions nondeterministically.The objective is to maximize the probability of reaching a goal-satisfying workspace configuration under partial observability, using ground-truth state mapping and initial configuration for evaluation.Source: §3 · p. 4

4. Robust Vision-Language Planning (RoVLaP)

RoVLaP combines VLM-based symbolic belief estimation with conformant probabilistic planning and an execution strategy for decisions under perceptual uncertainty. Its pipeline updates beliefs from visual observations, plans over likely states, and monitors execution for replanning.

4.1. VLM-Based Symbolic State Estimation

  • For each fluent, RoVLaP derives probabilities from VLM true-false-null predictions, treats null as uninformative, and updates factored beliefs through logarithmic opinion pooling.The full-state belief approximates conditional independence between fluents while enforcing known constraints.Source: §4.1 · p. 5

4.2. Conformant Probabilistic Planning with VLM-based Belief

  • RoVLaP selects a minimum-cardinality, most-likely state subset meeting probability threshold θ, then searches fluent flips to find states for conformant planning.The search extracts states in non-increasing order of constrained belief, although its worst-case explored-state count is exponential in |F|.Source: §4.2 · pp. 6–7
  • If δ(βF,s)<θ, a constraint-satisfying state enters the MLSS; with ρ>0 and weak calibration γ, repeated observations eventually yield a satisficing plan from the true state.The work characterizes VLM accuracy through cumulative fluent error and identifies minimum visibility rate and calibration as sufficient conditions.Source: §4.2 · pp. 7–8

4.3. Executing Conformant Probabilistic Plans

  • The executor monitors belief drift using improbable-plan, unsafe-action, and plan-exhaustion triggers, and replans when observations undermine the current plan.The resulting policy has a conditional robustness guarantee based on perceptual-belief accuracy; VLM queries dominate wall-clock time, while MLSS search and conformant planning retain worst-case exponential complexity.Source: §4.3 · p. 9

5. Use Case for Robust Visual Planning with a Simulated Household Robot

On ViPlan-HH, RoVLaP was evaluated against VLM-as-planner and VLM-as-grounder in partially observable household tasks, and it led success across every difficulty split.

5. Use Case for Robust Visual Planning with a Simulated Household Robot

  • The evaluation used ViPlan-HH, a household benchmark modified to remove privileged information about hidden objects, with egocentric RGB observations and GPT-4.1 for all methods.The benchmark includes simple, medium, and hard tasks categorized by task horizon.Source: §5 · pp. 9–10
  • RoVLaP achieved the highest success in every split; versus VLM-G, success improved 116.8% on simple and 166.8% on medium tasks, while it solved 66.7% of hard tasks.VLM-G solved no hard-task instances.Source: Table 1 · p. 10, §5 · p. 10
  • Against VLM-P, success improved by 333.5%, 704.2%, and 100.3% on simple, medium, and hard tasks; RoVLaP also solved some task families neither baseline solved.Source: §5 · p. 10
  • RoVLaP used between 1.9 and 10.0 times fewer planner calls than the baselines and executed between 1.2 and 6.8 fewer actions on average.Source: §5 · p. 10
  • For hidden bowls and ambiguous book-holding scenes, RoVLaP retained competing state hypotheses and selected plans feasible under either interpretation.In the drawer task it opens the cabinet first; in book sorting it avoids relying on a single holding judgment.Source: §5 · p. 10

6. Conclusion

The conclusion presents probabilistic state estimation as support for robust task planning in partially observable environments. Appendix A derives fluent probabilities from VLM token probabilities, including a null response for insufficient evidence.

6. Conclusion

  • RoVLaP derives a probabilistic, factored state representation from VLMs to update belief and synthesize robust task plans with off-the-shelf planners.Planned extensions include active sensing and evaluation in embodied systems across complex task-and-motion planning domains.Source: §6 · p. 10

Appendix A. Probabilistic Grounding Calculation

  • Grounding extracts VLM probabilities for “true,” “false,” and “null”; when null is more likely than true, the fluent probability is set to 0.5, otherwise true-false probabilities are normalized.Source: Appendix A · p. 14

Appendix B. Proof of Theorem 2

Appendix B proves that Algorithm 1 extracts constraint-satisfying states in non-increasing belief order, using a max-heap search over states reached by fluent flips.

Appendix B. Proof of Theorem 2

  • Along a canonical path formed by flipping differing fluents, unconstrained belief is non-increasing, supporting the heap-order argument.If a higher-belief state remained unvisited, the first unvisited state on its path would still be in the heap and would have belief at least as high.Source: Appendix B · pp. 14–16
  • Algorithm 1 extracts states in non-increasing unconstrained-belief order, as established by induction over the extracted states.The proof uses the max-heap ordering and shows that a state on a canonical path cannot outrank the next extracted state.Source: Appendix B · pp. 15–16
  • When Algorithm 1 extracts a constraint-satisfying state s, its belief is at least that of every state s′ not already in b.States that violate the constraints have zero belief; the remaining states not in b have not yet been extracted.Source: Appendix B · p. 16

Appendix C. Proof of proposition 3

This appendix bounds Algorithm 1’s time and space costs, then proves a sufficient condition for a constraint-satisfying state to enter the MLSS. The finite-step convergence proof proceeds via Theorem 5.

C.1. Time Complexity.

  • Algorithm 1 has time complexity O(|F|k log(|F|k)).The while loop runs at most k times, with up to |F| neighbor expansions per iteration.Source: §C.1 · pp. 16–17

C.2. Space Complexity

  • Algorithm 1 uses O(|F|2k) bits of space.The visited set and heap each store O(|F|k) states, represented as |F|-bit masks.Source: §C.2 · p. 17

D.1. Proof of Theorem 5

  • If a constraint-satisfying state s has δ(βF, s) < θ, then s belongs to MLSS(βF, θ, C).The proof derives a lower bound on s’s unnormalized weight and uses the assumed error bound to establish membership.Source: §D.1 · p. 18

D.2. Proof of corollary 8

Under weak calibration and a positive minimum visibility rate, repeated observations eventually place the true state in the MLSS. A conformant plan for that set is then satisficing from the true current state.

D.2. Proof of corollary 8

  • With minimum visibility rate ρ > 0 and weak calibration error margin γ, a finite number of updates suffices for a conformant MLSS plan to be satisficing from the true state.The proof relies on each fluent appearing at least ρT times within T timesteps.Source: §D.2 · pp. 18–20

Appendix E. Proof of Theorem 15

Appendix E compares the safety of CPP replanning with deterministic VLM grounding. The stated advantage applies when the planning threshold exceeds the most likely state’s probability.

Appendix E. Proof of Theorem 15

  • CPP replanning is at least as safe as deterministic VLM grounding when the planning threshold exceeds β(smax).The stated safety advantage grows exponentially with the entropy of the VLM-induced belief; if β(smax) > θ, the policy reduces to selecting that maximizer.Source: Appendix E · pp. 20–21

Appendix F. ViPlan-HH Domain and Evaluated Tasks

ViPlan-HH models household manipulation through a symbolic task abstraction and RGB observations, with hidden or occluded state handled under partial observability. The appendix describes the domain, example problem files, evaluation scope, and compared planning approaches.

Appendix F. ViPlan-HH Domain and Evaluated Tasks

  • ViPlan-HH is a simulated household domain with a mobile manipulator, egocentric RGB camera, typed objects, and goals involving navigation, manipulation, and containers.Objects outside the camera view, inside closed containers, or occluded are not directly observable; experiments remove privileged simulator information for hidden predicates.Source: Appendix F · p. 21

G.1. Domain File

  • The PDDL abstraction represents reachability, grasping, container state, placement, containment, adjacency, and slicing, while simulator execution handles low-level geometry.Navigation changes reachability, and opening or closing a container changes the reachability of its contents.Source: Appendix F · p. 21

Appendix F. ViPlan-HH Domain and Evaluated Tasks

  • Quantitative evaluation uses task families where conformant planning could be evaluated reliably; remaining families were excluded because the CPOR backend could fail to return.The cited table identifies task families used in the evaluation, and the reported exclusion is attributed to backend nontermination rather than policy outcomes.Source: Appendix F · p. 22, Table 2 · p. 22

G.2. Example “Sorting Books” Problem File

  • The example problem files cover sorting books and cleaning out drawers, including a bowl initially inside a closed cabinet with a sink placement goal.The example snippets specify the bowl inside the cabinet, the cabinet closed, and the bowl on the sink as the goal.Source: §G.2 · p. 27, §G.3 · pp. 27–28

H.1. Baselines

  • The experiments compare VLM-as-planner, VLM-as-grounder, and RoVLaP as approaches for integrating VLMs into a planning loop.Implementation details are deferred to Appendix I.Source: §H.1 · p. 28

H.2. Deeper Analysis

Across two household tasks, deterministic grounding can commit to an incorrect state when objects are hidden or visually misleading. RoVLaP retains alternative state hypotheses and constructs plans feasible across them.

H.2.1. SCENARIO 1: HIDDEN OBJECT

  • For the hidden-bowl task, RoVLaP plans to open the cabinet before navigating to the bowl, yielding a plan valid whether the bowl is inside or not.The VLM-as-planner baseline assumes the bowl is reachable and omits opening the cabinet, so its plan fails when the bowl is inside.Source: §H.2.1 · p. 28

H.2.2. SCENARIO 2: MISLEADING OBSERVATION

  • For the misleading book image, RoVLaP keeps both holding hypotheses and produces a plan feasible whether the robot is already holding the book or not.The VLM-as-grounder baseline can misread a nearby object as held and plan to place the book without first grasping it.Source: §H.2.2 · p. 29

Appendix I. Experimental Baselines

The appendices specify the baseline pipelines, RoVLaP procedure, prompting setup, and acronym meanings. RoVLaP uses predicate probabilities to form a belief and seek a conformant plan, while the prompts distinguish direct plan generation from state questions.

Appendix I. Experimental Baselines

  • VLM-as-planner outputs actions directly, VLM-as-grounder sends a grounded PDDL state to a classical planner, and RoVLaP uses predicate probabilities to select states for conformant planning.RoVLaP updates per-predicate beliefs, filters impossible states, searches for a plan over the selected subset, and updates belief during execution.Source: Appendix I · p. 29
  • Unlike the cited Merler et al. implementation, the experimental VLM-as-grounder setup does not use privileged ground-truth assignments for predicates about invisible objects.The cited implementation supplements its grounded state with simulator information for non-visible predicates.Source: Appendix I · p. 29

Appendix J. VLM Prompts

  • The prompts ask VLM-as-planner to generate plans, while RoVLaP and VLM-as-grounder answer state questions with “yes,” “no,” or “unknown.”The state-question prompt uses “unknown” when the image does not provide enough information.Source: Appendix J · pp. 30, 33

Appendix K. Acronyms

  • The acronym list expands RoVLaP as Robust Vision-Language Planning, VLM as Vision-Language Model, and CPP as conformant probabilistic planning.It also lists RVP as Robust Visual Planning.Source: Appendix K · pp. 33–34

Related papers

Loading 2609.16884v1…