PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models

Ashay Athalye, Nishanth Kumar, Tom Silver, Yichao Liang, Jiuguang Wang, Tomás Lozano-Pérez, Leslie Pack Kaelbling

arXiv:2501.00296v4cs.ROcs.AIcs.CVcs.LG

TL;DR

Robots must plan long-horizon tasks from low-level skills despite wide variation in objects, scenes, and goals. Pix2pred learns symbolic world models from demonstrations using VLM-proposed predicates, then plans with those models; the authors report generalization across simulation and real-world tasks, including novel objects, goals, and longer horizons.

  • Problem

    Robots must sequence and parameterize low-level skills across many objects, configurations, backgrounds, and goal specifications to solve long-horizon tasks.

    Source: §1 · p. 1

  • Method

    Pix2pred uses VLMs to propose pixel-grounded predicates and an optimization-based learner to select a compact symbolic world model from demonstrations for planning.

    Source: §1 · pp. 2–3

  • Results

    Pix2pred generalizes across novel objects, numbers of objects, goals, and horizons, achieving the highest success rates on novel instances with more objects, complex goals, and longer horizons.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Learned symbolic models support efficient planning for novel tasks, including two complex long-horizon tasks on a real-world Spot robot.

    Source: §1 · pp. 2–3

    The method assumes relevant scene objects have unambiguous descriptors and that objects are fully observable in all states.

    Source: §7 · p. 10

Abstract

from arXiv · show

Our aim is to learn to solve long-horizon decision-making problems in complex robotics domains given low-level skills and a handful of short-horizon demonstrations containing sequences of images. To this end, we focus on learning abstract symbolic world models that facilitate zero-shot generalization to novel goals via planning. A critical component of such models is the set of symbolic predicates that define properties of and relationships between objects. In this work, we leverage pretrained vision-language models (VLMs) to propose a large set of visual predicates potentially relevant for decision-making, and to evaluate those predicates directly from camera images. At training time, we pass the proposed predicates and demonstrations into an optimization-based model-learning algorithm to obtain an abstract symbolic world model that is defined in terms of a compact subset of the proposed predicates. At test time, given a novel goal in a novel setting, we use the VLM to construct a symbolic description of the current world state, and then use a search-based planning algorithm to find a sequence of low-level skills that achieves the goal. We demonstrate empirically across experiments in both simulation and the real world that our method can generalize aggressively, applying its learned world model to solve problems with a wide variety of object types, arrangements, numbers of objects, and visual backgrounds, as well as novel goals and much longer horizons than those seen at training time.

1 Introduction

The paper targets long-horizon robotics tasks whose required skills and parameters vary across objects, configurations, backgrounds, and goals. Pix2pred learns symbolic world models from a handful of demonstrations, then uses planning to generalize to novel tasks; experiments report strong performance on more complex problems.

1 Introduction

  • Robots must select and parameterize sequences of low-level skills across an extremely large space of objects, configurations, visual backgrounds, and goals.Even simple instructions can require substantially different skill sequences and parameters in different situations.Source: §1 · p. 1
  • In Cleanup, a novel task requires dumping a bin, using an eraser, wiping a table, and returning the eraser in a different room.The demonstrations cover wiping a table and throwing away an object; planning over distinctions such as object identity and surface clarity supports composing a different sequence.Source: §1 · p. 2, Figure 1 · p. 2
  • Predicate learning must determine predicate types, counts, and implementations without initial constraints, while selecting a model that supports efficient planning across tasks.The demonstrations alone are compatible with infinitely many possible world models.Source: §1 · p. 2
  • Pix2pred uses VLM perceptual grounding and common-sense capabilities within program synthesis to invent predicates and learn symbolic world models from demonstrations.It proposes feature-based and pixel-grounded predicates, then selects predicates and operators using an objective for efficient planning.Source: §1 · pp. 2–3
  • Pix2pred achieves the highest success rates on novel instances involving more objects, more complex goals, and longer horizons across three simulated domains.The paper also reports that a real-world Boston Dynamics Spot robot solved two complex, long-horizon tasks using models learned from fewer than 15 egocentric human demonstrations.Source: §1 · p. 3

2 Related Work

Prior work learns abstractions under assumptions such as pre-given predicates, online interaction, or dense language descriptions. Pix2pred instead learns predicates and operators offline from image-based demonstrations, using VLMs to propose and ground predicates and a planning-efficiency objective to select them.

2 Related Work

  • Pix2pred invents predicates and operators offline from image-based demonstrations without supervision on the kinds or number of predicates to learn.Some prior methods assume predicates or operators, learn through online interaction, or require dense language descriptions for demonstrations.Source: §2 · pp. 3–4
  • Unlike skills-to-symbols methods, pix2pred uses a VLM to propose and ground predicates from raw demonstrations and optimizes their selection for planning efficiency.The approach aims to learn from a few short, embodiment-agnostic demonstrations while biasing representations toward interpretable, commonsense predicates.Source: §2 · p. 4
  • Unlike online VLM predicate-invention methods, pix2pred learns from offline, embodiment-agnostic demonstrations containing only positive skill executions [3] [42].The cited online methods typically require exploratory skills and contrastive signals from successful and failed executions; pix2pred is presented as complementary to them.Source: §2 · p. 4
  • Pix2pred modifies the bilevel planning framework to handle noisy VLM-generated predicates, unlike the handcrafted, noiseless state-space assumption in Silver et al. [2].The paper identifies soft precondition intersection as one such modification.Source: §2 · p. 4

3 Background and Problem Setting

The paper frames decision-making as planning over image-based states, objects, and low-level skills. It learns symbolic models from demonstrations and uses them to plan for novel goals.

3.1 States, Actions, and Objects

  • A state combines camera images with object-centric feature vectors for typed objects, while each action instantiates a temporally extended skill with object and optional continuous parameters.Object descriptions are assumed to identify the specified objects in the images.Source: §3.1 · pp. 4–5

3.2 Predicates, Symbolic World Models, and Planning

  • Predicates classify states as true or false; together with skill-linked symbolic operators, they define an abstract world model usable for planning.The planning procedure evaluates predicates on the initial state, then calls a PDDL planner to achieve the goal.Source: §3.2 · pp. 5–6

3.3 Learning and Deployment

  • Each demonstration contains objects, a state-action trajectory, and a conjunctive goal; initial predicates include those needed to express demonstration goals.In Cleanup, the initial set includes WipedOffTable, InsideContainer, Holding, and HandEmpty.Source: §3.3 · p. 6
  • At deployment, the system receives objects of types seen in demonstrations, a novel initial state, and a novel goal expressed using initial predicates, and must produce an achieving action sequence.The paper studies long-term decision-making from image-based states and a set of user-specified objects.Source: §3 · p. 4, §3.1 · p. 4, §3.3 · p. 6

4 Learning Symbolic World Models

The method builds a candidate predicate pool from demonstrations, evaluates visual predicates with a VLM, and optimizes a compact symbolic model for planning.

4 Learning Symbolic World Models

  • The model-learning procedure combines invented predicates with initial predicates, learns skill-linked operators, and may train samplers for continuous skill parameters.It focuses on inventing visual predicates and operators from demonstrations.Source: §4 · p. 6

4.1 Proposing an Initial Pool of Visual Predicates

  • For each demonstration, a VLM receives sequential images, intervening skill names and arguments, and object descriptions, then proposes ground-atom strings.Proposals are parsed, invalid object names removed, atoms lifted to typed predicates, and duplicates discarded.Source: §4.1 · p. 6

4.2 Implementing Visual Predicates with a VLM

  • Visual predicates are evaluated by querying a VLM with a grounded predicate prompt and requesting true, false, or unknown labels, with unknown treated as false.For noninitial states, the query also includes the previous image state, prior labels, and executed action.Source: §4.2 · p. 7

4.3 Learning Symbolic World Models via Optimization

  • Hill-climbing selects predicates by optimizing a planning-based objective while learning operators over the selected set; early stopping and noise-aware operator learning address overfitting.Operators modeling less than a specified fraction of the data are deleted.Source: §4.3 · p. 7
  • Optimization can choose among closely related or synonymous predicates, favoring those that are accurately labeled and important for planning across demonstrations.The paper gives NextTo and Touching as examples.Source: §4.3 · p. 7

5 Experiments

Across simulation and real-world tasks, pix2pred generalizes to novel settings and goals, with predicate subselection improving performance over direct VLM selection. The experiments compare planning against baselines and examine failure modes.

Results and Analysis

  • pix2pred outperforms prior approaches on 4 of 5 simulated domains, often by a wide margin, and generalizes better to complex tasks than ViLa few-shot.ViLa with demonstrations performs comparably on the two-action Kitchen task, but drops sharply on Burger tasks of 15 or more actions.Source: Figure 4 · p. 8, §5 · p. 9

Experimental Setup

  • The evaluation spans two real-world Spot tasks and three simulated domains, including Burger tests with longer sequences and novel combinations of ingredients.Simulation varies object counts and locations across five seeds with ten test tasks per seed; real-world evaluation uses six Cleanup and ten Juice demonstrations.Source: §5 · pp. 8–9

Results and Analysis

  • pix2pred outperforms ViLA in real-world domains and generalizes across object instances, backgrounds, object counts, and novel goal compositions with 6 Cleanup and 10 Juice demonstrations.The table evaluates five tasks across two real-world domains and reports success out of three random initial states.Source: Table 1 · p. 9, §5 · p. 9
  • Explicit predicate subselection improves test performance over direct VLM selection; the no-subselection baseline fails, while selecting many unnecessary predicates bloats operators and overfits.In most domains the VLM proposes more than 100 predicates.Source: §5 · p. 9
  • Manual inspection attributes all real-world failures and most simulation failures to incorrect VLM predicate classification at test time rather than failure to learn a generalizing model.The authors distinguish predicate-classification errors from failures of the learned model to generalize to test planning.Source: §5 · p. 10

6 Conclusion

The conclusion presents pix2pred as a way to invent symbolic world models for planning directly from pixels, with few demonstrations and broad generalization.

6 Conclusion

  • pix2pred invents symbolic world models that support planning directly from input pixels.The conclusion reports learning from 3–12 demonstrations.Source: §6 · p. 10
  • The learned models use semantically meaningful predicates that afford efficient planning across novel object instances, object counts, task horizons, and goals.The conclusion describes generalization across a range of problems.Source: §6 · p. 10
  • The paper reports aggressive generalization from a handful of demonstrations across novel object instances, numbers, horizons, and goals.The conclusion characterizes the method as inventing symbolic world models for pixel-based planning.Source: §6 · p. 10

7 Limitations

Pix2pred currently assumes fully specified relevant objects, complete object observability, and demonstrations segmented into named, pre-existing skills. Its hill-climbing model-learning procedure can also be slow and sensitive to hyperparameters.

7 Limitations

  • Each task requires unambiguously described relevant objects and fully observable objects, assumptions that may not hold in novel real-world environments.Real-world object descriptors are specified manually and can be time-consuming.Source: §7 · p. 10
  • Hill-climbing can take many seconds to evaluate one candidate predicate set, especially with more demonstrations or a larger initial predicate pool.The procedure is also sensitive to hyperparameter choice.Source: §7 · p. 10
  • Demonstrations must be segmented into provided parameterized skills with function-matching names, and the skills and policies must already exist.The authors identify integration with low-level skill learning as future work.Source: §7 · p. 10

A.1 Symbolic Operators

Symbolic operators describe abstract transitions in terms of predicates and connect those transitions to executable skills. The appendix specifies how states are abstracted and how operators are grounded and executed.

A.1 Symbolic Operators

  • An abstract state contains the ground atoms whose predicates evaluate as true in the underlying state.The paper denotes this mapping as ABSTRACT(state, predicates).Source: §A.1 · p. 15
  • Each operator comprises typed variables, preconditions, add and delete effects, and a controller associated with the operator.The controller's discrete parameters must be among the operator variables.Source: §A.1 · p. 15
  • A grounded operator applies when its preconditions hold, producing a successor state by removing delete effects and adding positive effects.The operator specifies a corresponding grounded controller, whose continuous parameters are proposed by a sampler.Source: §A.1 · p. 15

A.2 Planner Implementation Details

The planner abstracts the initial image-based state, searches for an abstract plan, and executes the corresponding controllers sequentially. The implementation uses greedy execution, while more sophisticated planning is possible with a photorealistic simulator.

A.2 Planner Implementation Details

  • The abstract plan is converted into a controller sequence; each controller's continuous parameters are sampled before execution, and controllers run sequentially.The implementation follows a greedy execution strategy.Source: §A.2 · p. 15
  • The implementation abstracts the initial state using predicate classifiers, then calls a classical planner with the task objects and operators to find an abstract plan.The plan is a sequence of operators intended to achieve the goal, if one exists.Source: §A.2 · p. 15
  • With a photorealistic simulator for the transition function, more sophisticated task-and-motion planning strategies, including bilevel planning, could be used.The authors present these as possible alternatives to the implemented strategy.Source: §A.2 · p. 15

A.3 Operator Learning

The operator-learning procedure adapts cluster-and-intersect learning to noisy predicate values. It groups demonstrated transitions, induces operator arguments and effects, learns preconditions, and prunes operators supported by too little data.

A.3 Operator Learning

  • Demonstration transitions are abstracted with predicates and grouped when controllers and effects unify under mappings between objects.The procedure first represents demonstrations as state-action-next-state transitions.Source: §A.3 · pp. 15–16
  • For each transition group, objects in controller arguments and effects are replaced by same-type variables to induce operator arguments and add/delete effects.The induced effects are checked against the other transitions in that equivalence class.Source: §A.3 · p. 16
  • Preconditions are learned by a soft intersection: an atom qualifies when true in more than the specified fraction of transitions in its group.The experiments set the precondition fraction hyperparameter to 0.8.Source: §A.3 · p. 16
  • To reduce overfitting to noisy transitions, operators with too little data relative to the transitions for their controller are discarded.This pruning step addresses small equivalence classes formed when noise prevents transitions from unifying.Source: §A.3 · p. 16

A.4 Sampler Learning

The method learns one sampler per operator from operator-specific transitions, mapping object features to continuous skill parameters. Each sampler combines parameter prediction with validity filtering.

A.4 Sampler Learning

  • Each operator sampler predicts continuous skill parameters from the feature vectors of its argument objects.Training inputs concatenate features for objects mapped from operator arguments; the target is the continuous parameter vector used for the next skill.Source: §A.4 · p. 16
  • Operator-specific training datasets contain transitions from the equivalence class used to induce that operator.The argument-to-object mapping determines which object features form each training input.Source: §A.4 · p. 16
  • Each sampler uses a Gaussian predictor for parameter means and covariances, then a validity classifier to support rejection sampling.The classifier receives concatenated object features and a candidate parameter vector; negative examples use the same controller but transitions outside the operator’s training class.Source: §A.4 · p. 17

A.5 Additional Prompting Details

The appendix details VLM prompting for labeling ground atoms and proposing candidate predicates. Labeling uses temporal comparisons and accuracy aids, while proposal prompts seek diverse predicates for later selection.

A.5.1 Atom Labeling

  • Ground-atom labels are queried in batches with explanations, although separate queries might be more accurate but slower.The prompts include atom strings, and the initial-state prompt also requests True, False, or Unknown values.Source: §A.5.1 · p. 17
  • Chain-of-thought scene comparisons improved labeling accuracy, and object-name annotations were critical in simulation but unnecessary in real-world environments.Simulation annotations use object type plus an instance number and track individual objects over time.Source: §A.5.1 · pp. 17–18
  • For t > 0, the VLM receives current and previous image-based states, the action executed to reach the current state, and its prior response; for t = 0, it receives initial image-based state.The t = 0 prompt asks the model to label listed atom values True, False, or Unknown.Source: §A.5.1 · p. 17, Figure 5 · p. 18
  • Labeling accuracy was supported by training-time double-checking, while close-up crops of interacted objects were used only in Burger.The VLM is asked to recheck its earlier reasoning against the previous predicate values.Source: §A.5.1 · p. 19

A.5.2 Atom Proposal

  • Proposal prompts seek diverse, decision-relevant atoms by listing action-related changes and other useful predicates, with synonyms and antonyms for each.The initial pool must contain important predicates because later subselection can remove irrelevant ones but cannot select predicates that were never proposed.Source: §A.5.2 · pp. 19–20
  • In Coffee, limiting each trajectory’s proposal prompt to 5–10 predicates helps constrain hill climbing when the pool would otherwise exceed 200.Across n demonstrations, this yields 5n to 10n proposed atoms.Source: §A.5.2 · p. 21
  • In the Burger cook example, cooked(?p:patty) is accurately labeled and most relevant to the goal, so the approach selects it from the proposed predicates.The VLM also proposes clear, cool, grilling, raw, and uncook predicates for the patty.Source: Figure 6 · p. 22

A.6 Additional Experimental Details

This appendix covers experimental setups, real-robot implementation and limitations, and learned predicate and action-model examples across the paper’s domains. The examples show how predicates represent object states and how skill preconditions and effects represent task actions.

A.6.1 Experimental Evaluation and Hyperparameters.

  • The appendix reports domain hyperparameters, with applicable values selected using a validation set, and defines Ndemo as training-dataset size.Table 2 is identified as listing the domain parameters used in the experiments; its values are not included in the supplied passage.Source: §A.6.1 · p. 22, Table 2 · p. 22

A.6.2 Additional Environment Details.

  • Kitchen demonstrations trained a two-skill kettle-boiling sequence, while test tasks moved a kettle from the front-right to the back-right burner.The demonstrations turned on a knob and pushed a kettle onto the back-left burner; test plans were two steps and could be solved by replaying the demonstrations.Source: §A.6.2 · pp. 22–23
  • Burger experiments varied demonstrations and test goals across single- and multi-patty burgers, open-face burgers, and chopped-lettuce burgers.Bigger Burger used 12 demonstrations and tested 10 tasks involving two cooked patties, sometimes alongside an open-face burger with the robot starting with a raw patty. More Burger Stacks used 12 demonstrations and tested 5 or 6 open-face burgers, while Combo Burger used 12 demonstrations and tested two burgers with chopped lettuce on cooked patties, sometimes plus another cooked-patty burger.Source: §A.6.2 · pp. 24–25
  • Coffee demonstrations filled one or two cups, and the test task required filling three cups on the table.The described skill set included jug handling, machine operation, pouring, and rotating the jug until its handle was accessible.Source: §A.6.2 · p. 25
  • Cleanup and Juice demonstrations supported test variants involving new instances or backgrounds, more objects, and novel goals requiring unseen task sequences.Cleanup used six demonstrations for clearing, wiping, or dumping objects; novel goals included retrieving an eraser from a narrow bin before wiping. Juice used ten demonstrations involving fruit, cups, juicing, and emptying containers; a novel goal required emptying a full cup before running the juicer.Source: §A.6.2 · pp. 25–27

A.7.1 Real-Robot System Implementation

  • The real-robot system reused Kumar et al.’s [43] perception, mapping, localization, and sampling pipelines, with AprilTags required for movement skills.Object-centric states used DETIC [52] and SAM [53]; skills were implemented through Spot SDK calls, and image-based states used Spot2’s six onboard RGB cameras.Source: §A.7.1 · p. 27

A.7.2 Robot Domains Additional Details.

  • In Juice, the VLM struggled to identify cups below the juicer valves, so the authors manually marked the waste and juice regions for learning and test-time planning.For the robot domains, hand-related visual predicates were excluded because Spot’s onboard cameras could not see the hand; HandEmpty and Holding were supplied directly. The atom-proposal prompt was also capped at 25 atoms per demonstration after proposals sometimes exceeded 1,000 and slowed optimization.Source: §A.7.2 · p. 28

A.8 Examples of Learned Predicates and Learned Action Models per Environment

  • Learned-model examples include a Kitchen operator that establishes KettleBoiling and Burger predicates such as cooked0 and empty_hands0.The examples show STRIPS action models tied to low-level skills, including pushing the kettle onto a burner and picking a patty.Source: §A.8 · pp. 28–29
  • Combo Burger models include predicates for cooked patties, chopped lettuce, and robot clearance, with operators for cooking and chopping.The Cook operator adds cooked0 after its grill-related preconditions; Chop adds chopped0 for lettuce on a cutting board.Source: §A.8 · pp. 31–32
  • Coffee models include predicates for cups receiving liquid, filled jugs, and jugs inside the coffee machine.These predicates are named CupReceivingLiquid0, JugFilled0, and JugInMachine0.Source: §A.8 · p. 33
  • Coffee operators model placing a jug in the machine, filling it, and pouring liquid into a cup.TurnMachineOnAndFill adds JugFilled0, while PourSomeLiquid adds CupFilled and CupReceivingLiquid0.Source: §A.8 · pp. 33–34
  • The coffee-machine example models jug rotation, picking, placement, and pouring through predicates for jug pickability and placement, with operators that change holding, location, and cup-filled state.The listed operators encode skill preconditions and add/delete effects, including placing a jug in the machine and picking it up for a subsequent action.Source: §A.8 · p. 34
  • The Juice example combines predicates for machine, container, and object states with operators for placing containers, transferring contents, and running the juicer.Examples include closing the lid, placing containers at juice and waste valve regions, dumping contents, and running the machine to produce juice in a cup.Source: §A.8 · pp. 35–36
  • The listed learned predicates cover floor location, eraser identity, holding, empty hand, object-on-table, and table clearance; operators include picking, placing inside a trash can, and wiping, which adds WipedOfMarkerScribbles.Source: §A.8 · pp. 36–37
Loading 2501.00296v4…