PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning POMDP World Models from Observations with Language-Model Priors

Valentin Six, Frederik Panse, Mathis Fajeau, Lancelot Da Costa, Mridul Sharma, Alfonso Amayuelas, Tim Z. Xiao, David Hyland, Philipp Hennig, Bernhard Schölkopf

arXiv:2605.13740v1cs.LG

TL;DR

Learning POMDP world models from observation-only trajectories is difficult because prior methods often require hidden-state supervision or extensive interaction. Pinductor uses an LLM to induce and refine executable POMDP programs with a belief-based likelihood objective, matching privileged-state LLM baselines and outperforming tabular methods in MiniGrid.

  • Problem

    Learning POMDP world models from observation–action–reward trajectories alone remains difficult, while many LLM-based methods assume access to latent states during learning.

  • Method

    Pinductor uses an LLM to propose executable POMDP components and iteratively refine them using a belief-based likelihood score computed from observations and self-induced beliefs.

  • Results

    Pinductor matches privileged-state LLM baselines in performance and sample efficiency and significantly outperforms tabular POMDP baselines across tested MiniGrid environments.

  • Takeaways & Limitations

    Language-model priors can support sample-efficient POMDP world-model learning without privileged hidden-state access.

  • Takeaways & Limitations

    Evaluation is restricted to MiniGrid environments, leaving transfer to related environments and domains for future testing.

Abstract

from arXiv · show

Whether navigating a building, operating a robot, or playing a game, an agent that acts effectively in an environment must first learn an internal model of how that environment works. Partially-observable Markov decision processes (POMDPs) provide a flexible modeling class for such internal world models, but learning them from observation-action trajectories alone is challenging and typically requires extensive environment interaction. We ask whether language-model priors can reduce costly interaction by leveraging prior knowledge, and introduce \emph{Pinductor} (POMDP-inductor): an LLM proposes candidate POMDP models from a few observation-action trajectories and iteratively refines them to optimize a belief-based likelihood score. Despite using strictly less information, \emph{Pinductor} matches the performance and sample efficiency of LLM-based POMDP learning methods that assume privileged access to the hidden state, while significantly surpassing the sample efficiency of tabular POMDP baselines. Further results show that performance scales with LLM capability and degrades gracefully as semantic information about the environment is withheld. Together, these results position language-model priors as a practical tool for sample-efficient world-model learning under partial observability, and a step toward generalist agents in real-world environments. Code is available at https://github.com/atomresearch/pinductor.

1 Introduction

Pinductor addresses observation-only POMDP world-model learning by using LLM priors to propose and refine executable models from sparse trajectories. It matches privileged-state LLM methods and outperforms tabular baselines in MiniGrid evaluations.

  • Agents must learn internal models of hidden states, dynamics, observations, and rewards before acting effectively in unfamiliar environments.
  • POMDP learning is difficult because classical methods often require extensive interaction, strong structural assumptions, or careful hand engineering.
  • Pinductor uses an LLM to propose executable transition, observation, reward, and initial-state programs from observation–action–reward trajectories, then refines them with belief-based likelihood scoring.
  • Pinductor learns without ground-truth latent states at training or inference time, addressing the privileged-state assumption common in prior LLM-based methods.
  • Pinductor matches privileged-state LLM baselines in reward and sample efficiency while outperforming non-LLM baselines across five partially observable MiniGrid tasks.

2 Related work

Prior work uses executable programs, structured theories, or learned latent models for reinforcement learning and POMDPs, but typically relies on privileged information, extensive descriptions, hand-designed hypothesis spaces, or strong assumptions. Pinductor instead uses LLM priors to induce latent-state POMDP models from partially observed trajectories.

  • POMDP Coder uses LLM-proposed executable POMDP components but assumes privileged hidden-state access during demonstrations and post-hoc online interaction.
  • Other LLM approaches use extensive natural-language task descriptions or deploy LLMs as planners rather than learning executable world models.
  • Code-based world-model research represents models as executable programs refined through language-based feedback, including fully observable reinforcement-learning environments.
  • POMDP learning methods span Bayes-adaptive, spectral, variational, active-learning, and neural latent-world-model approaches, often requiring substantial data or structural assumptions.
  • Theory-based reinforcement learning supports sample-efficient planning with structured programs but searches hand-designed hypothesis spaces tailored to benchmark environments.

3 Background on POMDPs

POMDPs model sequential decisions when the underlying state is hidden, requiring an agent to represent uncertainty over states and observations. Belief filtering and belief-space planning depend on learned transition, observation, reward, and initial-state models.

  • A POMDP models sequential decision-making when the agent cannot directly observe the underlying state.
  • Its transition model predicts next states, its observation model predicts observations, and its reward function specifies immediate rewards for state transitions.
  • Because observations are generally non-Markovian, the agent maintains a belief state representing an approximate posterior over latent states.
  • Belief filtering requires an initial-state distribution, transition model, and observation model, which must be learned from experience in this setting.
  • After learning POMDP components, the agent can plan in belief space using value iteration, POMCP, or deterministic approximations such as DA*.

4 Methodology

Pinductor induces executable POMDP world models from state-free trajectories, using LLM-generated components, particle-filtered belief scoring, and iterative execution-guided refinement. The selected model supports belief-space planning and online updates.

  • Problem formulation: Pinductor learns executable POMDP models from observation–action–reward trajectories without hidden-state labels.Its generate–evaluate–refine–plan loop uses particle filtering to assess whether candidate latent beliefs explain observed trajectories.
  • Model proposal: The LLM receives a task description, offline dataset, and code API, then generates candidate transition, observation, reward, and initial-state components.The API specifies the relevant state, action, and observation spaces.
  • Belief-based scoring: Particle filtering propagates beliefs under realized actions, reweights particles by observation compatibility, and returns likelihood scores and diagnostics for refinement.The score evaluates observation-action fit; reward and termination mismatches enter indirectly through local LLM diagnostics.
  • Belief-based scoring: Pinductor softens deterministic observation predictions with a distance kernel, assigning positive likelihood to observations similar to particle-generated predictions.For MiniGrid, the distance compares visible grid structure, agent direction, and carried object.
  • Iterative refinement: Refinement-by-execution grows a candidate pool by selecting parents with UCB1, generating revised candidates, and retaining a near-best final model.The refinement prompt includes scalar scores, concrete failure cases, and transition-model disagreement signals.
  • Planning and online interaction: The selected model maintains particle beliefs for belief-space planning, while newly collected trajectories trigger further online refinement.Actions are chosen by a POMDP planner using the updated belief state.

5 Experiments

Experiments evaluate Pinductor across five MiniGrid environments using task performance, belief quality, and offline sample efficiency. It matches a privileged-state LLM baseline, outperforms non-LLM baselines, and supports increasingly concentrated beliefs with few trajectories.

  • Experimental setup: Experiments span five MiniGrid environments ranging from elementary EMPTY and CORNERS tasks to more complex LAVA, FOUR ROOMS, and UNLOCK tasks.The suite tests model discovery as transition and reward structure becomes more complex.
  • Main task performance: Pinductor matches the privileged-state LLM baseline while substantially outperforming non-LLM baselines, including a hidden-state-access tabular baseline.Mean episode reward and win rate are reported across environments, with 95% percentile confidence intervals.
  • Belief dynamics: Beliefs become more concentrated and assign increasing mass to the true hidden state as observations accumulate.Pinductor’s smoother entropy reduction follows from graded observation-distance reweighting rather than hard exact matching.
  • Offline sample efficiency: Pinductor achieves comparable performance to the state-access LLM baseline at the same offline trajectory counts despite receiving less information per sample.Both LLM-based methods reach relatively strong performance after only a few trajectories.
  • Offline sample efficiency: The results indicate that LLM priors help propose useful candidate programs from sparse offline data rather than fully identifying model components from scratch.This interpretation is tied to the strong few-trajectory performance of both LLM-based methods.

6 Conclusions and Limitations

The paper concludes that Pinductor can induce executable POMDP models without hidden-state supervision while matching privileged-state LLM baselines and outperforming tabular baselines. Its evaluation remains limited to MiniGrid, and several pipeline components and API-call variance remain open issues.

  • Conclusions: Pinductor induces executable POMDP world models from belief-based likelihood signals without access to hidden states.The method also produces beliefs that converge on the true latent state in the reported experiments.
  • Conclusions: The method matches privileged-state LLM baselines, outperforms tabular baselines, learns from few trajectories, and supports downstream planning under partial observability.These conclusions concern the tested domains and reported experiments.
  • Limitations: Evaluation is restricted to MiniGrid, leaving transfer to related environments and other domains, including comparisons with deep RL, for future study.The stated scope boundary limits conclusions about broader generalization.
  • Limitations: The observation distance, planner, and demonstration buffer are fixed, while reliance on LLM API calls induces high variance.The paper identifies optimizing these components and reducing variance as future directions.

A Methodological Details

Pinductor evaluates candidate POMDP programs with a softened observation likelihood and a particle-filtered expected log-likelihood, while treating the objective as a model-ranking signal.

  • Distance-kernel likelihood: The observation distance combines grid mismatch with penalties for differing agent direction and carried object.The grid, direction, and carried-object components define task-specific observation compatibility.
  • Distance-kernel likelihood: The distance-kernel likelihood softens deterministic LLM-generated observations, assigning greater compatibility to closer predictions.The bandwidth κ controls sharpness: smaller values penalize mismatches more severely, while larger values tolerate near-misses.
  • Belief-based scoring: The particle-filtered score evaluates candidate POMDP models using beliefs induced by their own filtering dynamics.The objective is a posterior expected kernel log-likelihood over particle-filtered beliefs.
  • Belief-based scoring: The score is used only to rank executable candidate models and is not an estimate of exact marginal likelihood.It is a differentiability-free model-ranking signal.

A.2 Particle filtering and rejuvenation

Pinductor uses particle filtering to propagate candidate latent states, weight them by softened observation compatibility, resample beliefs, and rejuvenate collapsed particle populations.

  • Particle filtering: Particles are initialized from the candidate initial-state model and propagated through its transition model after each recorded action.The LLM-generated observation program then produces a prediction for weighting.
  • Particle filtering: Each propagated particle receives a kernel weight based on the candidate observation prediction and recorded observation.Weights are normalized and used for the observation update.
  • Particle filtering: Per-step resampling reconstructs the next belief from integer particle counts proportional to normalized current-step weights.The score uses current-step kernel likelihoods rather than a cumulative product.
  • Rejuvenation: Belief rejuvenation begins when post-observation particle mass falls below K, indicating that the population cannot represent a full particle set.Fresh particles are rolled through the recorded action history and retained according to cumulative kernel likelihood.
  • Rejuvenation: Rejuvenation restores diversity while conditioning on the observed history, although transient particle counts can exceed K.The score remains well-defined because it depends on normalized weights.
  • Implementation: The experiments use K = 10 particles and κ = 0.2 for offline evaluation and online belief tracking.The same filtering procedure supports both model scoring and planning.

A.3 Feedback, refinement, and candidate selection

Pinductor refines executable candidate models using global scores, local failure diagnostics, disagreement-based feedback, and tree-based selection strategies.

  • Feedback signals: Each evaluated candidate produces three feedback objects for the next refinement prompt.These comprise global compatibility, local diagnostics, and QBC disagreement signals.
  • Feedback signals: Local diagnostics identify runtime errors, low-likelihood segments, observation mismatches, and reward or termination mismatches for targeted code edits.The LLM can repair generated code rather than regenerate the entire model.
  • Feedback signals: QBC highlights high-disagreement transition contexts using normalized vote entropy computed on particles sampled from filtering beliefs.The signal is diagnostic only and does not enter the model-selection objective.
  • Candidate refinement: Pinductor maintains a persistent refinement tree and selects parent nodes with UCB1.The exploration bonus favors less-expanded branches while mean kernel scores favor promising candidate families.
  • Candidate selection: After refinement, the final model is sampled from a near-best candidate set rather than chosen by a strict argmax.This preserves access to the best-scoring candidate while allowing structurally different candidates with similar scores.

A.4 Hyperparameters

Pinductor’s belief-based objective evaluates models entirely from observed trajectories, using softened observation compatibility to maintain latent explanations aligned with downstream planning.

  • Hyperparameters: Pinductor uses K = 10 particles, κ = 0.2, five candidates per refinement round, temperature T = 0.1, and a 5,000-expansion planner budget.It also sets λdir = λcarry = 1, αgrid = 3, and ε = 10^-6.
  • Hyperparameters: Increasing the entropy coefficient improved Pinductor’s exploration and average rewards, whereas it produced no significant gain for POMDP Coder.The coefficients were 1.0 for Pinductor and 0 for POMDP Coder.
  • Filtering and likelihood: Particle filtering predicts through transitions, compares predicted observations with recorded observations, and reweights particles by softened compatibility.This adapts standard filtering to executable, potentially deterministic LLM-generated programs.
  • Filtering and likelihood: Rejuvenation rolls newly sampled initial states through the observed action history and retains trajectories compatible with the observations.This replenishes beliefs when imperfect LLM-generated models would otherwise cause particle collapse.
  • Planning alignment: The same filtered belief supports both candidate scoring and downstream planning, without comparing against privileged hidden states.A model scores well only when its own filtering process maintains latent trajectories that explain the observation stream.
  • Objective interpretation: The score is computable without hidden-state labels and ranks executable programs by expected observation energy under their induced beliefs.It evaluates the operational belief needed by the planner rather than an oracle state comparison.
  • Filtering and likelihood: The softened kernel gives positive likelihood to realized observations even when the generated observation program is deterministic.The bandwidth controls how sharply observation mismatches are penalized.
  • Objective interpretation: The objective should not be interpreted as a variational ELBO or exact marginal likelihood.It is a particle-filtered expected likelihood, equivalently a negative expected observation energy under posterior beliefs.

D Experimental details

Stochastic MiniGrid variants test whether Pinductor remains effective when layouts and relevant dynamics vary across episodes.

  • Stochastic MiniGrid variants evaluate robustness to episode-level variation in concrete layouts and model-relevant quantities.Corners is excluded because its goal corner, starting position, and direction already vary across episodes; Lava and Unlock receive increased variation.

D.2 Offline datasets

The experiments use mixed offline trajectories and controlled prompt-information levels, while clarifying how ground-truth-model baselines and planner approximation should be interpreted.

  • Offline datasets: Offline datasets mix successful and unsuccessful episodes, including failures and timeouts when available.This increases behavioral diversity and exposes both positive terminal events and failure modes to model induction.
  • Offline datasets: Both LLM-based POMDP methods use identical datasets across experiments for fair comparison.
  • Ground-truth model baselines: Ground-Truth Models are manually specified model components, not optimal-policy oracles.The downstream planner remains approximate, so generated models can sometimes achieve higher realized reward.
  • Prompt information: Main experiments use the strongest prompt level, L3, while the ablation varies information from no description at L0 to the full description at L3.The descriptions progressively add grid size, layout, goals, and episode-level variation.

E General details on environments

The evaluation spans five partially observable MiniGrid environments and uses observability comparisons to illustrate what Pinductor sees during representative episodes.

  • Observability: Pinductor receives only a 3×3 partial field of view, with visited cells retained and never-visited cells masked.
  • Corners: Corners tests position disambiguation because near-symmetric local views make the goal corner ambiguous.
  • Lava: Lava tests whether the model encodes a single safe passage and terminal zero-reward hazards.Sparse rewards require combining trajectory cues with environment semantics.
  • Four Rooms: Four Rooms tests long-horizon navigation through narrow doorways under uncertainty about random goal placement.
  • Unlock: Unlock tests compositional planning across key pickup, door unlocking, traversal, and goal reaching.The model must represent carrying, door-state transitions, and key-door color matching.
  • Baseline: The tabular baseline separately estimates initial-state, transition, reward, and observation components as frequency tables using privileged hidden-state information.Unseen conditions use a silent default.

G Ablations and Additional Results

Ablations show threshold-like dependence on LLM capability, robustness under stochasticity, task-dependent prompt sensitivity, and reduced performance when semantic labels are weakened.

  • LLM choice: LLM capability shows a threshold-like pattern: weaker models may obtain near-zero reward, while capable models recover key dynamics and then largely saturate.This pattern is clearest in LAVA and also appears in UNLOCK, where stronger models achieve comparable nonzero performance.
  • LLM choice: Table 1 reports mean episode reward, 95% confidence intervals, win rate, and average reward across LAVA and UNLOCK for different LLMs.The ablation varies only the LLM used to propose model code, using n = 10 seeds.
  • Stochastic robustness: Pinductor’s stochastic-environment performance drops versus deterministic versions but remains comparable to the state-access LLM baseline.This suggests that removing hidden-state supervision does not cause disproportionate loss under stochasticity.
  • Prompt information: Prompt-information effects are task dependent, with some environments recoverable from priors and trajectories while others improve only after structural cues are added.
  • Semantic information: Replacing meaningful environment and object names with less informative labels reduces performance across environments.The result indicates that semantic cues help form better initial hypotheses and refinements.

H Implementation and Reproducibility Details

The experiments document compute resources, deterministic evaluation, uncertainty estimation, and licensing details. Runtime is summarized robustly across seeds, while reward and belief uncertainty uses bootstrap intervals.

  • H.1 Computational Resources: Table 2 reports one-seed wall-clock time at N = 10 demonstrations using median and interquartile range across seeds.This choice addresses heavy-tailed runtime outliers caused by incorrect model hypotheses and long inference traces.
  • H.1 Computational Resources: All experiments ran on a single local CPU workstation without GPU acceleration, while LLM calls used OpenRouter.Wall-clock measurements include both local computation and remote LLM API latency.
  • H.2 Reproducibility: Evaluation randomness is controlled with fixed run and episode seeds across Python, NumPy, Torch, and environment resets.Each experimental condition uses 10 run seeds, with deterministic episode seeds for reproducible comparisons.
  • H.2 Statistical reporting: Reward and belief error bars are 95% percentile bootstrap confidence intervals computed over 10 seed-level estimates.Each seed-level estimate first averages the metric over three evaluation episodes.
  • H.2 Statistical reporting: Bootstrap intervals avoid normality assumptions for bounded metrics with n = 10 independent seeds, while runtime uses median and IQR because it is heavy-tailed.The usual Gaussian approximation can be misleading near reward or win-rate boundaries.
  • Licensing: MiniGrid is used under Apache 2.0, LLM APIs under their commercial terms, and reused POMDP Coder code is cited without an explicit repository license.

I Details on prompting methods

This appendix makes the prompting contrast concrete: Pinductor learns from partial observations without realized hidden states, whereas Curtis et al. use privileged state tuples. Both methods generate, evaluate, and refine executable models, but differ in access, granularity, and feedback.

  • I Details on prompting methods: Both methods use offline REx and online generate–evaluate–refine regimes, although Curtis et al.’s deterministic-benchmark online hook is gated off.The displayed Curtis et al. online example therefore reuses an iteration-1 refinement call.
  • I Details on prompting methods: Pinductor prompts contain observation–action–reward sequences, whereas Curtis et al.’s prompts include privileged hidden-state input/output tuples.Pinductor’s no-hidden-state claim concerns realized state values; the state schema remains available through the code API.
  • I Details on prompting methods: Pinductor makes one joint call for (ρ0, T, O, R), while Curtis et al. make four component-specific calls at each refinement step.The appendix shows Curtis et al.’s transition-function call as the representative example.
  • I Details on prompting methods: Pinductor feeds particle-filter kernel scores and per-step disagreement diagnostics into refinement, while Curtis et al. use disagreements against recorded hidden states.
  • I.1 Initial model proposal (offline, iter 0, LAVA): The prompting example describes a 10x10 grid with boundary walls, a variable lava-column gap, an agent starting at (1,1), and a bottom-right goal.The LLM is instructed to infer initial states from environment descriptions and first observations because true initial states are unavailable.
  • I.1 Initial model proposal (offline, iter 0, LAVA): The prompt treats successive observations as views of one persistent world evolving under transitions, not as independent frames.The agent can temporarily mask stable world contents in its field of view.
  • I.1 Initial model proposal (offline, iter 0, LAVA): Observations are 3x3 partial fields of view with agent direction and carrying status, while absolute position and the full grid remain hidden.Rewards and termination are transition outcomes rather than fields that the observation function must reproduce.
  • I.2 Refinement iteration (UNLOCK for Pinductor online; LAVA iter-1 for Curtis et al.): The refinement example reports an overall particle-filter score of -18.86, with 0 denoting perfect prediction and more negative values indicating worse prediction.It also reports best particle distance 0.000, median distance 3.011, and distance variance 3.067.
Loading 2605.13740v1…