Learning POMDP World Models from Observations with Language-Model Priors

1 Introduction

• POMDPs represent uncertainty over states, transitions, observations, and rewards, but classical learners often require many interactions or strong assumptions, while manual specification is demanding.LLM-written executable models offer precise, auditable alternatives to using an LLM directly as a simulator [9] [10].Source: §1 · pp. 1–2
• Most LLM-guided model-learning methods assume latent-state access, and POMDP Coder [14] still receives intermediate ground-truth states after each episode.This leaves settings without online or post-hoc state access beyond the reach of methods requiring that supervision.Source: §1 · p. 2
• Pinductor induces executable transition, observation, reward, and initial-state distributions from observation–action–reward trajectories alone, without ground-truth latent states.An LLM proposes candidate programs and iteratively refines them using a belief-based likelihood score.Source: §1 · pp. 2–3
• Pinductor’s performance scales with LLM capability and degrades when explicit semantic information is withheld, indicating reliance on both the model and textual environment information.The introduction reports evaluation on MiniGrid environments of varying complexity.Source: §1 · p. 2
• Its kernel-based objective turns observation predictions into soft likelihoods and scores candidate models under likelihood expected over beliefs induced by their own filtering dynamics.The per-step repair signal is computable from observation–action–reward trajectories alone.Source: §1 · pp. 2–3
• Across five partially observable MiniGrid tasks, Pinductor matches the reward and sample efficiency of a privileged-state LLM baseline and outperforms non-LLM baselines.The introduction also reports that its beliefs become increasingly concentrated on the true latent state during planning.Source: §1 · pp. 2–3

LLM-guided POMDP and model induction

• Unlike POMDP Coder [14], Pinductor learns executable POMDP models from observation–action–reward trajectories without hidden-state access during training or online interaction.Other LLM approaches rely on extensive natural-language task descriptions [16] or use LLMs as planners rather than model learners [17].Source: §2 · p. 3

LLMs for code-based world models

• Unlike prior code-based world models for reinforcement learning [19], fully observable environments [11] [20] [21], and general algorithms [22], Pinductor learns latent-state generative models from trajectories with unobserved states.This places Pinductor within verbalized machine learning, where models are represented in an LLM’s token space and refined through language-based feedback [18].Source: §2 · p. 3

Learning in POMDPs

• Pinductor complements Bayes-adaptive [24], spectral and variational [25] [26], active-learning, and neural latent-world-model methods by searching for auditable POMDP programs from few trajectories.It assumes a compact latent state space and code interface, while prior methods typically require substantial data, specified model classes, tractable inference, or strong structural assumptions.Source: §2 · p. 3

Theory-based reinforcement learning

• Unlike theory-based RL methods [29], which search hand-designed hypothesis spaces with benchmark-tailored primitives, Pinductor uses an LLM prior to search executable POMDP components from sparse partially observed experience.Theory-based RL demonstrates the value of structured executable world models for sample-efficient planning.Source: §2 · p. 3

3 Background on POMDPs

• A POMDP represents partially observed sequential decision-making with finite state, action, and observation sets, transition and observation probabilities, rewards, an initial-state distribution, and a discount factor.The transition and observation models describe state changes and resulting observations; the reward, initial distribution, and discount factor support decision-making.Source: §3 · p. 3
• Because observations are generally non-Markovian, an agent maintains and updates a belief distribution over latent states using Bayes’ rule, requiring an initial distribution and transition and observation models.In this setting, those models are not given and must be learned from experience.Source: §3 · p. 4
• The paper focuses on learning transition, observation, reward, and initial-state models from partially observed trajectories, then planning in belief space with standard offline, online, or deterministic methods.Examples include value iteration, POMCP, and DA*.Source: §3 · p. 4

4.1 Problem formulation

• The learner receives state-free trajectories with actions, observations, rewards, and termination signals, and learns a probabilistic model that explains them and supports downstream tasks.Trajectory horizons may vary, and hidden states are never observed.Source: §4.1 · p. 4

4.2 Model proposal

• Given a task description, small offline dataset, and code API, the LLM proposes executable POMDP components; experiments use 10 manually collected trajectories, and Fig. 2 depicts the pipeline.The pipeline softens the generated observation model for filtering, then uses the selected model for planning and can add online episode feedback to the replay buffer.Source: §4.2 · pp. 4–5, Figure 2 · p. 5

4.3 Belief-based model evaluation

• Pinductor evaluates candidate models by particle-filtering latent states under observed actions and scoring whether predicted observations explain the trajectory, using a softened observation likelihood for MiniGrid.For MiniGrid, the observation distance combines visible-grid, agent-direction, and carried-object differences.Source: §4.3 · p. 5

4.4 Feedback and refinement

• REx feeds execution failures, observation-distance outliers, reward and termination mismatches, and transition-model disagreement to the LLM, then uses UCB1 refinement and samples from a near-best candidate set.Refinement builds a tree of candidates; the near-best set includes scores within one empirical standard deviation of the best.Source: §4.4 · pp. 6–7

4.5 Planning and online interaction

• The selected model supports belief-space planning with a particle belief and an A*-style planner; after execution, new trajectories are added and trigger further online refinement.The belief is updated with the same distance-kernel observation likelihood used in model evaluation.Source: §4.5 · p. 7

5.1 Experimental setup

• The experiments span five MiniGrid environments from elementary tasks such as EMPTY and CORNERS to more challenging LAVA, FOUR ROOMS, and UNLOCK settings.Baselines include POMDP Coder [14], a tabular model with privileged hidden states, and a uniform-random policy.Source: §5.1 · p. 7, §5.2 · p. 8
• Evaluation measures average episode reward and win rate, with belief entropy and accuracy used as diagnostics of latent-state inference rather than as supervision.The task, belief-quality, and sample-efficiency evaluations assess downstream planning under partial observability.Source: §5.1 · p. 7, §5.2 · p. 8

5.2 Main Results

• Across five MiniGrid environments, Pinductor performs comparably to privileged-state POMDP Coder [14] and substantially outperforms non-LLM baselines, including a hidden-state-access tabular model.The comparison uses observation–action–reward trajectories for Pinductor and tests whether belief-based feedback can replace hidden-state supervision.Source: §5.1 · p. 7, Figure 3 · p. 8, §5.2 · p. 8
• Pinductor’s beliefs become more concentrated and accurate as observations accumulate, with smoother entropy reduction from graded observation similarity rather than hard exact matching.The diagnostics indicate that the induced POMDPs support informative posterior updates for planning under partial observability.Source: §5.2 · p. 8, Figure 4 · p. 9
• Pinductor reaches strong performance with few offline trajectories and matches the state-access LLM baseline at the same sample counts, as LLM priors help identify plausible structure quickly.The demonstrations help propose candidate programs rather than fully identifying transition, observation, and reward models from scratch.Source: §5.2 · pp. 8–9, Figure 5 · p. 9

Conclusions

• Pinductor learns executable POMDP world models without hidden-state access, matching privileged-state LLM baselines, significantly outperforming tabular baselines, and producing beliefs that converge on the true latent state.It learns from a small number of trajectories using a belief-based likelihood signal.Source: §6 · p. 9

Limitations

• Evaluation is limited to MiniGrid; future work should test transfer to environments and domains with different structure [11] and benchmark against deep RL.Source: §6 · p. 9
• Future work could use the LLM prior to optimize the observation distance, planner, and demonstration buffer, which are currently fixed, and reduce high variance from LLM API calls [14].Source: §6 · p. 10

A Methodological Details

• Appendix A.1 specifies the distance-kernel likelihood, while A.2 and A.3 cover particle filtering and refinement, selection, and final model selection.These are implementation details omitted from the main methodology section for space.Source: A Methodological Details · p. 12

A.1 Distance-kernel likelihood

• MiniGrid observation distance combines grid distance with direction and carried-object mismatch penalties: dobs(o′, o) = dgrid(g′, g) + λdir1[θ′ ≠ θ] + λcarry1[c′ ≠ c].The grid is the partial field-of-view grid; θ denotes agent direction and c the carried object.Source: §A.1 · p. 12
• The grid-distance calculation uses visible non-agent cells, with ε = 10^-6 for smoothing, αgrid = 3 for super-linear weak-match penalties, and λdir = λcarry = 1.The passage identifies the visible-cell set as V.Source: §A.1 · p. 12
• The LLM-generated observation program supplies the predicted observation used to form a softened observation model.κ controls likelihood sharpness: smaller values penalize mismatches more severely, while larger values tolerate near-misses more.Source: §A.1 · p. 12
• The particle-filtered kernel score is a posterior expected kernel log-likelihood under the softened observation model, up to fixed normalization constants.It evaluates probabilistic models P_m.Source: §A.1 · p. 12
• This score serves only as a differentiability-free model-ranking signal and does not estimate the log marginal likelihood.Source: §A.1 · p. 12

A.2 Particle filtering and rejuvenation

• For each trajectory, the filter represents the belief with K particles, initialized from the candidate initial-state model and propagated through its transition model before observations are generated and kernel weights assigned.The observation program is generated by the LLM.Source: §A.2 · pp. 12–13
• After each observation update, particles are resampled using current-step kernel likelihoods rather than cumulative weights, and the next belief is reconstructed from integer counts proportional to those weights.This bootstrap-style resampling yields a posterior approximation; the effective sample size is computed from normalized weights.Source: §A.2 · p. 13
• When total post-observation weight falls below K, rejuvenation draws fresh initial-state particles, replays the recorded actions, and retains rollouts in proportion to cumulative kernel likelihood.This restores diversity while conditioning on the observed history; the population can temporarily exceed K, so diagnostic ESS may exceed K while the score remains defined by normalized weights.Source: §A.2 · p. 13
• Experiments use K = 10 particles and κ = 0.2, applying the same filtering procedure to offline model evaluation and online belief tracking during planning.Source: §A.2 · p. 13

• The scalar score summarizes global compatibility with offline trajectories and guides both UCB1 parent selection and final model selection.Source: §A.3 · p. 13
• Local diagnostics expose runtime, likelihood, observation, reward, and termination failures for targeted edits; execution-error candidates remain in the pool but cannot be selected as final models.Source: §A.3 · p. 13
• QBC uses normalized vote entropy on filtering-belief particles to surface high-disagreement observation contexts and actions for the prompt, but does not affect model selection.Source: §A.3 · p. 14

UCB1 parent selection

• UCB1 expands a persistent refinement tree, balancing branches’ mean kernel scores with an exploration bonus that stays finite for unexpanded nodes; the exploration constant is 1.0.Source: §A.3 · p. 14

Final selection

• Sampling from the near-best set avoids overcommitting to small noisy score differences; T = 0.1 allows structurally different near-ties while retaining the best-scoring candidate.Source: §A.3 · p. 14

A.4 Hyperparameters

• Defaults are K = 10 particles, κ = 0.2, M = 5 candidates per REx round, T = 0.1, ε = 10−6, and 5,000 planner expansions.Pinductor also uses λdir = λcarry = 1 and αgrid = 3; the entropy coefficient is 1.0 for Pinductor and 0 for POMDP Coder.Source: §A.4 · p. 14
• Increasing the entropy coefficient improved Pinductor’s exploration and average rewards, but produced no significant performance gain for POMDP Coder.The experiments used coefficients of 1.0 for Pinductor and 0 for POMDP Coder, its implementation default.Source: §A.4 · p. 14

A.5 Role of the components

• Pinductor separates LLM model proposals, particle-filter learning signals, distance-kernel evaluation, QBC diagnostic refinement, and belief-space planning into distinct roles.Planning uses the same representation as evaluation, connecting model proposal, latent-state inference, refinement, and action selection.Source: §A.5 · p. 14

B Particle Filtering Background

• For deterministic LLM-generated observation programs, Pinductor softens prediction mismatches into distance-kernel weights, allowing realized observations to retain positive likelihood during particle filtering.When particle mass becomes too small, it rejuvenates the belief by sampling candidate initial states, rolling them through the action history, and weighting them by cumulative kernel likelihood.Source: B Particle Filtering Background · p. 15
• Pinductor scores models using the same filtered belief used for planning, asking whether latent trajectories explain observations rather than matching privileged hidden states.The score can be computed from observations, actions, rewards, and termination signals, and applies to executable models with deterministic transition and observation functions.Source: B Particle Filtering Background · p. 15, C Distance-Kernel Likelihood and… · p. 17

C Distance-Kernel Likelihood and Energy Interpretation

• The model-ranking score evaluates the final models by their overall benchmark comparison.No benchmark comparison is reported in these supplied passages; the score is instead described as a filtered expected log soft-observation likelihood.Source: C Distance-Kernel Likelihood and… · p. 16
• The distance-kernel score is equivalent, up to model-independent constants, to minimizing expected observation energy under the candidate model’s filtered belief.The bandwidth κ controls soft-likelihood variance and shapes the belief; the score is a particle-filtered expected likelihood, not an ELBO or exact marginal likelihood.Source: C Distance-Kernel Likelihood and… · p. 17

D.1 Stochastic MiniGrid variants

• Stochastic Lava and Unlock variants test whether generated models capture task structure rather than memorize one layout.Corners is excluded because it is already stochastic. In Lava, variation is increased for the agent’s start position and direction and the goal position; in Unlock, it is increased for wall and goal positions.Source: §D.1 · p. 17

D.2 Offline datasets

• Unlike POMDP Coder’s success-only oracle trajectories, the offline datasets mix successful episodes with failures and timeouts, improving some environments and diversifying evidence about model structure.Both LLM-based methods use the same datasets to ensure fair comparison.Source: §D.2 · p. 17

D.3 Ground-truth model baselines

• Ground-Truth Models use manually specified, verified components, but are not an optimal-policy oracle; generated models can sometimes achieve higher realized reward.The downstream planner remains approximate; the baseline uses a POMDP Coder-like planning stage with handcrafted models.Source: §D.3 · pp. 17–18

D.4 Environment descriptions and natural-language sweep

• Main experiments use the strongest environment description, L3; the L0–L3 sweep varies prior information from no description to the full task description.The sweep measures sensitivity to the environment prior and increasingly explicit information about the environment and preferred POMDP structure.Source: §D.4 · p. 18

CORNERS

• CORNERS descriptions range from none (L0) to a 10x10 grid (L1), corner goal and interior start (L2), then randomized corner goal, start, and direction with no obstacles (L3).At L3, the start and goal corner vary between episodes.Source: CORNERS · p. 18
• Lava descriptions progress from none (L0) to a 10x10 grid with lava (L1), then a one-gap lava barrier and fixed start and goal (L2), with barrier and gap positions varying at L3.The agent starts at (1, 1) and the goal is at the bottom-right corner.Source: CORNERS · p. 18

FOURROOMS

• FOURROOMS progresses from no description (L0) to a 19x19 grid (L1), then four doorway-connected rooms with agent and goal in different rooms; L3 randomizes their positions and agent direction while fixing layout.The L3 agent and goal are placed at random positions in different rooms.Source: FOURROOMS · p. 18

UNLOCK

• UNLOCK progresses from no description (L0) to a locked-door grid (L1), then a key-accessible section and goal behind the door; L3 randomizes agent, key, goal, and direction while fixing layout.The key is in the accessible section, and the goal lies behind the locked door.Source: UNLOCK · p. 18

E General details on environments

• Pinductor sees only a 3×3 partial field of view, while the main baseline [14] receives post-hoc access to the full underlying state.The comparison replays the same recorded Pinductor trajectory under both observation models; previously visited cells remain visible and unvisited cells are masked for Pinductor.Source: E General details on environments · p. 19
• In Corners, a randomly placed corner goal in a rotationally near-symmetric empty 10×10 room makes position disambiguation difficult under the small field of view.Observations from different corners are nearly identical, so the agent must reason about which corner contains the goal while navigating.Source: E General details on environments · p. 19
• In Lava, the model must identify lava as a terminating, zero-reward hazard and use language and trajectory cues because reward is sparse.Otherwise, the planner may take an apparent shortcut and terminate on lava.Source: E General details on environments · p. 19
• Four Rooms tests long-horizon navigation through narrow doorways to a goal whose position varies across episodes, with each doorway choice made under belief about goal location.The 3×3 field of view leaves the agent unable to see the goal when it must commit to traversing particular doorways.Source: E General details on environments · p. 19
• Unlock tests compositional planning: the model must represent carrying, door-state changes after unlocking, and key-door colour matching from observation traces.The required sequence is to locate and pick up the key, unlock and traverse the door, then reach the fixed goal behind it.Source: E General details on environments · p. 20

F.1 Tabular Baseline

• The tabular baseline separately estimates initial-state frequencies, transition frequencies, and mean rewards and termination probabilities for state-action-next-state triples.At inference, it samples initial states and transitions proportionally to their observed frequencies, and samples termination as a Bernoulli outcome.Source: §F.1 · p. 20
• The tabular observation model samples observations by their frequencies after state-action pairs; unseen conditions use silent defaults, and the method requires privileged hidden-state access.These frequency tables are learned from the offline dataset.Source: §F.1 · p. 20

Performance analysis as a function of LLM choice

• LLM capability shows a threshold-like effect: weaker models often fail under sparse rewards, while stronger models reach comparable higher performance on LAVA and UNLOCK.Table 1 varies only the LLM used to propose model code, reporting mean episode reward and win rate across the two tasks.Source: G Ablations and Additional Results · p. 21, Table 1 · p. 21

Robustness to stochastic environments

• Under stochastic dynamics, Pinductor remains comparable to the state-access LLM baseline, suggesting observation-only induction does not cause a disproportionate performance loss.Performance falls relative to deterministic versions, but belief-based evaluation still guides model induction when observations and transitions are less predictable.Source: G Ablations and Additional Results · p. 21, Figure 10 · p. 21

Prompt-information ablation

• The effect of prompt information is task dependent: some environments are partly recovered from priors and trajectories, while others improve when prompts provide structural cues.The ablation varies prompt detail from none (L0) through minimal grid-size information (L1) and layout and goal information (L2) to the full description (L3).Source: G Ablations and Additional Results · p. 22, Figure 11 · p. 22

Semantic-information ablation

• Replacing meaningful environment and object names with less informative labels reduces performance across environments, showing that semantic cues help form and refine model hypotheses.This supports a mechanism in which language-model priors propose structured hypotheses and trajectory-based filtering evaluates and corrects them.Source: G Ablations and Additional Results · p. 22, Figure 12 · p. 22

H.1 Computational Resources

• Runtime at N = 10 demonstrations is reported as median [IQR] minutes across seeds because inference traces can produce heavy-tailed outliers.Source: §H.1 · p. 23, Table 2 · p. 23
• Experiments ran on one local CPU workstation without GPU acceleration, with LLM calls through OpenRouter, so wall-clock time includes remote API latency.Source: §H.1 · p. 23, Table 3 · p. 23

H.2 Randomness Control

• Each run seeds Python, NumPy, and Torch; LLM queries also receive the run seed, making sampling reproducible to the extent allowed by the provider.Source: §H.2 · p. 23
• Evaluation episodes use deterministic Python and NumPy seeds and matching environment resets, giving methods the same initial instance for each run-seed and episode pair.Source: §H.2 · p. 23
• The authors log the served backend and model identity to support post-hoc diagnosis of LLM-based runs.Source: §H.2 · p. 23
• Reward and belief error bars use 95% percentile bootstrap confidence intervals over 10 seed-level means, first averaging the three episodes within each seed.The procedure resamples the 10 seed-level estimates and reports the 2.5th and 97.5th percentiles.Source: §H.2 · p. 23
• Bootstrap intervals avoid normality assumptions for bounded metrics near 0 or 1, where Gaussian intervals can mislead with only 10 independent seeds.Source: §H.2 · p. 23

H.3 Licenses and Third-Party Assets

• MiniGrid is used under Apache 2.0; Claude and Qwen APIs follow their commercial terms, and Curtis et al. [14] code is reused with citation despite no explicit repository license.Source: §H.3 · p. 24

I Details on prompting methods

• Pinductor and Curtis et al. both use offline and online generate–evaluate–refine loops, but differ in state access, model-call granularity, and refinement feedback.Pinductor uses observation–action–reward trajectories, one joint call for (ρ0, T, O, R), and particle-filter scores with per-step disagreement diagnostics; Curtis et al. use privileged state tuples, four component calls, and state-level prediction disagreements. Pinductor receives the state schema but not realized state values.Source: I Details on prompting methods · p. 24
• Curtis et al.’s online refinement hook is skipped on the deterministic MiniGrid benchmark when new-episode coverage does not strictly degrade.Thus its displayed online example reuses an iteration-1 refinement call; Pinductor’s example is a true post-episode-0 call.Source: I Details on prompting methods · p. 24

I.1 Initial model proposal (offline, iter 0, LAVA)

• The initial-proposal prompt describes a 10x10 lava grid whose lava-column and gap positions vary across episodes, and asks the LLM to model its dynamics from partial observations.It emphasizes that observations are successive views of a persistent world, with the agent able to mask otherwise stable cells.Source: §I.1 · pp. 24–25
• The prompt asks the LLM to infer a plausible hidden initial-state distribution from the environment description and each episode’s first observation.The prompt states that true initial states are not directly observed.Source: §I.1 · p. 25
• The prompt distinguishes perceptual fields from outcomes: reward and termination are transition outcomes, while absolute position and the full grid are not observations.The observation API includes the agent’s field of view and may include direction or carrying information when represented in the underlying state.Source: §I.1 · pp. 26–27
• The proposal instructions require a general world model that favors semantic rules over memorization, covers plausible cases, stays parsimonious, and avoids episode- or step-specific rules.When samples disagree, they ask for one coherent, general hypothesis rather than stacked special cases.Source: §I.1 · p. 29
• The inferred LAVA environment is a 10 × 10 grid with boundary walls, mostly empty interior, and a lava column containing one passable gap.The lava column and gap row are randomized at the start of each episode.Source: §I.1 · p. 29
• The agent starts at (1, 1), while the goal is at (8, 8), where entry yields reward +1 and ends the episode.The proposed implementation samples the lava column and gap when initializing the state.Source: §I.1 · pp. 29, 31
• Forward movement advances one cell unless blocked by a wall, boundary, or lava; turning changes direction, while pickup, toggle, and done have no effect in this environment.The implementation returns the unchanged state for the listed no-effect actions.Source: §I.1 · pp. 29, 32
• The observation is a 3 × 3 field of view centered on the agent and rotated to face forward, with the agent visible at image[1, 2].The model’s observation structure includes the image, agent direction, and carried-object field.Source: §I.1 · pp. 29–30
• The reward logic returns 1.0 and ends the episode when next_state.agent_pos differs from state.agent_pos; otherwise, it returns 0.0 and leaves the episode active.Source: §I.1 · p. 32
• The prompt asks for runnable code that models next states conditioned on states and actions, using only a few supplied experiences from a larger environment distribution.It also asks for a reward function following the provided template.Source: §I.1 · pp. 32, 34
• The initial prompt asks the LLM to explain next-state distributions in English and implement the transition function without overfitting to the samples.It requires the function to follow the supplied template, with helper functions scoped inside it and no new classes or imports.Source: §I.1 · p. 36
• The proposed LAVA dynamics are deterministic: turns change direction, forward movement is blocked by boundaries, walls, or lava, and other actions leave the state unchanged.The grid and inventory remain static across transitions.Source: §I.1 · p. 36

I.2 Refinement iteration (UNLOCK for Pinductor online; LAVA iter-1 for Curtis et al.)

• The refinement example is Pinductor’s post-episode-0 online refit with joint particle-filter feedback; it uses an 11x6 two-room grid whose layout stays fixed while agent start, key, and goal positions vary between episodes.The rooms are connected through a locked door at column 5; the key is in the accessible left room, the goal is behind the door in the right room, and the agent starts in the left room with random position and direction.Source: §I.2 · p. 37
• The refinement prompt asks the model to improve initial-state, observation, transition, and reward distributions using action-observation samples while the true state remains hidden.The prompt directs the model to use the observed data to check its understanding of the world dynamics.Source: §I.2 · p. 37
• The observation interface consists of a 3x3 forward image, agent direction, and carried object, while absolute position and the full grid are unavailable.The prompt states that code accessing obs.agent_pos will crash.Source: §I.2 · p. 39
• Reward and termination are transition outcomes, not observation fields; the observation function should reproduce perceptual fields and may set direction or carrying when present in the underlying state.The prompt notes that reward and termination appear alongside each observation in the recorded data, but are not perceptual signals for obs_func.Source: §I.2 · p. 39
• -18.86 was the candidate model’s Overall PF score, where 0 indicates perfect prediction and more-negative scores indicate worse prediction.The evaluation reports an Overall PF score of -18.86.Source: §I.2 · p. 43
• The Median particle distance was 3.011, compared with a Best particle distance of 0.000, indicating that one state explained observations better than the particle belief overall.The evaluation also reports a distance variance of 3.067; the passage interprets a large best-to-median gap as indicating transition or inference problems.Source: §I.2 · p. 43
• Mean effective sample size was 324.90, while the minimum ESS was 1.54; low ESS indicates particle degeneracy and may signal model mismatch or too few particles.The passage states that the maximum possible ESS is the number of particles plus rejuvenation.Source: §I.2 · p. 43
• Prediction errors occurred in contexts where observations changed between steps, which the evaluation guidance associates with transition rules mishandling state changes or agent pose.Reported poor-context average scores include -2.1026, -1.4617, and -1.4572; each is listed for a context with n=1.Source: §I.2 · p. 43
• Candidate models disagreed on 2/10 particles for one reported field of view and on 1/10 particles for four others, while agreeing on 8/10 or 9/10, respectively.Source: §I.2 · p. 43
• The refinement prompt asks for plain-English error analysis followed by function updates, favoring semantic rules, broad case coverage, parsimony, and one coherent hypothesis over memorization.It also says not to use episode or step references and to retain cases that already worked.Source: §I.2 · p. 44
• The diagnosis identifies stale objects in initialization, a missing partition wall and boundary setup, inconsistent position types, and incomplete forward-movement blocking as issues to fix.The suggested movement rules allow passage through empty cells, goals, open doors, and keys while blocking walls and closed or locked doors.Source: §I.2 · pp. 44–45
• The earlier implementation addressed grid initialization, wall and door placement, position-type consistency, reward bounds checks, and avoiding object placement on the agent.These were listed as key fixes to the prior implementation.Source: §I.2 · p. 46
• The example environment is a 10×10 grid with boundary walls, a lava barrier with one gap, and lava-column and gap positions that vary across episodes.The agent starts at (1,1), and the goal is at the bottom-right corner.Source: §I.2 · p. 46
• The refinement prompt presents a partially correct model and counterexamples, then asks the model to explain the transition distribution and improve its code.The examples include transitions impossible under the current model.Source: §I.2 · pp. 46, 50
• The prompt requires an implemented transition function that runs on a state-action pair and returns a sampled next state.It also requests a natural-language account of the modeled next-state distribution conditioned on prior state and action.Source: §I.2 · p. 50
• The prompt prohibits examples, new or rewritten classes, and new imports, and asks for generalized ranges rather than indices fitted to demonstrations.The transition function must be returned in a Python code block.Source: §I.2 · p. 50
• The model corrected its transition rule after examples showed lava traversal, making only structural walls impassable.It recommended removing the lava check from walkability.Source: §I.2 · pp. 50–51