PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Critic-Guided Synthetic Experience in Latent World Models

Jorge de Freitas, Christos Ziakas, Alessandra Russo

arXiv:2610.39100v1cs.LGcs.AI

TL;DR

Offline policy learning is constrained by costly or unsafe real-world data collection, while errors can accumulate in imagined world-model rollouts. CGSE locally refines offline-policy actions and uses a frozen latent world model and critic to generate synthetic transitions. On three OGBench manipulation tasks, it improves average success by 7.1% relative to offline FQL, without additional real-environment interaction or a test-time planner.

  • Problem

    World models can generate experience, but compounding errors in imagined rollouts limit policy learning in latent space.

    Source: §1 · p. 1

  • Method

    CGSE refines offline-policy action candidates within a trust region, uses a frozen latent world model to generate transitions, and trains the actor with the critic kept fixed.

    Source: §1 · p. 1, §3.1 · p. 3, §3.3 · p. 4

  • Results

    7.1% relative improvement in average task success over offline FQL on three OGBench manipulation tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Zero-step critic-guided action refinement accounts for the majority of the measured improvement, while the method requires no real-environment interaction or test-time planning.

    Source: §1 · p. 2, §6 · p. 6

    Weaker learned dynamics on scene and puzzle-3x3 likely limit the benefit of model-guided generation on those tasks.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We study action refinement methodologies, introducing Critic-Guided Synthetic Experience (CGSE) generation in world models. Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the $Q$-function within a trust region. We then use a frozen latent world model to imagine the future state given each updated action, and use the critic to choose the best one, appending this synthetic transition to the replay buffer. Our method requires no additional real-environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of $7.1\%$ on the average success rate.

1 Introduction

World models can generate synthetic experience, but latent representations are prediction-oriented and imagined-rollout errors can accumulate. CGSE instead uses the model locally to generate transitions for offline policy improvement.

1 Introduction

  • Latent representations are optimized for prediction rather than task completion, and errors accumulate over imagined rollouts, drifting value estimates toward unlikely real-environment states.World models are learned statistical approximations, unlike traditional simulators with explicit rules and underlying dynamics.Source: §1 · p. 1
  • CGSE treats the world model as a controlled generator of additional experience, starting from offline states and optimizing candidate actions within the learned dynamics model.It scores candidates with an offline-trained critic rather than latent-space distance; trust-region guidance keeps optimization local to the data distribution.Source: §1 · pp. 1–2
  • 7.1% relative improvement in average task success over offline FQL on three OGBench manipulation tasks, without additional real-environment interaction or a test-time planner.The method uses a frozen latent world model to generate critic-guided synthetic transitions and shows that keeping value estimates grounded in offline data supports actor-only improvement.Source: §1 · p. 2

2 Preliminaries

The preliminaries define the MDP and action-value objective, then describe the frozen latent model and offline FQL agent used by the method.

2 Preliminaries

  • An MDP is specified by state and action spaces, transition dynamics, reward, and discount; its policy maximizes expected discounted return, which the action-value function encodes for a state-action pair.The method performs policy optimization, transition generation, and value estimation in a learned latent space to avoid planning directly in high-dimensional observations.Source: §2 · p. 2
  • The LeWM model encodes observations as latent states and predicts the next latent conditioned on an action chunk; its encoder and predictor remain frozen and provide a differentiable, one-step deterministic simulator.It is self-supervised on offline play data with latent prediction loss and SIGReg regularization.Source: §2 · p. 2
  • The base agent uses action-chunked, best-of-N Flow Q-Learning with a conditional flow policy and an ensemble critic trained by chunk-level temporal-difference learning on recorded rewards and successors.At deployment, the critic selects the highest-valued of four flow samples; subsequent actor-only fine-tuning uses mixed replay.Source: §2 · p. 2

3 Critic-Guided Synthetic Generation in World Models

CGSE uses an offline critic to refine actions and short model rollouts to select synthetic transitions, rather than using the world model for unconstrained long-horizon simulation. The actor learns from mixed replay while the critic and world model stay fixed.

3 Critic-Guided Synthetic Generation in World Models

  • The method generates synthetic transitions to augment offline policy learning, using the offline critic to improve candidate actions and short model rollouts to expose predicted consequences.Selected behavior is transferred to the actor while value estimates remain grounded in real offline experience.Source: §3 · pp. 2–3

3.1 Critic-guided Action Refinement

  • FMQ-style refinement applies a single normalized critic-gradient displacement within a Euclidean trust region to offline-policy proposals, avoiding iterative action optimization.The short-rollout score is an action-ranking surrogate based on the ensemble-mean Q; it is not an H-step return estimate. Gradients pass through latent predictions, while continuation actions are detached and their sampling noise is fixed during differentiation.Source: §3.1 · p. 3

3.2 Generating Synthetic Transitions

  • Each selected action yields a single-step model successor and transition for replay; multi-step lookahead is reserved for action scoring, although chained generation can accumulate prediction error.A real-state re-anchoring control instead samples an offline latent before every transition.Source: Figure 1 · p. 3, §3.2 · p. 4

3.3 Policy Improvement with Synthetic Experience

  • The fixed actor, critic, and world model generate transitions that are appended to replay, while mixed-batch actor updates leave the critic frozen and exclude synthetic rewards and successors from the actor loss.Generated successors can still become later current states in chained rollouts, affecting the actor’s training distribution; the method adds no real-environment interaction and needs no test-time planner.Source: §3.3 · p. 4, §3.1 · p. 4

4 Experiments

CGSE improves average success over offline FQL under matched training steps, while ablations attribute most of the improvement to critic-guided action refinement. The experiments also show that generation is useful locally, with longer rollout drift motivating single-step stored transitions.

4.1 Setup

  • Experiments use OGBench cube-single, scene, and puzzle-3x3, with shared play data and frozen LeWM representations; CGSE fine-tunes an offline checkpoint for 500,000 steps.The from-scratch offline FQL baseline receives 1M updates, matching the total gradient steps across methods.Source: §4.1 · p. 4

4.2 Results

  • 7.1% relative improvement in average success rate over offline FQL, with CGSE evaluated against a same-gradient-step baseline.On cube-single, success rises from 84.0% to 90.7%; scene and puzzle-3x3 gains are within baseline seed variability.Source: §4.2 · p. 5, Table 1 · p. 5
  • Critic-guided refinement accounts for most of the improvement; the experiments do not establish a significant additional benefit from model-based lookahead.CGSE improves over the no-world-model variant by 1.8% relative, with a 95% paired interval of [−2.1, 5.6] points.Source: §4.2 · p. 5

4.3 Impact of Locality and Critic Grounding

  • Latent prediction drift rises from 3.40 at one model step to 6.10 at two, motivating single-step stored transitions and H=2 only for action scoring.Generated successors remain close to offline latents, with nearest-neighbour L2 ≈0.62 and norms on the same 7.8±1.2 shell as real states.Source: §4.3 · p. 5, Table 3 · p. 6
  • On cube, unrefined policy samples reach 80.4±3.2% final success, best-of-N reaches 86.3±3.1%, and CGSE reaches 90.7 ± 0.9%.When generated transitions also update the critic, success falls to 30.7 ± 25.5%; the comparison changes several factors together and does not isolate their effects.Source: §4.3 · pp. 5–6, Table 2 · p. 5

5 Related work

The paper situates its approach among model-based RL and offline RL, where learned models augment learning and critics guide policy actions. It uses a frozen latent world model as a synthetic-experience source rather than a test-time planner.

Model-based RL

  • Model-based RL uses learned-dynamics rollouts to augment policy or value learning, while offline variants constrain model use in uncertain regions.Latent world models have also been used for planning and control; this work instead uses one as a source of synthetic experience.Source: §5 · p. 6

Offline RL

  • Offline policy steering includes behavioral regularization, pessimistic values, in-sample maximization, and critic-guided diffusion or flow policies.The paper builds on flow Q-learning with action chunking and FMQ, refining flow-policy proposals and scoring them through latent rollouts.Source: §5 · p. 6

6 Conclusion

The conclusion presents CGSE as local, critic-guided synthetic interaction that improves average task success without real-environment interaction or test-time planning. Its gains concentrate on cube-single, while weaker dynamics and frozen visual representations constrain multi-object performance.

6 Conclusion

  • CGSE-generated transitions improve average task success by 7.1% relative without real-environment interaction or test-time planning.Gains are concentrated on cube-single.Source: §6 · p. 6
  • Zero-step critic-guided action refinement accounts for the majority of the improvement in matched controls.Source: §6 · p. 6
  • Weaker action-contingent dynamics and frozen visual representations bottleneck performance on multi-object tasks like scene and puzzle-3x3.The conclusion calls for broader environment evaluation, stronger world models, compute-budget scaling studies, and task-aware representation learning.Source: §6 · p. 6

A Implementation details

Implementation uses a frozen LeWM latent model and an offline FQL actor-critic, then generates synthetic transitions while training the actor. The replay buffer mixes offline and generated data, with CGSE using 32 proposals and H=2 lookahead.

A Implementation details

  • The frozen world model uses a ViT-Tiny encoder to R192 and a Transformer latent predictor trained on OGBench play data.Each model step spans five primitive actions; training uses latent prediction loss and SIGReg regularization.Source: A Implementation details · p. 9
  • CGSE uses FMQ-style refinement with N=32 proposals, look-ahead H=2, Q-at-every-step scoring, and trust-region radius η=0.02.Synthetic generation and actor training run for 500,000 steps with batch size 256 and learning rate 10−4.Source: A Implementation details · p. 9
  • The replay buffer begins with 200k offline chunk transitions and appends one generated transition per update, reaching an expected generated fraction of 71.4% at 500k updates.Its 2M capacity avoids eviction during these runs.Source: A Implementation details · p. 9
  • Selector timing uses an NVIDIA A100 GPU and excludes compilation, rendering, and policy-update time, measuring generation cost rather than total training time.Source: A Implementation details · p. 9

B Imagined trajectory restarts

Chained imagined trajectories terminate when they reach a latent goal-distance threshold or 40 model steps, then restart from an offline latent. The threshold affects termination only; real-state re-anchoring avoids needing one.

B Imagined trajectory restarts

  • Chained trajectories end when the latent is within distance 2.0 of the goal or after 40 model steps, then restart from an offline latent.The distance check controls trajectory termination only.Source: B Imagined trajectory restarts · p. 9
  • Action scoring and actor training require no synthetic reward, and the distance threshold’s sensitivity has not been tested.Source: B Imagined trajectory restarts · p. 9
  • Real-state re-anchoring samples an offline latent before every transition and therefore requires no goal threshold.Source: B Imagined trajectory restarts · p. 9

Related papers

Loading 2610.39100v1…