WorldSample: Closed-loop Real-robot RL with World Modelling

1 Introduction

• IL is limited by demonstration coverage, while RL can improve through trial-and-error beyond demonstrated behaviors.When policies encounter underrepresented states, small errors can compound and lead to failure.Source: §1 · p. 1
• Physical rollouts are slow to collect, may require human reset or intervention, and each reveals only one action-outcome path.Rollouts may also cause hardware wear or unsafe contact.Source: §1 · p. 2
• WorldSample expands real-robot rollouts into multiple physically grounded synthetic trajectories through an action-conditioned world model.The real-synthetic loop uses online rollouts to adapt the model to the task distribution.Source: Figure 1 · p. 2, §1 · p. 2
• PPL regulates synthetic-data use through Q-aware sample selection and uncertainty-guided scheduling.These controls aim to mitigate synthetic noise and prevent generated data from destabilizing learning.Source: §1 · p. 2

2.1 Real World Online RL

• Online RL methods remain sample-inefficient for real-world deployment, while offline data and human interventions still depend on costly interactions and resources.The central bottleneck is the cost of physical trial-and-error training.Source: §2.1 · p. 3

2.2 World Models Based Interaction

• World-model methods optimize policies in generated environments or use models to produce experience, but imagined rollouts can diverge from physical dynamics.Visual hallucination and action-following errors can widen the gap from real interactions.Source: §2.2 · p. 3

3.1 WorldSample Pipeline

• WorldSample couples a real-synthetic data loop with PPL to enlarge the effective training distribution while mitigating value overestimation and hallucination-induced noise.Physical rollouts ground synthetic generation and also provide policy-training data.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Real-Synthetic Data Loop

• Counterfactual trajectory segments are sampled from real rollouts with local action perturbations rather than from a random action prior.The resulting labeled synthetic trajectories combine real observations, sampled actions, model-predicted futures, and reward-model feedback.Source: §3.2 · pp. 3–4
• Asynchronous generation lets robot learning continue collecting rollouts and updating the policy while world-model workers generate synthetic trajectories.Online rollouts also post-train the model, aligning generation with higher-diversity data coverage.Source: §3.2 · p. 4

3.3 Policy-Paced Learning

• Because imperfect world models introduce transition error that can compound through Bellman backups, PPL separates sample-level value balancing from policy-level scheduling.The critic's synthetic-data influence is weighted by a value-balancing term and a training-stage-dependent schedule.Source: §3.3 · pp. 4–5
• Q-aware selection preserves a value-balanced contrast between positive and negative synthetic trajectories, reducing shifts toward uniformly optimistic or conservative critic estimates.Positive trajectories expand rare successes, while negative trajectories constrain visually plausible but unsuccessful behaviors.Source: §3.3 · p. 5
• Uncertainty-guided scheduling suppresses synthetic influence when policy entropy on real states is high and introduces it more gradually as the policy becomes more concentrated.The schedule limits synthetic data from dominating physical experience.Source: §3.3 · pp. 5–6

4.1 Experimental Settings

• WorldSample is tested on five task categories using a Galaxea A1X robot, with comparisons against HIL-SERL, VLAW, and WMPO under shared demonstration and evaluation protocols.The tasks are Pushing, Insertion, Sorting, Pick & Place, and Assemble; each method uses 20 human demonstrations for initialization, while VLAW also uses 50 synthetic demonstrations.Source: §4.1 · p. 6

4.2 Experimental Results

• WorldSample raises average success from 56% to 82% and reduces average training steps from 56K to 23K versus HIL-SERL, with higher success on all five tasks than model-based baselines.Training time falls from 83 minutes to 64 minutes; overall, computational cost decreases 59% and physical interactions 23%.Source: §4.2 · p. 6, Table 1 · p. 7
• WorldSample generally reaches higher success earlier; on harder long-horizon tasks, training-time reductions are task-dependent even as final success improves under lower real-world training cost.The authors attribute the gains primarily to extracting more learning signals from each physical rollout.Source: §4.2 · pp. 6–7, Figure 3 · p. 7
• Online rollout adaptation raises world-model PSNR from 8.501 to 28.428 and SSIM from 0.334 to 0.906, while lowering LPIPS from 0.7432 to 0.0343.Combining demonstrations and rollouts further improves PSNR and SSIM; the dual-view model reaches 29.887 PSNR and 0.925 SSIM.Source: §4.2 · p. 7, Table 2 · p. 8, Figure 4 · p. 8

4.3 Ablation Studies

• On Insertion, full PPL reaches 95% success in 10K steps, versus 86% in 32K steps without PPL.Removing uncertainty-guided scheduling lowers success to 61%; removing Q-aware selection lowers it to 76% and raises intervention rate to 42%.Source: §4.3 · p. 8

5 Conclusion

• WorldSample expands physical rollouts into synthetic transitions through world models while retaining real interaction as the grounding source.PPL regulates generated data through Q-aware selection and uncertainty-guided scheduling.Source: §5 · p. 8
• PPL is intended to enable useful augmentation while mitigating value overestimation and hallucination-induced noise.Its mechanisms are Q-aware selection and uncertainty-guided scheduling.Source: §5 · p. 8
• Experiments on contact-rich and precision manipulation tasks show improved success, faster convergence, and reduced real interaction cost over baselines.Source: §5 · p. 8

• WorldSample focuses on single-task learning within a relatively fixed scene distribution, leaving instance-, task-, and scene-level generalization for future work.Source: §6 · p. 9

• The real-world suite covers pushing, insertion, sorting, pick & place, and assembly tasks with different manipulation demands.The setup uses a 6-DoF Galaxea A1X arm, binary gripper, and wrist and third-person RealSense cameras.Source: Figure 6 · p. 13, §A.1 · p. 13
• Each task starts from 20 human demonstrations and uses a common evaluation protocol, intervention strategy, and performance metrics.Source: §A.1 · p. 13
• Success rate, convergence training time, and online training steps are reported; a rollout succeeds when it reaches the task goal within the episode horizon.Intervention rate measures the fraction of episode execution requiring human correction.Source: §A.1 · p. 13

• All methods use the HIL-SERL actor architecture and the same real-world task suite; they differ in how real and synthetic data improve the policy.Source: §A.2 · p. 13
• HIL-SERL is the real-robot RL baseline, mixing demonstrations, online rollouts, and human interventions for off-policy actor-critic training.It starts each task with 20 demonstrations and retains its original training pipeline.Source: §A.2 · pp. 13–14
• VLAW is adapted as an imitation-learning baseline, training by behavior cloning on 20 real demonstrations and 50 generated successful trajectories.Policies receive 20K behavior-cloning steps before real-world evaluation.Source: §A.2 · p. 14
• WMPO uses world-model-generated rollouts for policy optimization without requiring real-world interaction during training.It is trained for 10K steps on three easier tasks and 20K steps on two harder tasks, stopping at world-model convergence.Source: §A.2 · p. 14

B.1 WorldSample Framework

• The real replay buffer stores online rollouts, while the demonstration buffer stores expert demonstrations and human-guided corrections; WorldSample adds synthetic data without changing the underlying learner or architecture.Source: §B.1 · p. 14

B.2 Counterfactual Trajectory Generation

• Counterfactual action sequences perturb real trajectories locally while preserving temporal structure, fixing inactive dimensions at zero, and projecting actions into the valid range.Experiments use scale jitter s=0.20 and noise σ=0.05; generated actions remain physically close to real behavior.Source: §B.2 · p. 14

B.3 World Model Adaptation

• Online rollouts further adapt the action-conditioned world model to policy-induced states, object appearance, and contact dynamics.Held-out video comparisons show improved fidelity over demonstration-only adaptation, measured with PSNR, SSIM, and LPIPS.Source: §B.3 · pp. 14–15, Figure 7 · p. 15

B.4 Policy-Paced Learning

• Q-aware selection controls the synthetic-buffer mix of successful and failed trajectories using critic Q-values, balancing sparse positive outcomes against overly optimistic values for failures.Source: §B.4 · p. 15

C.1 Uncertainty-Guided Scheduling

• Synthetic data use is scheduled from actor entropy on real states, with its contribution increasing as the policy becomes more concentrated during training.The synthetic ratio is capped at 0.30 to prevent generated data from dominating physical experience.Source: §C.1 · p. 15, §B.4 · p. 15

C.2 Q-Aware Sample Selection

• Balancing successful and failed generated trajectories suppresses overly optimistic value estimates while retaining useful synthetic successes for value propagation.PPL maintains more stable value estimates than naive synthetic-data usage during online training.Source: §C.2 · p. 15, Figure 9 · p. 16
• Actor entropy determines how strongly synthetic trajectories are introduced, preventing early synthetic bias and supporting sample efficiency after the policy stabilizes.The schedule tracks policy uncertainty during training.Source: Figure 8 · p. 16