HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy

Zengjue Chen; Peidong Liu; Jiawei Li; Qi Wang

Source-linked AI summary

HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy

Zengjue Chen, Peidong Liu, Jiawei Li, Qi Wang

arXiv:2609.09941v1Sep 9, 2026cs.RO

TL;DR

World-model hallucinations can bias VLA policy optimization, while real-robot reinforcement learning is costly, sample-inefficient, and potentially unsafe. HaWMPO estimates rollout reliability with an action-conditioned hallucination-aware model and uses Reward-Soft in GRPO to reduce unreliable trajectories’ influence, consistently outperforming existing baselines across LIBERO task suites.

Problem
World-model hallucinations can introduce biased state transitions that mislead VLA policy optimization, while real-robot reinforcement learning requires costly physical trial and error and may pose hardware and safety risks.
Method
HaWMPO uses an action-conditioned hallucination-aware model to estimate rollout reliability and incorporates hallucination scores into GRPO through a Reward-Soft mechanism.
Results
The proposed framework consistently outperforms existing baselines across multiple LIBERO task suites.
Takeaways & Limitations
Explicitly modeling hallucinations and reducing unreliable synthetic trajectories’ influence improves the stability and effectiveness of world-model-based reinforcement learning.
Takeaways & Limitations
The framework is mainly evaluated on relatively short-horizon LIBERO manipulation tasks, so its scalability to more complex long-horizon real-world scenarios remains underexplored.

Abstract

from arXiv · show

Generalist robot policies have demonstrated strong generalization across robotic manipulation tasks, yet their success rates remain limited in com- plex long-horizon scenarios. Recent methods improve Visual-Language-Action (VLA) policies through online reinforcement learning on real robots, but such training relies on costly physical interactions, suffers from low sample efficiency, and may introduce hardware and safety risks. World models offer a promising alternative by enabling policy optimization with imagined rollouts. However, long-horizon rollouts generated by world models often suffer from prediction hal- lucinations, producing biased state transitions that can mislead policy learning. To address this issue, we propose Hallucination-aware World Model-based Pol- icy Optimization (HaWMPO), a closed-loop reinforcement learning pipeline for VLA policy post-training with world models. Specifically, HaWMPO introduces an action-conditioned hallucination-aware model to estimate the reliability of gen- erated image sequences, and incorporates hallucination scores into group relative policy optimization through a Reward-Soft mechanism, suppressing unreliable ac- tion chunks during training. On the LIBERO benchmark, HaWMPO achieves the best average success rate, with gains of 15.0% over the base model and 2.8% over the strongest baseline; real-world experiments on a G1 robot further validate its effectiveness, raising the average success rate on two manipulation tasks from 67.5% to 80.0%.

1 Introduction

HaWMPO addresses costly and risky real-robot reinforcement learning by filtering unreliable world-model rollouts before policy updates. Its closed-loop pipeline estimates hallucination reliability and uses Reward-Soft within GRPO to stabilize optimization.

Real-robot online reinforcement learning remains costly, sample-inefficient, and potentially hazardous because it requires extensive physical trial and error.
World-model hallucinations can create biased state transitions that mislead policy optimization and weaken reinforcement-learning stability and policy performance.
HaWMPO combines an action-conditioned world model, hallucination-aware model, and reward model for closed-loop VLA policy optimization.
The hallucination-aware model quantifies the reliability of world-model-predicted image sequences under specified VLA action inputs using continuous hallucination scores.
Reward-Soft integrates hallucination scores into GRPO to reduce unreliable action chunks’ influence on reward shaping and policy updates.

2 Related Work

Related work applies reinforcement learning and world models to improve VLA policies while reducing reliance on offline demonstrations and real-world interaction. These approaches address sparse rewards, distribution shifts, and virtual experience generation.

Reinforcement learning is increasingly used to overcome imitation learning’s limitations on out-of-distribution cases, long-horizon tasks, and sparse rewards.
World models support VLA pre-training, post-training, and reinforcement learning by simulating environment distributions and physical dynamics.

3 Methodology

HaWMPO optimizes VLA policies in imagined world-model rollouts while estimating hallucination reliability and modulating rewards to reduce misleading updates.

3.1 Hallucination-Aware Model for World Model: HAM predicts a continuous chunk-level hallucination score from an eight-frame generated video chunk, its conditioning actions, current observation, and initial anchor image.Higher scores indicate less reliable predictions; observed future sequences are needed only to construct training targets.
3.1 Hallucination-Aware Model for World Model: HAM is trained from successful and failed trajectories using visual similarity, depth consistency, optical-flow trajectory consistency, and image-quality metrics.The composite target is a proxy for physical validity rather than a guarantee.
3.2 Reward-Soft Mechanism for Hallucination-Aware RL: Reward-Soft averages hallucination scores within each action chunk and transforms the raw reward as ˆR = (1 − α · H) · R.The hallucination score H, raw reward R, and tunable α determine the modulated reward ˆR.
3.3 Hallucination-Aware World Model-Based Policy Optimization: The training loop performs parallel synthetic rollouts, estimates chunk hallucination, computes modulated rewards, normalizes group-wise advantages, and updates the VLA with GRPO.The described implementation uses 8 parallel imaginative environments.
HaWMPO integrates a VLA policy, action-conditioned world model, hallucination-aware model, reward model, and Reward-Soft-enhanced GRPO for online policy optimization.The pipeline uses imagined environments rather than frequent real-world robot interactions.

4 Experiments

Experiments evaluate HaWMPO in LIBERO simulation, real-world G1 manipulation, hallucination detection, and ablations of Reward-Soft and HAM. HaWMPO achieves the strongest reported average success rates while showing task-dependent penalty preferences and benefits from hallucination-aware training.

4.2 Simulation Experiments: 63.7% average LIBERO success rate is 15.0 points above OpenVLA-OFT, 6.9 above WMPO, and 2.8 above WoVR*, the strongest baseline.HaWMPO also leads on Spatial at 77.2% and Object at 52.2%, while remaining comparable to WoVR* on Goal.
4.3 Real-World Experiments: 80.0% average success across two G1 manipulation tasks exceeds the base policy by 12.5 points and WoVR by 7.5 points.HaWMPO reaches 85% on Tissue-to-Box and 75% on Headphone-on-Stand using 20 trials per method per task.
4.4 Analysis of the Reward-Soft Mechanism: α = 0.3 performs best on Object and Spatial, whereas Goal benefits from α = 1 because its rollouts accumulate more severe hallucinations.At 200 steps, α = 0.3 yields 52.0% on Object and 77.2% on Spatial, while α = 1 yields 66.2% on Goal.
4.5 Analysis of the Hallucination-Aware Model: HAM achieves AUROC 0.9375 and AP 0.8952 on 50 annotated LIBERO chunks, providing a hallucination signal without observed future frames during imagined rollouts.The comparison is descriptive, and statistical significance of differences was not established.
4.6 Ablation Study: Removing HAM reduces performance on Spatial and Object, with the Spatial advantage of the full model remaining 8.0 points at 200 training steps.On Goal, α = 1 reaches 66.2% for the full method versus 64.0% for the ablated variant, showing the effect depends on penalty selection.

5 Conclusion

HaWMPO optimizes VLA policies with a hallucination-aware world-model reinforcement-learning pipeline. By downweighting unreliable synthetic trajectories through HAM and Reward-Soft, it improves the stability and effectiveness of world-model-based policy optimization across LIBERO task suites.

5 Conclusion: HaWMPO combines an action-conditioned hallucination-aware model with GRPO and Reward-Soft to reduce unreliable trajectories' influence during VLA policy optimization.The framework explicitly estimates rollout reliability and incorporates predicted hallucination scores into training.

6 Limitations

The work identifies limitations in its hallucination-aware model and evaluation scope. Its supervision may miss some hallucinations, while scalability to complex long-horizon settings and end-to-end adaptation remain underexplored.

The hallucination-aware model relies on manually designed feature, depth, and trajectory signals that may not capture every hallucination type.
Evaluation mainly covers relatively short-horizon LIBERO manipulation tasks, leaving scalability to complex long-horizon real-world scenarios underexplored.
The hallucination-aware model and world model are trained separately, which may limit system adaptability during online reinforcement learning.

A.1 Data Collection

The hallucination-aware model is trained from chunkwise world-model generations on collected real data, using multiple metrics extracted from real and predicted samples.

Chunkwise world-model inference on collected real data constructs the predicted dataset used to train the hallucination-aware model.
Features from DINOv3, Depth-Anything-v3, and trajectory optical flow methods support calculation of four hallucination metrics for each chunk.

A.2 Training Data Analysis

Training-data analysis combines geometric, visual, motion, and semantic metrics into continuous supervision, with composite scores separating quality groups more reliably than image quality alone.

The four metrics measure depth consistency, MUSIQ image quality, optical-flow trajectory consistency, and DINOv3 semantic-appearance similarity.
Samples are divided into Pos top 10%, Unlabelled middle 40%, and Neg bottom 50%, prioritizing motion-related hallucinations over image quality.
Composite correlates most strongly with DINO Similarity at 0.80 and Trajectory Accuracy at 0.72, while image quality has nearly no correlation with the other metrics.
Continuous Composite scores supervise eight-frame predictions to detect abnormal motion, object drift, and semantic inconsistency without relying on superficial sharpness.
Composite scores increase from Neg to Pos with near-complete IQR separation, while image quality differs little and poorly discriminates hallucinations.
PCA assigns 53.2% variance to PC1, driven mainly by trajectory and DINO metrics, and 26.9% to PC2, driven by image quality and depth.

A.3 Baseline

The baseline and simulator setup extend WoVR with an action-conditioned world model for closed-loop imagined rollouts, while physical experiments compare HaWMPO against SFT and WoVR.

A.3 Baseline: The method extends WoVR without changing its world-model structure, omitting WoVR’s orthogonal PACE strategy from the comparison.
A.4 Real-World Experiments: HaWMPO reaches 80.0% average physical success, versus 67.5% for the SFT baseline and 72.5% for WoVR, across two G1 tasks.
A.5 World Model as Simulator: The simulator conditions visual predictions on commanded motion through action-token cross-attention and temporally grouped action embeddings.
A.5 World Model as Simulator: Each simulator step uses five RGB context frames, an eight-step action chunk, and associated action context to predict future visual observations.
A.5 World Model as Simulator: Closed-loop rollouts repeatedly update context with the last four generated frames while keeping the reference frame fixed.
A.5 World Model as Simulator: The world model remains frozen during reinforcement learning, while separate reward and hallucination-aware models evaluate generated observations for Reward-Soft optimization.
A.5 World Model as Simulator: Qualitatively plausible generated frames may still misrepresent contact dynamics, and autoregressive context reuse can accumulate errors.