Dyna-Mind: Learning to Simulate from Experience for Better AI Agents

• Long-horizon tasks such as web navigation and computer use require accurate world models that capture complex environment structure and dynamics.Source: §1 · p. 1
• Human-inspired vicarious trial and error mentally evaluates possible futures before action, motivating simulation as a capability for adaptive long-horizon decision-making.Source: §1 · p. 1
• Strong reasoning models can simulate structured environments like Sokoban but show sharply lower simulation accuracy and task success in more complex domains such as ALFWorld.Source: §1 · p. 2
• Dyna-Mind trains agents from real environment experience using ReSim search-tree reasoning traces in stage 1 and Dyna-GRPO online reinforcement learning with outcome rewards and intermediate states in stage 2.Source: §1 · p. 2
• Across Sokoban, ALFWorld, and AndroidWorld, ReSim teaches simulation and Dyna-GRPO learns better policies for long-horizon, planning-intensive tasks.Source: §1 · p. 2

• Reactive agents act from immediate observations, while search-based and hierarchical methods improve long-horizon tasks at the cost of extra environment interactions or complex coordination.Source: §2 · p. 2
• Training methods have progressed from supervised learning with human or synthesized data toward reinforcement learning with verifiable rewards, while the essential reasoning for long-horizon tasks remains unclear.Source: §2 · pp. 2–3
• Dyna-style methods train world models from real rollouts, perform simulated rollouts, and update policies using real and simulated data, but can require modular systems or strong reasoning models.Source: §2 · p. 3
• Dyna-Mind instead integrates simulation into a single (V)LM agent’s reasoning through training, without relying on strong reasoning models.Source: §2 · p. 3

3 DYNA-MIND

• Dyna-Mind addresses how to synergize world simulation with reasoning and improve simulation ability for long-horizon task performance.Source: §3 · p. 3

3.2 REASONING WITH SIMULATIONS (RESIM)

• RESIM builds expanded search trees from real environment interactions, evaluates partial rollouts, and aggregates them into a single simulation-guided reasoning trace for training.It uses rollout models, value estimates, and a (V)LM aggregator before executing the selected action.Source: §3.2 · p. 4
• RESIM then uses its search-tree-derived responses as supervised targets so agents generate simulation-guided reasoning directly from the current state and limited history.Source: §3.2 · p. 4

3.3 DYNA-GRPO

• RESIM is computationally expensive because constructing its training data requires a rollout model, value function, and aggregation model.This motivates the online DYNA-GRPO stage.Source: §3.3 · p. 4
• DYNA-GRPO uses SIMROLLOUT to execute planned actions, collect real future states, and prompt refined responses that improve simulation during reinforcement-learning rollouts.Source: §3.3 · p. 5
• DYNA-GRPO alternates simulation improvement with direct policy improvement, combining refined and standard rollouts under GRPO so simulation capability is integrated into ordinary decision-making.Simulation refinement rewards responses that solve tasks and improve over standard rollout rewards.Source: Figure 3 · p. 5, §3.3 · p. 6

4.1.1 MAIN RESULTS

• DYNA-GRPO reaches 77.1% on Sokoban and 90.8% on ALFWorld, outperforming RLOO and GRPO while improving task performance through online reinforcement learning.The comparison is reported for DISTILL(RESIM) plus each stage-2 RL method in Table 1.Source: §4.1.1 · p. 8
• ReSim achieves 96.4% success on Sokoban and 87.7% on ALFWorld, outperforming other methods by using ground-truth simulations from search trees.DeepSeek-R1 is near-perfect on Sokoban but struggles to model ALFWorld, whereas ReSim performs strongly in both environments.Source: §4.1 · p. 6, §4.1.1 · p. 7
• DISTILL(RESIM) matches DISTILL(R1) on Sokoban, outperforms DISTILL(V3) and DISTILL(R1) on ALFWorld, and generates 11x fewer tokens than DISTILL(R1).The distilled traces consist almost entirely of planning through simulation.Source: §4.1.1 · p. 7

4.1.2 MEASURING SIMULATION ABILITY

• Table 2 reports success rate, Sim Score, and their correlation, showing that DISTILL(RESIM) and DYNA-GRPO improve simulation ability alongside task performance over their baselines.ReSim maintains strong success rates across Sokoban and ALFWorld, while DeepSeek-R1 struggles in ALFWorld because it has difficulty modeling the environment layout.Source: §4.1.2 · p. 8, Table 2 · p. 8

4.2 ANDROIDWORLD

• AndroidWorld evaluation uses 20 Android apps, 116 parameterized task types, screenshot-only inputs, and a maximum of 15 steps, but ReSim remains limited by rollout-model weaknesses.The rollout model struggles with some GUI interfaces and button functions and cannot recover after multiple mistakes.Source: §4.2 · pp. 8–9, §4.2.1 · p. 9

4.2.1 MAIN RESULTS

• On AndroidWorld, ReSim improves inference performance, and both training stages outperform baselines on in-distribution and out-of-distribution tests, including Qwen2.5-VL-72B and ReSim.The results transfer to Qwen2.5-VL-7B and 32B through DISTILL(RESIM).Source: §4.2.1 · p. 9, Table 3 · p. 9

5 CONCLUSION

• RESIM teaches agents to simulate future states, while DYNA-GRPO further refines reasoning and simulation through online RL using outcome rewards and intermediate interactions.Together, the methods are reported to improve policies for long-horizon, planning-intensive tasks across three benchmarks.Source: §5 · p. 10

B ETHICS STATEMENT

• Dyna-Mind agents could be misused for tasks such as scamming or disseminating false information, motivating safety filters and other guardrails.The paper does not condone unlawful or morally unjust uses of the framework.Source: B ETHICS STATEMENT · p. 15

C ADDITIONAL ALGORITHMIC DETAILS

• SIMROLLOUT refines a response by executing its extracted plan, observing actual future states, and prompting the model to produce an improved response.The refined response becomes the next action, and the process repeats until completion or a step limit.Source: C ADDITIONAL ALGORITHMIC DETAILS · p. 15
• Algorithm 2 operationalizes simulation refinement by storing refined state-action trajectories for policy learning and refining simulations before transitioning with the refined action.The procedure repeats for multiple episodes while tasks remain unfinished and below the maximum step count.Source: C ADDITIONAL ALGORITHMIC DETAILS · p. 15

D.1 EXAMPLE TASKS AND ACTIONS

• Sokoban requires pushing boxes to destinations while avoiding obstacles in a grid world with up, down, left, and right actions.Successful play requires spatial planning to avoid deadlock situations.Source: §D.1 · p. 15
• SIMROLLOUT achieves better performance than REACT on Sokoban and ALFWorld when models receive ground-truth next-state information.Table A1 directly compares simulation refinement with direct prompting across the two environments.Source: Table A1 · p. 16
• ALFWorld is a text-based embodied environment where agents locate and interact with objects to complete household tasks using natural-language instructions.Its valid actions depend on the current state.Source: §D.1 · p. 16

D.2 RESIM IMPLEMENTATION DETAILS

• For text games, RESIM uses DeepSeek-V3 to generate 256 rollouts, fine-tunes Qwen2.5-32B-Instruct on one correct rollout per task, and trains a value function from repeated policy rollouts.The value estimate is the discounted probability of successful completion from each state, with γ = 0.95 in both environments.Source: §D.2 · p. 16
• For complex environments such as AndroidWorld, the method directly prompts pretrained Qwen2.5-VL-72B and GPT-4o as rollout and value function models.Source: §D.2 · p. 16

D.3 SIMULATION REFINEMENT PERFORMANCE

• All evaluated models improve task success when provided with next-state information, while GPT-4o and GPT-4.1 improve more than Qwen2.5-7B-Instruct.The authors attribute weaker-model limitations to the difficulty of correcting mistakes through non-trivial reasoning.Source: §D.3 · p. 16
• The results identify world-modeling error, especially in tasks such as ALFWorld, as a significant bottleneck for long-horizon reasoning.This conclusion follows from the observed benefit of supplying future-state information during action selection.Source: §D.3 · p. 16

D.4 ADDITIONAL TRAINING DETAILS

• DYNA-GRPO continues training the best stage-1 model with identical applicable hyperparameters across RLOO, GRPO, and DYNA-GRPO.Training uses batches of 8 tasks, group size G = 8, learning rate 1e-6, and 300 steps for Sokoban and ALFWorld.Source: §D.4 · p. 16
• The DYNA-GRPO rollout procedure samples multiple policy rollouts from the current state, aggregates them into a reasoning response, and advances the environment with the selected ReSim action.The procedure repeats until termination or tmax, storing each state-action transition in the resulting trajectory.Source: §D.4 · p. 17
• Sokoban uses nT = 10 and nπ = 10, whereas ALFWorld uses nT = 10 and nπ = 20; both settings train on Qwen2.5-7B with 8xH100.Source: §D.4 · p. 17

D.5 SIMULATION SCORE PROMPTS

• Simulation scoring extracts an agent’s action plan and imagined next states, executes the plan to obtain ground-truth states, and uses an LLM judge to compare them.The per-turn scores lie in [0, 1] and are averaged across trajectory turns to produce an overall simulation score.Source: §D.5 · p. 17

E.1 EXAMPLE TASK AND ACTIONS IN ANDROIDWORLD

• AndroidWorld examples pair a task screenshot s_t with a model-generated action a_t, including a click rendered at the next state for clarity.The example uses the dockerized AndroidWorld environment and screenshot-only modality.Source: §E.1 · p. 17

E.2 ADDITIONAL TRAINING DETAILS

• AndroidWorld training and evaluation use its dockerized environment with an adapted action space.Source: §E.2 · p. 17
• AndroidWorld DYNA-GRPO training uses 60 RL steps, nT = 2, nπ = 8, batch size 8, group size G = 8, and learning rate 1e-6.The reduced training budget reflects AndroidWorld’s high computational and time costs.Source: §E.2 · p. 17
• The AndroidWorld example visualizes a screenshot input and a generated click action, with the action rendered in green at the next state.Source: Figure A1 · p. 18
• Training uses Qwen2.5-VL-7B-Instruct and Qwen2.5-VL-32B-Instruct, labeled DISTILL-7B and DISTILL-32B, respectively, with 16xH100.Source: §E.2 · p. 18

E.3 OTHER IMPLEMENTATION/EVALUATION DETAILS

• AndroidWorld experiments focus on Qwen2.5-VL because the authors could not fully reproduce reported UI-Tars performance or find compatible evaluation scripts.The end-to-end SFT + RL setup requires VLMs that directly interact with Android’s GUI.Source: §E.3 · p. 18
• The appendix includes a SIMROLLOUT prompt that refines an agent response using actual next-state information obtained by executing its extracted plan.Source: Table A2 · p. 19
• Sokoban and ALFWorld examples show states alongside model-generated actions, while their prompts extract plans and imagined observations from agent responses.Source: Table A3 · p. 20, Table A4 · p. 21, Table A5 · p. 22
• Separate prompts evaluate imagined next states against actual next states as references for Sokoban and ALFWorld.Source: Table A6 · p. 23, Table A8 · p. 25