Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning

Chengshuai Shi; Wenzhe Li; Xinran Liang; Yizhou Lu; Wenjia Yang; Ruirong Feng; Seth Karten; Ziran Yang; Zihan Ding; Gabriel Sarch; Danqi Chen; Karthik Narasimhan; Chi Jin

Source-linked AI summary

Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning

Chengshuai Shi, Wenzhe Li, Xinran Liang, Yizhou Lu, Wenjia Yang, Ruirong Feng, Seth Karten, Ziran Yang, Zihan Ding, Gabriel Sarch, Danqi Chen, Karthik Narasimhan, Chi Jin

arXiv:2605.00347v1May 1, 2026cs.LGcs.AIcs.CL

TL;DR

Long-horizon VLM decision-making in games remains limited beyond short-horizon RL settings. Odysseus adapts PPO with a lightweight turn-level critic and achieves stable, sample-efficient training with substantial gains and generalization.

Problem
Existing VLM fine-tuning approaches rely on large-scale SFT or short-horizon RL, leaving 100+ turn game decision-making underexplored.
Method
Odysseus combines lightweight supervised initialization with multi-task RL using adapted PPO, a turn-level critic, positive-advantage filtering, and auto-curriculum sampling.
Results
Odysseus outperforms frontier models by at least 3× in game progress and improves in-game and cross-game performance by up to 41.5%.
Takeaways & Limitations
Stable RL and pretrained VLM action priors support effective training of embodied agents in long-horizon, multimodal environments.
Takeaways & Limitations
The sample-efficiency comparison assumes pretrained VLMs provide perception and action priors unavailable to agents trained from scratch.

Abstract

from arXiv · show

Given the rapidly growing capabilities of vision-language models (VLMs), extending them to interactive decision-making tasks such as video games has emerged as a promising frontier. However, existing approaches either rely on large-scale supervised fine-tuning (SFT) on human trajectories or apply reinforcement learning (RL) only in relatively short-horizon settings (typically around 20--30 turns). In this work, we study RL-based training of VLMs for long-horizon decision-making in Super Mario Land, a visually grounded environment requiring 100+ turns of interaction with coordinated perception, reasoning, and action. We begin with a systematic investigation of key algorithmic components and propose an adapted variant of PPO with a lightweight turn-level critic, which substantially improves training stability and sample efficiency over critic-free methods such as GRPO and Reinforce++. We further show that pretrained VLMs provide strong action priors, significantly improving sample efficiency during RL training and reducing the need for manual design choices such as action engineering, compared to classical deep RL trained from scratch. Building on these insights, we introduce Odysseus, an open training framework for VLM agents, achieving substantial gains across multiple levels of the game and at least 3 times average game progresses than frontier models. Moreover, the trained models exhibit consistent improvements under both in-game and cross-game generalization settings, while maintaining general-domain capabilities. Overall, our results identify key ingredients for making RL stable and effective in long-horizon, multi-modal settings, and provide practical guidance for developing VLMs as embodied agents.

1 Introduction

The paper studies reinforcement-learning training of VLMs for Super Mario Land, where successful decision-making requires coordinated perception, reasoning, and action over trajectories often exceeding 100 turns. It identifies stabilization techniques, pretrained action priors, and an open training framework as key ingredients for effective long-horizon VLM training.

Motivation: Super Mario Land requires coordinated perception, reasoning, and action over extended trajectories often exceeding 100 turns, while generalization spans diverse levels with varying layouts and dynamics.The environment remains challenging even for frontier models.
Algorithmic ingredients: Adapted PPO achieves strong stability and sample efficiency in long-horizon game environments where popular critic-free methods perform poorly.The approach introduces a lightweight turn-level critic and positive-advantage filtering.
VLM-based RL training v.s. classical deep RL: Pretrained VLMs provide strong action priors that improve reinforcement-learning sample efficiency and reduce reliance on manual action-space engineering.This contrasts with classical deep RL, which trains policies from scratch.
An open training framework for practical agentic tasks: Odysseus integrates lightweight supervised initialization with multi-task RL and supports stable training over tens of millions of interaction samples.The framework achieves substantial gains across the game over the base model and outperforms open-source and proprietary frontier models by a large margin.
Conclusion: Once reinforcement learning is stabilized, foundation-model priors further facilitate learning in 100+ turn decision-making environments.The authors present this as a practical foundation for training multi-modal foundation models as embodied agents.

2 Related Work

Related work has applied RL for VLM agents mainly to short-horizon games and environments, while foundation-model approaches to embodied control often rely on large-scale supervised demonstrations. This work instead studies RL for long-horizon, visually grounded control in Super Mario Land.

Games and Simulated Environments: RL for VLMs has mainly been evaluated in short-horizon environments such as AlfWorld, Sokoban, and FrozenLake.These scenarios typically involve relatively short interaction horizons, around 20–30 turns.
Games and Simulated Environments: Super Mario Land offers a compact testbed for long-horizon VLM control with richer spatial grounding and closed-loop interaction.The paper positions the game as a contrast to prior short-horizon settings.
Foundation Models for Decision-Making: Pretrained foundation models have enabled capable embodied agents, but these approaches commonly depend on supervised fine-tuning with large action-labeled demonstration datasets.Prior applications include robotic manipulation and cross-game generalization.
RL for Foundation-Model Agents: RL research for multi-turn language and vision-language agents often uses specialized trajectory decomposition, token-level advantage estimation, or hierarchical credit assignment.These methods are typically evaluated on environments with relatively short horizons of 20–30 turns.

3 VLMs for Decision-Making in Super Mario Land

Super Mario Land is used as a long-horizon VLM decision-making testbed requiring spatial perception, reasoning, and precise control over trajectories often exceeding 100 turns. The interaction protocol combines rendered frames and textual instructions, structured chain-of-thought, discrete actions, and frame skipping for effective execution.

Testbed: Super Mario Land requires accurate spatial perception, reasoning, and precise motor control across extended trajectories often exceeding 100 turns.The game includes diverse levels with obstacles and adversaries.
Testbed: The game contains 12 levels across 4 worlds, with 10 levels used after excluding World 2 Level 3 and World 4 Level 3 for distinct control mechanisms.
Testbed: Foundation models struggle zero-shot with timing-sensitive tasks such as jumping gaps and avoiding moving threats, producing brittle policies that rarely progress beyond initial stages.
Environment interface: The agent observes rendered pixel frames and textual prompts rather than the full game-RAM state, and selects from discrete combinations of standard controller inputs.The game’s underlying state corresponds to the full internal state of game RAM.
Interaction protocol: Structured chain-of-thought requires explicit perception, step-by-step reasoning, and a final answer describing the action to execute.The perception step identifies Mario, obstacles, enemies, and interactive elements; reasoning plans responses such as timed jumps or movement toward pipes.
Interaction protocol: A frame-skip mechanism repeatedly applies each chosen discrete action because one emulator frame produces negligible movement.This ensures the action has an observable effect in the game environment.

4 Algorithmic Ingredients of Stable RL for VLMs

Stable long-horizon RL for VLMs requires moving beyond critic-free methods toward an adapted PPO design with efficient turn-level value estimation. A lightweight CNN critic and positive-advantage filtering are key components evaluated for improving training stability and performance.

Motivation: Critic-free GRPO and Reinforce++ methods fail to learn policies making consistent multi-step progress in long-horizon dense-reward settings.This failure occurs across both outcome-reward and process-reward variants.
Adapted PPO: Turn-level critics reduce the overhead of PPO value learning compared with token-level critics based on large models.The proposed change addresses the computation and memory costs associated with learning a token-level critic.
Lightweight Critic: CNN critics can effectively stabilize training without requiring a computationally expensive secondary VLM as the value network.The approach is particularly suited to environments with rich visual state information.
Advantage Filtering: Positive-advantage filtering clips negative advantages at zero to reduce optimization instability during training.Samples with negative advantages, ˆAt < 0, are filtered out.
Controlled Comparison: PPO-based methods achieve substantially stronger and more stable improvements than critic-free methods, highlighting the importance of a learned critic for long-horizon credit assignment.Among critic-free methods, only GRPO with outcome rewards shows observable post-training improvements.

5 VLM-Based RL Training versus Classical Deep RL

This section evaluates VLM-based RL against classical deep RL through sample efficiency. Pretrained VLMs provide visual and action priors that improve exploration and reduce manual action-space engineering.

Motivation: The section tests whether VLM-based RL is more sample-efficient than classical deep RL trained from scratch.The comparison focuses on sample efficiency in the same long-horizon game environment.
Experimental Results: 0.38M training samples: PPO with a turn-level CNN critic substantially outperforms critic-free methods, while positive-advantage filtering further stabilizes training.This comparison concerns VLM-based RL training methods.
Experimental Results: Roughly 2× higher sample efficiency: VLM-based RL outperforms classical PPO training a CNN policy from scratch, even without action-space engineering.The VLM-based method uses PPO with a turn-level CNN critic and positive-advantage filtering.
Motivation: Pretrained VLMs encode visual representations and world knowledge, providing priors for perception and action that classical agents lack.Classical agents must learn perception and control from scratch, whereas VLM-based RL can leverage prior knowledge.
Experimental Results: Classical PPO with the original 22-action space makes slow progress because exploration is difficult in the large combinatorial action space.An engineered action space improves classical RL by restricting the policy to fewer, more semantically meaningful actions, but it requires manual design.
Conclusion: These results support pretrained VLMs as knowledgeable RL priors that improve sample efficiency and reduce manual engineering in embodied decision-making.The findings attribute the benefit to strong inductive biases that reduce the exploration burden in long-horizon tasks.

6 Odysseus: An Open and Practical Training Framework

Odysseus is an open framework that combines lightweight supervised fine-tuning initialization with multi-task reinforcement learning for practical VLM decision-making agents. Its training pipeline uses adapted PPO with positive-advantage filtering and an inverse-trajectory-weighted auto-curriculum to balance learning across game levels.

Framework: Odysseus integrates supervised fine-tuning initialization and multi-task RL training into an open framework for practical decision-making agents.The primary instantiation targets Super Mario Land, while the framework is intended to inform broader settings.
Supervised fine-tuning: Around 5,000 frames across 10 levels are annotated by GPT-o3 with structured perception, reasoning, and answer fields for lightweight initialization.The dataset is curated from two walkthrough videos that complete the game.
Supervised fine-tuning: The SFT stage improves domain knowledge and environment perception, while action-control optimization is deferred to RL rather than learned from expert trajectories with annotated actions.The authors intentionally make SFT smaller than previous works and focus it on perceptual grounding.
Reinforcement learning: Adapted PPO with positive-advantage filtering optimizes action selection after SFT, using trajectories collected from multiple game levels for multi-task training.The method builds on the algorithmic findings reported earlier in the paper.
Auto-curriculum: Inverse trajectory weighting samples level k with w_k ∝ 1/N_k, up-weighting shorter-trajectory levels and down-weighting longer ones to reduce optimization bias toward easier levels.Here, N_k is the average trajectory length for level k in the current batch.

7 The Effectiveness of Odysseus

Odysseus substantially improves Super Mario Land performance over the base and frontier VLMs, with SFT initialization strengthening RL training. Despite training on only five levels, it generalizes across in-game and cross-game settings while preserving general-purpose multimodal capabilities.

Game performance: Odysseus typically triples, and often increases by an order of magnitude, average level progress over the base model while significantly outperforming frontier models.The comparison is reported on the five levels used for RL training.
Ablation study: SFT initialization enables more effective RL: Odysseus consistently outperforms Odysseus-Zero across all levels, although Odysseus-SFT alone shows no observable gain.Odysseus-Zero is trained directly from the base model without SFT initialization.
Generalization: 32.2% and 41.5% are Odysseus’s average relative improvements on in-game off-policy and unseen states, respectively, while cross-game improvement averages 23.1%.The evaluations use states from training levels, held-out levels, and all 32 levels of Super Mario Bros.
Generalization: Odysseus-series models maintain the base model’s strong capabilities on general-purpose multimodal benchmarks despite extensive game-environment training.The benchmarks include MMMU, MathVision, and RealWorldQA, covering STEM-oriented and real-world reasoning tasks.

8 Conclusions · A Extended Related Work · B Details of the Interaction Protocol

The paper studies reinforcement-learning training for VLMs in 100+ turn Super Mario Land episodes, emphasizing an adapted PPO method with a lightweight turn-level critic. It situates this approach within foundation-model and classical-RL decision-making research and specifies a structured visual interaction protocol.

8 Conclusions: The study targets long-horizon VLM decision-making in Super Mario Land, where each episode requires 100+ interaction turns.It introduces an adapted PPO algorithm with a lightweight turn-level critic to improve training stability and sample efficiency over critic-free methods.
A Extended Related Work: Related work spans classical game and simulated-environment benchmarks, including ALE and MuJoCo, which supported the study of interaction-based learning.Later benchmarks extend this paradigm to more complex settings, including multi-agent reinforcement learning.
A Extended Related Work: Foundation-model decision-making has included sequence modeling with Transformers trained from scratch on offline trajectories, alongside approaches demonstrating task generalization and skill composition.The passage presents these as distinct paradigms in the evolution of foundation models for sequential decision-making.
A Extended Related Work: The paper argues that a comparatively simple PPO-based approach with the right critic design can make RL stable and effective.This conclusion is supported by rigorous ablations in long-horizon, visually grounded environments requiring 100+ turns and chain-of-thought reasoning.
B Details of the Interaction Protocol: The agent’s objective is to progress through levels, collect coins and power-ups safely, and ultimately rescue Princess Daisy.The prompt identifies Super Mario Land as the environment and Game Boy buttons as the control interface.
B Details of the Interaction Protocol: The interaction protocol instructs the VLM to analyze each game screen, reason step by step, and return a button sequence in a prescribed answer format.The final action is represented as a list whose buttons must be selected from a fixed set of seven controls.
B Details of the Interaction Protocol: At most 2 buttons may be pressed simultaneously in one turn.Available controls include jump, run/shoot, climbing, crouching or entering pipes, horizontal movement, and no operation.
B Details of the Interaction Protocol: At each turn, the agent observes the current frame and prompt, while the 160 × 144 game image is up-sampled by 8 to 1280 × 1152 before action execution.The final action is extracted from the <answer></answer> field and executed as button presses in the environment.

C Details of RL Algorithms for VLM Training · C.1 Advantage Constructions

This section specifies the shared surrogate-loss setup and the advantage constructions used by the adapted PPO, GRPO, and Reinforce++ algorithms. The methods differ primarily in how turn-level training advantages are formed from critics, outcome rewards, process rewards, or discounted returns.

C Details of RL Algorithms for VLM Training: The appendix provides implementation details for the RL algorithms discussed in Section 4, including the proposed adapted PPO variant.
C.1 Advantage Constructions: All methods use trajectory batches collected by the previous policy πold and are instantiated with the same surrogate loss.
C.1 Advantage Constructions: The algorithms differ in their advantage estimators, with ϵlow and ϵhigh defining clipping factors and ˆAt denoting the advantage at turn t.
C.1 Advantage Constructions: PPO with a turn-level critic learns Vϕ(ot) to approximate the policy value from st and uses discounted return-to-go ˆRt = ∑i≥t γi−tri as its target.
C.1 Advantage Constructions: PPO computes raw per-turn advantages and then applies batch-level variance normalization before obtaining the final advantage.When positive-advantage filtering is used, signed advantages are retained and normalized instead.
C.1 Advantage Constructions: GRPO with outcome rewards assigns every turn in a trajectory the same cumulative trajectory-level return, equivalent to using γ = 1 for return-to-go.The outcome return is standardized across the batch, and positive advantage filtering sets ˆAt = max(0, ˆAt).
C.1 Advantage Constructions: GRPO with process rewards standardizes turn rewards over the current batch and constructs each training signal as the undiscounted return-to-go.Positive advantage filtering again sets ˆAt = max(0, ˆAt).
C.1 Advantage Constructions: Reinforce++ uses PPO’s turn-level discounted return-to-go, ˆRt = ∑i≥t γi−tri, and standardizes it over the entire batch as the advantage.

C.2 Experimental Details and Additional Results for Section 4.2 … E.1 Details of the CNN Critic

The supplementary sections specify shared VLM training configurations, classical deep RL comparison settings, and the CNN critic design used in adapted PPO. They also document the evaluation figures and hyper-parameter tables supporting these experiments.

C.2 Experimental Details and Additional Results for Section 4.2: All Section 4.2 algorithms train every component of Qwen3-VL-8B-Instruct, including the vision encoder, multi-modal projector, and language-model backbone.The adapted PPO experiments share configurations, with CNN-critic architecture and hyper-parameters detailed separately.
D Details of Comparisons with Classical Deep RL: The classical deep RL environment matches VLM training, including rewards and frame-skipping mechanisms, while the action spaces differ.VLMs can output up to two simultaneous buttons from seven choices; the classical-RL action space is described as “original” in the supplied passage.
D Details of Comparisons with Classical Deep RL: Classical RL uses Stable-Baselines 3 PPO with CnnPolicy and its default NatureCNN backbone.The learning rate is selected by grid search over [5 × 10−5, 1 × 10−4, 1.5 × 10−4, 2.5 × 10−4].
D Details of Comparisons with Classical Deep RL: Classical RL training runs for up to 2 × 10^6 state-action pairs, with evaluations performed periodically.The supplied passage states that evaluations occur every 1…; the interval is truncated.
E Details of Training and Evaluations of Odysseus: The VLM-based RL methods are compared in Figure 7 using separate unsmoothed individual runs.The figure is the reported comparison for VLM-based RL training methods.
D Details of Comparisons with Classical Deep RL: The supplementary tables document the engineered classical-RL action space and PPO-via-SB3 hyper-parameters.Table 5 covers the engineered action space, while Table 6 covers classical RL experiments with PPO via SB3.
E.1 Details of the CNN Critic: The adapted PPO critic uses the same CNN backbone as Mnih et al. (2015), with detailed critic-learning hyper-parameters provided in Table 7.This specifies the critic backbone but not the individual Table 7 hyper-parameter values.
E Details of Training and Evaluations of Odysseus: The VLM-based RL and classical RL approaches are compared in Figure 8 using separate unsmoothed individual runs.The supplied materials also identify complete classical-RL experimental results in Figure 8.

E.2 Other Training Details

Training updates all components of Qwen3-VL-8B-Instruct, uses Qwen3-VL repository configurations for SFT initialization, and runs RL under a substantially modified VeRL framework.

Model Training: All experiments train the Qwen3-VL-8B-Instruct base model, including its vision encoder, multi-modal projector, and language model backbone.The training is performed on all components rather than only selected modules.
Training Frameworks: SFT initialization uses the Qwen3-VL repository, while RL training uses a substantially modified VeRL framework with configurations listed in Tables 8 and 9.The passage attributes the framework to Sheng et al. (2024).

E.3 Evaluation Details · F Additional Visualizations

Evaluation statistics use repeated runs and matched inference/training hyperparameters, with separate sampling protocols for proprietary and open-source models. Additional visualizations show Odysseus overcoming failure modes of the base model through more accurate scene interpretation and action timing.

E.3 Evaluation Details: 8 runs are used for close-source proprietary models, while 256 runs are used for remaining open-source models when computing summary statistics.Statistics are reported as means and standard errors.
E.3 Evaluation Details: Inference uses the same set of hyperparameters as training for every model.
E.3 Evaluation Details: In-game generalization states are manually collected across 10 Super Mario Land levels, whereas cross-game evaluation starts from the beginning in 32 Super Mario Bros. levels.
F Additional Visualizations: Representative visualizations show the base model failing to progress while Odysseus continues successfully across game scenarios.The figures also compare differences in the models’ generated chains of thought.
F Additional Visualizations: Odysseus jumps over an enemy at the correct timing, while the base model repeatedly outputs right because it misjudges the distance between Mario and the enemy.
F Additional Visualizations: Odysseus jumps onto the higher platform to cross a gap, while the base model moves right and down after mistaking a background object for an enterable pipe.