Source-linked AI summary
Reinforcement Learning for the Game of Balatro: A Fast Simulator and Two-Policy Training Approach
Ajay Vikram Periasami, Anjaneya Gupta, Ethan Cheng, Ivan Rosales
TL;DR
Balatro’s large state-dependent action space, long sparse-reward horizon, and stochasticity make RL difficult, and published results remain scarce. The paper combines a fast Rust simulator and masked interface with jointly trained blind and shop policies and routed rewards. Training shows a sharp rise to above 99% first-blind clears, but mean maximum ante remains about 3 of 8, so the system does not yet complete runs.
Problem
Balatro combines a large state-dependent action space, long horizons with sparse rewards, and stochastic transitions, while published RL results remain scarce.
Source: §1 · pp. 1–2Method
The paper jointly trains tactical blind and strategic shop policies on shared state, with phase-specific actions and explicit cross-agent reward routing.
Source: §5.1 · p. 5, §5.4 · p. 8Results
The first-blind clear rate crosses 0.99 by iteration 173 as mean return rises roughly an order of magnitude, but mean maximum ante reaches only about 3 of 8.
Source: §5.6 · p. 9, §7 · p. 10Takeaways & Limitations
The joint system learns the tactical first-blind subproblem and makes progress through the game, but is not yet a run-completing policy.
Source: §5.6 · p. 9The V1 interface omits parts of the joker and voucher pool and caps dynamic slots, while the study lacks held-out seed and live-client evaluation.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Balatro, a 2024 roguelike deckbuilder, presents three difficulties that have historically derailed reinforcement learning methods: a 3,000-action state-dependent action space, horizons of several hundred decisions with sparse rewards, and pervasive stochasticity in both card draws and shop generation. Published RL results on the game remain scarce. We introduce an end-to-end pipeline comprising (i) a high-throughput Rust simulator that is automatically state-diffed against the live game client via BalatroBot, (ii) a condensed observation-action interface with dynamic per-step masking of legal moves, and (iii) a multi-agent PPO architecture that decomposes play into a tactical blind policy and a strategic shop policy that share the full game state but act in disjoint phases. Long-horizon credit assignment is partially addressed by per-agent reward routing with explicit cross-agent credit. On the main joint run, training metrics show a sharp phase transition in which the first-blind clear rate converges above 99% and mean episode return increases by roughly an order of magnitude, plateauing at a mean maximum ante of 3 out of 8; single-agent baselines under identical rewards oscillated between greedy and hoarding failure modes and did not stably clear the first blind. These results establish the two-policy decomposition as a strong reactive baseline for Balatro and localise the remaining gap to long-horizon strategic planning rather than tactical blind play.
1 Introduction
Balatro combines tactical poker-hand decisions with longer-term deck, joker, and economy choices across stochastic runs. Its state-dependent legal actions, long horizon, sparse rewards, and randomness make it a challenging RL setting.
1 Introduction
- A run spans eight antes of three blinds each, with shops between blinds where purchases and saving compete through future interest.Players must reach each blind’s chip target by playing hands; shops offer cards, jokers, consumables, packs, and vouchers.Source: §1 · p. 1
- Tactical hand choices depend on cards, jokers, enhancements, boss effects, and remaining hands, while purchases can shape synergies and the economy many blinds later.A locally attractive purchase can starve the economy and hurt progress at a future boss.Source: §1 · pp. 1–2
- Balatro poses three RL difficulties: a large state-dependent action space, hundreds of decisions with sparse rewards, and stochastic draws and shops.The paper notes that published RL results on Balatro remain scarce.Source: §1 · p. 2
2 Background
Prior Balatro work includes community RL and tool-using LLM evaluation, while related game-playing research spans deckbuilders, card games, and complex strategy games. The paper identifies state-dependent actions, long horizons, and randomness as central challenges.
2 Background
- A community multi-agent PPO project reports about a 30% simulated win rate and 25% real-game win rate on White Stake.Its author notes that the environment supports a substantial but partial subset of the game.Source: §2 · p. 2
- BalatroBot enables external agents to drive the live game, and BalatroLLM and BalatroBench evaluate LLM agents on measures such as average final round.These real-game systems are slower than simulators and are described as complementary to simulator-driven RL.Source: §2 · p. 2
- Related card and deckbuilder RL includes Slay the Spire, while board-game and complex-game work suggests large data budgets and careful engineering for human-level play.The paper also notes that masking and architecture design can reduce sample complexity.Source: §2 · p. 2
- Balatro’s large state-dependent action space, long horizon, and randomness across rounds are identified as likely sources of difficulty and possible seed overfitting.The paper says its implemented methods were chosen partly to address these properties.Source: §2 · p. 2
3 Simulator
The simulator aims to combine throughput with game-logic accuracy and version parity, using a Rust engine, configurable interfaces, dynamic action masking, and automated comparisons with the live client.
3 Simulator
- Direct client training is faithful but throttled by frame execution and graphics, while community Python gyms can be faster yet outdated; the paper builds a Rust simulator to reconcile these tradeoffs.Balance patches can create version drift between gym-trained strategies and the current game.Source: §3 · p. 2
3.1 Building the Simulator
- The simulator reproduces scoring behavior, including recursive triggers and held-in-hand bonuses, using execution order and balance values from recent Lua source.The Rust core is made available to training code through Python bindings.Source: §3.1 · p. 3
Core API Methods
- The core API provides Reset for initialization, Step for action-driven progression and next observations, and Get State for internal-state access used in debugging and analysis.Figure 1 contrasts the original game with the web app implementation.Source: Core API Methods · p. 3, Figure 1 · p. 3
3.2 Action and Observation Space
- The maximalist interface exposes about 3,000 actions and a 20,000-dimensional observation space, with dynamic masks restricting choices to valid actions for the current state.Masks cover game phase, resource and slot constraints, and card presence, so the agent samples only legal actions.Source: §3.2 · pp. 3–4
3.3 Implementation Subsets
- Because the full observation space is costly to train on and dynamic slots can expand, the authors use a constrained V1 subset with buffer capacity for some state variation.The subset reduces the action space while retaining limited room for changes such as hand size or joker slots.Source: §3.3 · p. 4
3.4 Verifying Simulator Accuracy
- Verification progressed from manual action synchronization through WASM to automated step-by-step state comparisons with the live client via BalatroBot.Logged traces supported deterministic bug replay and regression testing.Source: §3.4 · pp. 4–5
4 Core Reward Scheme
The core reward scheme is sparse and event-driven, assigning most transitions zero reward and reserving signal for hand scoring and terminal outcomes.
4 Core Reward Scheme
- A cleared blind rewards 1; an uncleared hand with hands remaining earns its chip total divided by the target, capped at 1, while running out of hands or stalemating yields -1.Most other transitions return zero, and the Python wrapper adds shaping signals and cross-agent routing on top of this base scheme.Source: §4 · p. 5
5 Multi-agent RL
The section presents jointly trained blind and shop specialists, with separate state representations and routed rewards. Training shows strong first-blind performance but plateaus near ante 3 rather than completing runs.
5.1 Motivation: why two policies?
- Separate blind and shop specialists address local tactical play versus long-reach economic choices; they share full state but act in assigned phases.Early single-agent runs oscillated between greedy play and hoarding.Source: §5.1 · p. 5
5.2 The blind policy
- Hand-card self-attention summarizes relational choices, while an autoregressive head selects play or discard, builds a card set, then commits or stops.The action structure trained more stably than a flat blind-policy head in early experiments.Source: §5.2 · p. 6
5.3 The shop policy
- Shop attention evaluates items in the context of owned assets, while separate heads score item-specific and abstract actions.The shared shop-context representation also serves the value head and was found to stabilize shop training.Source: §5.3 · p. 7
5.4 Per-agent rewards
- Reward shaping routes acting-step rewards and dense agent-specific bonuses; blind-clear cash and round rewards go to shop, while run-win rewards go to both.Blind bonuses include hand quality, discard credit, urgency, graduated loss penalties, ante-scaled clears, and efficiency; shop bonuses address interest preservation and joker growth.Source: §5.4 · pp. 7–8
5.5 Training
- The two policies train jointly with PPO; the main run spans 4,000 iterations, with shop entropy decaying from 0.02 to 0.001 by 4M steps.The blind agent uses a constant entropy coefficient of 0.001.Source: §5.5 · p. 8
5.6 Analysis
- Training curves are exploration-enabled on-policy batch metrics, and their five correlated measures are joint diagnostics rather than independent tests.Only the first 1,000 of 4,000 iterations are plotted; the 40-iteration rolling mean delays the clear-rate transition.Source: §5.6 · pp. 8–10
- 0.99 first-blind clear rate by iteration 173 gives way to mean max ante 2.6–3.0 from iteration ∼400; the curves do not establish run wins.The five training metrics are smoothed over a 40-iteration window.Source: §5.6 · p. 9, Figure 3 · p. 10
6 Limitations
The paper's claims are bounded by simulator coverage, compute scale, seed evaluation, lack of search, and having only one full-length main run.
6 Limitations
- The V1 interface excludes parts of the joker and voucher pools and caps dynamic slots, leaving some late-game strategies out of reach by construction.The passage specifically names legendary jokers and stacking beyond the buffer.Source: §6 · p. 10
- At approximately 20.5M environment steps, the main run is one to two orders of magnitude shorter than typical compute budgets for mastering complex stochastic games.The authors frame the results as evidence of learnability rather than mastery.Source: §6 · p. 10
- Without held-out seed evaluation or live-client evaluation, train–eval gaps and overfitting to simulator-specific RNG cannot be ruled out.Source: §6 · p. 10
- The policies are purely reactive, so claims about the difficulty of particular blinds or builds are conditional on using no decision-time lookahead.Source: §6 · p. 10
- The main result comes from one full-length training run, with shorter-horizon ablations; variance across full-length runs is uncharacterized.Source: §6 · p. 10
7 Conclusion
The paper presents an end-to-end Balatro RL pipeline and a jointly trained blind and shop policy. The main run clears the first blind above 99% but plateaus near ante 3, leaving long-horizon strategy unresolved.
7 Conclusion
- The main joint run reaches over 99% first-blind clear rate, then plateaus at a mean maximum ante of approximately 3 out of 8 without completing full runs.The trajectory includes a sharp transition from first-blind failure to longer, economically meaningful episodes.Source: §7 · p. 10
- The pipeline combines a high-throughput Rust simulator with live-client parity checks, dynamic action masking, and jointly trained tactical blind and strategic shop policies.The policies share the game state and use per-agent reward routing.Source: §7 · p. 10
- Single-agent baselines oscillated between greedy and hoarding modes and did not stably clear early blinds, while the main open gap remains long-horizon strategic planning.Source: §7 · p. 10
8 Future Directions
Future work proposes search-based methods and broader simulator coverage, as well as a language-model training recipe using gameplay traces and game-score rewards.
8 Future Directions
- The authors propose moving beyond model-free PPO toward MCTS-based AlphaZero or MuZero to add lookahead for long-term card-combination scoring.They also identify continued refinement of the Rust simulator and automated verifier as priorities.Source: §8 · p. 11
- Planned simulator work includes adding legendary jokers and exotic vouchers to state coverage and expanding BalatroBot verification with asynchronous checks.The stated goal is faithful simulator representation across all seeds.Source: §8 · p. 11
- Another proposed direction is to warm-start an open language model from gameplay traces produced by a teacher, then refine it with GRPO using game score as reward.The authors suggest this recipe as a fit for decisions involving joker synergies and shop purchases.Source: §8 · p. 11
A V1 Action and Observation Space Mappings
The appendix identifies a V1 mapping for a 74-action subset and a V1 observation mapping with 1,449 dimensions.
A V1 Action and Observation Space Mappings
- Table 2 presents the V1 action-space mapping, comprising 74 discrete actions.Source: Table 2 · p. 12
- Table 3 presents the V1 observation-space mapping, comprising 1,449 dimensions.Its left and right halves are contiguous and split for typesetting.Source: Table 3 · p. 13