PaperlayerView exact revision on arXiv

Source-linked AI summary

Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players

Changlian Ma, Limin Wang

arXiv:2610.33845v1cs.AIcs.LGcs.MA

TL;DR

The paper asks whether LLM agents can take a hard real-world problem from raw data to a system that holds its own against people, in a game with no faithful simulator and prior work limited to advice or simplified versions. AutoTavern uses specialised research agents, a shared verified memory, and live-game feedback to improve its simulator, opponents, policy, and research process. In 20 days it built a full-content simulator and a live-ladder agent whose performance rose against reality-calibrated opponents, while direct LLM play fell short.

  • Problem

    Existing evidence for LLM research agents came from benchmark tasks or paper reviews, leaving open whether they could take a hard real-world problem from raw data to a system that holds its own against people.

    Source: §1 · p. 1

  • Method

    AutoTavern uses specialised LLM research agents, typed versioned memory, verification gates, and live-game feedback to improve the simulator, opponents, policy, and research process.

    Source: §1 · p. 2, §4 · pp. 4–5

  • Results

    TavernAgent’s first-place rate against reality-calibrated opponents rose from 1.9% to 41.5%, and it reached about 15,000 rating on the live ladder.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On TavernBench, the LLMs that built a ladder-level agent fall well short when making decisions directly and decide too slowly for a live turn.

    Source: §1 · p. 3

Abstract

from arXiv · show

Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people? We study this in Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden information and nearly a thousand interacting cards. We present AutoTavern, a self-improving research system: specialised LLM research agents share a typed research memory, every claim passes verification gates, and live games close the loop, so the agents improve the simulator, opponents and policy and, from the same feedback, their own research process; a human sets goals and rules on ambiguous mechanics. In 20 days AutoTavern built TavernGym, the first faithful, full-content reproduction of the live game, and delivered 23 versions of TavernAgent, trained with PPO against opponents calibrated to real games. TavernAgent plays complete games on the live ladder by itself: its rating rose from 6,000 to about 15,000 and, on the public leaderboard's measure, its last 20 records average a placement of 1.60 with 80% first places, better than the server's ten highest-rated players on every statistic (2.34, 59%). TavernBench, 960 tasks in 40 scenarios, shows that the system the agents built is the strongest player: TavernAgent answers 91% of the play tasks against at most 63% for frontier LLMs, and decides in 0.03 s where they need up to 26 s; within the second a live turn allows, it keeps 90% and no LLM more than 26%. LLM agents build far better than they play. We will open-source TavernBench and TavernGym at https://github.com/MCG-NJU/AutoTavern.

1 Introduction

Prior game AI relied on human experts to build environments and algorithms, while evidence for LLM research agents had not established whether they could take a hard real-world problem to human-level play. AutoTavern uses feedback from live games to improve its simulator, agent, and research process, producing a full-game simulator, a live-ladder agent, and a decision benchmark.

1 Introduction

  • The open question is whether LLM agents can carry a hard real-world problem from raw data to a system that holds its own against people, where performance is measured in the real game.Battlegrounds combines stochastic shops and battles, hidden information, and nearly a thousand interacting game objects; no faithful simulator had been available.Source: §1 · pp. 1–2
  • AutoTavern uses specialised LLM agents, typed versioned research memory, verification gates, and live-game feedback to improve the model, training world, and research process.It ran about 10,000 experiments and delivered 23 agent versions in 20 days, with 418 human messages in all.Source: §1 · p. 2
  • TavernGym is presented as the first faithful, full-content reproduction of the live game, covering all 978 objects in patch 36.6.3 and aligned rule by rule with real games.The simulator is deterministic and runs 190–280 complete eight-player games per second on one core.Source: §1 · p. 2
  • TavernAgent’s 23 versions raised first-place rate against reality-calibrated opponents from 1.9% to 41.5%; on the live ladder it climbed from rating 6,000 to about 15,000.The paper also reports recent results better than the server’s ten highest-rated players on every statistic.Source: §1 · p. 2
  • TavernBench contains 960 tasks in 40 scenarios across eight categories, and its results show that LLMs build far better than they play.The benchmark compares direct decisions with the system the agents built; its play score counts only answers made within a live turn’s one-second allowance.Source: §1 · pp. 2–3, Figure 1 · p. 2

2 Related Work

Prior Battlegrounds work addressed partial or simplified settings, while large-scale game RL generally starts from an existing environment. AutoTavern differs by using agents to build and correct the training world, evaluating on a live ladder, and benchmarking direct LLM play under real-time constraints.

AI for Hearthstone and Battlegrounds

  • Earlier Battlegrounds systems used a simplified simulator, omitted heroes, simulated combat only, or advised players; none reproduced the full game or trained an agent to play it.The cited work includes a Monte Carlo tree search assistant evaluated on early turns of recorded games.Source: §2 · p. 3

Large-scale game RL

  • Large-scale game RL systems combine self-play or human data with large-scale training and start from an existing environment; AutoTavern instead focuses on building and correcting the training world.Its learner uses PPO with a transformer.Source: §2 · p. 3

LLM research agents

  • AutoTavern agents improve their research process through lessons and verification gates, with progress measured on a live ladder and a reality loop re-anchoring the simulator and opponents.This differs from systems that improve their scaffolds by rewriting their own code.Source: §2 · p. 4

LLMs as game players

  • TavernBench adds a stochastic multiplayer game of economy and combat, simulator-verified answers, a specialised-agent reference, and a real-time view to existing LLM game benchmarks.Source: §2 · p. 4

3 Hearthstone Battlegrounds

Battlegrounds is an eight-player elimination game with alternating recruit and automatic combat phases. Players manage a warband amid random offers and interacting mechanics, and the game ends with a placement from first to eighth.

3 Hearthstone Battlegrounds

  • Each of eight players spends gold during the recruit phase to buy, sell, refresh or freeze minions, upgrade the tavern, play cards, use a Hero Power, and arrange up to seven minions.Gold rises from 3 on turn 1 to 10, and tavern tiers run from 1 to 7.Source: §3 · p. 4
  • Random tavern offers come from a shared pool, and interacting mechanics include Battlecries, deathrattles, auras, magnetic attachments, spells, and trinkets.Three copies of a minion merge into a golden version that offers a discovery from the next tier.Source: §3 · p. 4
  • Combat is automatic against one opponent with random targets; the loser takes damage, players are eliminated until one remains, and the outcome is placement from 1 to 8.Source: §3 · p. 4
  • The game is modeled as a partially observable stochastic game with terminal placement utility; players see their own state and only opponents’ public leaderboard and last-seen warbands.Patch 36.6.3 contains 978 game objects, and a game lasts about 15 turns and 450 atomic actions per player.Source: §3 · p. 4

4 The AUTOTAVERN System

AUTOTAVERN improves the policy and its research process by repeatedly anchoring its proxies to live games and checking claims before they alter shared state. A multiscale cycle coordinates specialized agents, verification gates, and limited human input.

4 The AUTOTAVERN System

  • The system counters proxy drift with a reality loop and false findings with verification gates, while updating artifacts and lessons when errors are caught.Agents exchange findings through shared memory; every proxy is re-anchored to the real game at a fixed cadence.Source: §4 · p. 4

4.1 The iteration cycle

  • An iteration observes and diagnoses, plans and implements an intervention, trains and evaluates it, deploys it to the ladder, then records lessons and corrections.The cycle runs on three timescales: policy updates in seconds to minutes, experiments within hours, and live-game feedback within days.Source: §4 · p. 5, §4.1 · p. 5

4 The AUTOTAVERN System

  • Agents work through versioned, typed stores that preserve findings, corrections, rule decisions, experiments, handovers, and lessons across sessions.Findings are not overwritten; corrections point back to earlier entries.Source: §4 · p. 5
  • Live games are replayed in TAVERNGYM to expose discrepancies, which agents fix in rules, input encodings, or opponents before training resumes.Caught errors also update the research memory and gates, so later iterations can avoid repeating them.Source: §4 · p. 5

4.1 The iteration cycle

  • The human researcher sets goals and resolves mechanics ambiguous in game data; a scheduler runs registered experiments within a resource budget.The scheduler keeps successor experiments queued so work can continue for days without a human present.Source: §4.1 · p. 5

4.2 Autonomy in numbers

  • In 20 days, agents registered about 10,000 experiments and delivered 23 versions; the human researcher sent 418 messages, 0.6% of all messages.Human messages mainly requested progress reports (42%) or set or changed a goal (24%).Source: §4.2 · p. 6

5 TAVERNGYM: Reproducing the Real Game

TAVERNGYM is a deterministic Rust engine designed to reproduce the live game and support fast, reproducible training and evaluation. Its rules and data were checked against the live game, and its interface exposes only player-visible information.

Design

  • A seeded generator makes games reproducible bit for bit, while generated card data, patch overlays, and a rule hash track game content and builds.A shared effect context serves recruit and combat; arena-based state is cheap to copy and serialize.Source: §5 · p. 6

Fidelity to the live game

  • All 978 objects from patch 36.6.3 are implemented, with 375 verification records and 220 confirmed rule decisions documenting checks.Unit tests, fuzzing with invariant checks, and an exact legal-action mask guard the implementation; live replays prompted 46 rule and card-data fixes and 73 input-encoding fixes.Source: §5 · p. 6

Speed and interface

  • A complete eight-player game with scripted bots takes 3.6–5.3 ms on one core, or 190–280 games per second.The interface provides 245 discrete actions with an exact legal mask, player-view observations, batching, and exact save and restore.Source: §5 · p. 6

6 TAVERNAGENT: Learning to Play the Live Game

TAVERNAGENT is trained with PPO against opponents calibrated to real games, then repeatedly evaluated and deployed. Its exam performance improved across versions, and it also climbed the live ladder.

Model and training

  • TAVERNAGENT represents game entities as tokens and uses a pointer head for masked actions, alongside value and placement heads.PPO training uses about one million decisions per iteration; each 300-iteration stage covers about 310 million decisions and starts from the previous stage under a KL anchor.Source: §6 · p. 6

Live ladder

  • Figure 3 tracks rating after each of 392 games, versions in play, pauses for fixes, placements, and client failures.The caption identifies two pauses in which the loop fixed problems exposed by the ladder.Source: Figure 3 · p. 7

Reality-calibrated opponents

  • Training against calibrated opponent libraries raised first-place rate from 20.9% to 27.2% on the main exam and from 10.3% to 13.5% on the harder exam.The ablation used the same checkpoint and training budget; exam libraries were not used for training.Source: §6 · p. 7, Table 3 · p. 7

Iterative improvement

  • The 23 delivered versions raised first-place rate from 1.9% to 41.5% and average placement from 3.40 to 2.33 on one simulator build, one exam, and three unseen seeds.The comparison traces versions TA-1 to TA-23.Source: §6 · p. 7, Table 3 · p. 7

Live ladder

  • Over 392 live games on days 5–18, rating rose from 6,000 to 14,007; latest versions averaged 2.37 placement, 32.6% first places, and 91.3% top-four finishes.By the study's end, rating was about 15,000; its last 20 public-leaderboard records averaged 1.60 placement versus 2.34 for the ten highest-rated players.Source: §6 · p. 7

7 TAVERNBENCH: Build, Don’t Play

TAVERNBENCH tests whether general-purpose LLMs can make the decisions made by TAVERNAGENT, using simulator-labelled tasks and comparisons under a live-turn time budget. TAVERNAGENT substantially outperforms direct LLM play, and the results distinguish rule execution from choosing successful plays.

7 TAVERNBENCH: Build, Don’t Play

  • 91% of play tasks are answered by TAVERNAGENT, versus 63% for the best directly playing LLM, GPT-6 Astra.The benchmark comprises 960 tasks in 40 scenarios and eight categories; all systems answer the same 368 play-out tasks in the protocol.Source: §7 · p. 8
  • With TAVERNGYM as a tool, GPT-6 Astra’s score on combat, positioning, ordering, and targets rises from 62% to 97%.TAVERNAGENT is near-perfect on these tasks, where LLMs otherwise must simulate fights mentally.Source: §7 · p. 8
  • Within the one-second live-turn budget, TAVERNAGENT retains 90% accuracy, compared with 26% for DeepSeek and 37% for Jev.TAVERNAGENT answers in 0.03 s; median LLM task times range from 1.4 to 26 s, except DeepSeek-V4.1-Flash without reasoning at 0.9 s.Source: §7 · p. 8
  • Turn-level errors compound across dozens of decisions, so a model’s per-decision accuracy does not translate directly into equivalent turn performance.A late recruit turn lasts about two minutes and can contain up to about a hundred operations; a decision taking more than a second cannot be played.Source: §7 · pp. 8–9
  • Frontier LLMs execute rules well but perform worse on play tasks built from those same rules.GPT-6 Astra scores 95%, Claude Opus 92%, and Gemini 3.8 Flash 90% on rule execution; knowing card effects does not establish which play wins.Source: §7 · p. 9
  • TAVERNAGENT answers 75% of turn-plan tasks in 0.03 s, while the best LLM reaches 54%; its policy also leads on choices cut from natural games at 81%.Turn plans are scored by playing them out; the other LLMs score 19–43% on those plans.Source: §7 · p. 9

8 Lessons from Self-Improvement

Live-game feedback and verification exposed six failures that simulated metrics had missed. Correcting them improved the simulator, evaluation, deployment, and research process.

8 Lessons from Self-Improvement

  • A search appeared to gain 23.6 first-place-rate points by using copied games, but reseeding each sample removed the gain (−0.65 ± 1.14 points).The copies carried random-number state, exposing future shop and combat outcomes; the error was corrected inside the reality loop.Source: §8 · p. 9
  • Fixing two offsetting errors—overstated agent growth and overly strong late-game opponents—raised the exam first-place rate to 43–49%.The exam had previously matched live results only because the errors canceled each other.Source: §8 · p. 9
  • Selecting the best of many snapshots overstated gains by 1.2–2.1 points on selection seeds but showed −0.7 on unseen seeds, prompting gate G4..Source: §8 · p. 9
  • Opponent libraries should be measured against the opponents the learner meets, rather than uniform draws.The authors recalibrated opponent libraries and exams to opponents encountered in live games.Source: §8 · p. 9
  • TA-10’s exam first-place rate nearly doubled to 24.2% from 12.8%, yet its ladder rating fell from 8,968 to 7,658 within 32 games.The loop traced the fall to client failures and an early- and mid-game weakness hidden by the exam; TA-11 resumed the climb after retraining.Source: §8 · p. 9
  • A controller–simulator encoding mismatch caused excess whole-warband sales in live replays, leading gate G2 to add bit-level encoder comparison.The differing input flag followed a trinket choice.Source: §8 · p. 9

9 Conclusion

AUTOTAVERN built a faithful Battlegrounds simulator and a live-ladder agent through a self-improving research loop. On TAVERNBENCH, the agent outperforms direct LLM play, supporting the conclusion that LLM agents build far better than they play.

9 Conclusion

  • In 20 days, AUTOTAVERN built TAVERNGYM and TAVERNAGENT, which plays complete live-ladder games at the level of the server’s best players.The authors conclude that frontier LLMs, including those that built the system, fall well short when asked to make the decisions directly and decide too slowly for a live turn.Source: §9 · p. 9
  • The authors will open-source TAVERNBENCH and TAVERNGYM.The conclusion points to the AutoTavern GitHub repository.Source: §9 · p. 9

A The AUTOTAVERN system in detail

AUTOTAVERN’s research system coordinates role-specific agents and records findings, while live-game discrepancies drive fixes to the game reproduction and its interfaces. TAVERNGYM combines full patch content, verification, a player-facing RL interface, and high simulation throughput.

A The AUTOTAVERN system in detail

  • Long-lived research agents owned roles, while short-lived agents handled probes, gated features, or checks; coordination used versioned findings, rule decisions, open questions, handovers, lessons, and experiment records.The experiment registry contained about 10,000 experiments.Source: A The AUTOTAVERN system in detail · pp. 11–12
  • Live-game discrepancies led to 184 fixes: 35 to rules and card behavior, 11 to card data and offer rules, 73 to input encoding, 51 to the client controller, and 14 opponent-library or exam recalibrations.Each underlying defect was counted once.Source: A The AUTOTAVERN system in detail · p. 12

B TAVERNGYM details

  • TAVERNGYM’s Rust engine uses shared recruit and combat effects, event-hook card tables, copyable serializable state, and one seeded random generator.Builds and rule sets receive separate hashes, and checkpoints trained under a different rule hash are not resumed.Source: B TAVERNGYM details · p. 13
  • TAVERNGYM implements all 978 objects from patch 36.6.3 and verifies mechanics with recorded checks, unit tests, and fuzzing of random legal actions.The checks include 375 records, with 220 confirmed rulings in the decision registry and unresolved rulings tracked separately.Source: B TAVERNGYM details · p. 13
  • The RL interface provides 245 discrete actions with exact legality masks and observations limited to player-visible information.A batched environment runs multiple eight-player games, and running games can be saved and restored exactly.Source: B TAVERNGYM details · p. 13
  • A single core runs 190–280 complete eight-player games per second with scripted bots and 52–76 thousand late-game combats per second..Source: B TAVERNGYM details · p. 13

C TAVERNAGENT details

TAVERNAGENT combines a token-based policy with staged training and calibrated opponent libraries. Its final version outperforms earlier versions head-to-head, while its live-ladder results compare favorably with the server’s ten highest-rated players on the public leaderboard’s measure.

C TAVERNAGENT details

  • The policy represents game entities as tokens, combines card identity, static features, and rule-text embeddings, then scores masked actions with a pointer head.A shared transformer trunk also feeds value and auxiliary placement heads.Source: C TAVERNAGENT details · p. 13, Figure 5 · p. 14
  • The largest tested network performed best on placement and top-four finishes in both exams, so TAVERNAGENT uses it.This was an early from-scratch comparison using the same recipe and number of decisions; it does not compare the delivered versions.Source: C TAVERNAGENT details · p. 14, Table 8 · p. 14
  • Opponent libraries were calibrated to opponents encountered by the agent, with graded difficulty; exam libraries were held out from training.Library seats match target strength curves using board entries built from human statistics and recorded games.Source: C TAVERNAGENT details · p. 14
  • Training progressed from simplified rules to full content and player-visible information, then to first-place and library-opponent goals, with unseen-seed rechecks before delivery.Later stages chained 300-iteration runs from the previous stage under a KL anchor for the first 100 iterations.Source: C TAVERNAGENT details · p. 14
  • The final version wins a quarter of same-lobby games, twice its fair share, against seven earlier delivered versions.The comparison seats eight delivered versions together, without library opponents.Source: C TAVERNAGENT details · p. 15, Table 9 · p. 15
  • Across 111 heroes, TAVERNAGENT’s placements correlate weakly with human statistics (Pearson r = 0.13), with relative strengths differing by hero.It performs strongly with heroes people rate as weak and relatively poorly with heroes whose powers require planning across turns.Source: C TAVERNAGENT details · p. 15, Figure 7 · p. 16

D Live deployment

  • The live deployment covered one game and one patch; the AI chose every operation, while a human pressed the named shortcut key without making decisions.Games were played in solo mode on the authors’ account; illegal client proposals were abandoned and corrected before play resumed.Source: D Live deployment · p. 16
  • On the last 20 public-leaderboard records, TAVERNAGENT averages 1.60 placement with 80% first places, ahead of each of the ten highest-rated players.Those players average 2.34 placement and 59% first places together; the leaderboard estimates placements from rating changes using the same measure for all.Source: D Live deployment · p. 17, Table 11 · p. 17

E TAVERNBENCH details

TAVERNBENCH evaluates decisions across scenarios built from real game states, with labels derived from simulator execution and play outcomes. Its details also compare simulator-assisted LLM answers and report the resources used to build the system.

E TAVERNBENCH details

  • Static tasks reshape real game states for scenarios, while strategy and natural-choice tasks come from decisions in the main-exam lobby.Rule tasks execute actions on copied scenes; random-outcome tasks are retained only when reseeded copies agree.Source: E TAVERNBENCH details · p. 18
  • Decision-task labels come from simulated outcomes, keeping choices only when the best option meaningfully beats the median and few options are near-best.An answer is correct when its option is within three win-rate points of the best.Source: E TAVERNBENCH details · p. 18
  • Strategy tasks compare four legal turn plans by playing each out 128 times; ending the turn is never the best plan, and labels are outcomes, not labeller preferences.A task is retained only when the best plan beats the second-best in mean placement by 0.25 places and two standard errors.Source: E TAVERNBENCH details · pp. 18–19
  • A task’s latency estimate combines the median time to the first answer token with answer time at the model’s median output speed.The reported estimates include typical thinking time; measured times can vary with queueing and access path.Source: E TAVERNBENCH details · pp. 19–20
  • Jev answers typed candidate-choice tasks without free-text generation, returning a selected option and probabilities for every option.It answered every task without error in a median 0.26 seconds per task.Source: E TAVERNBENCH details · p. 20
  • TAVERNAGENT answers 92% of play tasks across all 960 benchmark tasks, and 91% on the 368 tasks every system answers.The benchmark spans 40 scenarios; rule-execution scenarios have no agent counterpart because TavernGym executes those rules for it.Source: Table 13 · p. 18, E TAVERNBENCH details · p. 20
  • With TavernGym tools, LLMs close most of the gap to TAVERNAGENT, but errors and untested candidates remain, especially in large purchase-sequence spaces.The evaluation gives five LLMs up to twelve calls per task to test actions and inspect resulting states.Source: E TAVERNBENCH details · p. 20, Table 15 · p. 20

F Compute resources

  • Training the 23 delivered TAVERNAGENT stages used about 1,900 GPU-hours, while other training runs used about 16,700 GPU-hours.Evaluations and analyses used about 1,300 GPU-hours over the 20-day project.Source: F Compute resources · p. 20
Loading 2610.33845v1…