PaperlayerView exact revision on arXiv

Source-linked AI summary

SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire

Scott C. Lowe, Sander W. Keemink

arXiv:2610.12596v1cs.AIcs.CLcs.LG

TL;DR

Long-horizon agent performance is difficult to assess when benchmarks are short or episodic and do not separate prior knowledge, reasoning, and luck. SpireBench evaluates models in the complete Slay the Spire game and finds that strong models match experts in Act 1 but fall behind as runs lengthen, with risk miscalibration and ineffective cross-run memories.

  • Problem

    Agentic benchmarks are mostly short-horizon, and existing game evaluations often omit player decisions or leave stored knowledge, reasoning, and luck difficult to disentangle.

    Source: §1 · p. 1

  • Method

    SpireBench evaluates models playing the complete Slay the Spire game at Ascension 20, with a consistent reskin and comparisons to human and non-LLM references.

    Source: §1 · p. 1, §3 · p. 5

  • Results

    Strongest models win Act 1 as often as expert streamers but fall behind from Act 2, reaching 0–10% Act 4 wins versus 46–65% for streamers.

    Source: §4 · p. 6

  • Takeaways & Limitations

    Models are competent at near-term fights but miscalibrate risk over acts and runs, while their written memories do not compound into better play.

    Source: §5 · p. 9

    The model ordering is measured on one character in a turn-based game from parsed text, and ten seeds cannot reliably order neighboring models.

    Source: §5 · p. 9

Abstract

from arXiv · show

Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck. We introduce SpireBench, in which models play the complete commercial deck-builder Slay the Spire (StS), where early decisions decide long-term success. Across 20 large language models (LLMs) at the game's highest difficulty, frontier models beat heuristic baselines and match the average human player but remain far behind experts. This gap widens as the run lengthens: the strongest models win the first act as often as experts, then fall behind. What separates strong models from weak is how well they play each fight; all of them misjudge risk over a run, preserving health rather than spending it to grow stronger. A comprehensive, self-consistent renaming of every game term shows that part of their performance rests on prior knowledge of the game, and models do not improve from the memories they write across runs, which are often wrong and overly cautious. Our harness is the first to give a model the complete original game, and we compare it with existing harnesses. Overall, SpireBench offers an unsaturated long-horizon benchmark with tools to locate where and why agents fail.

1 Introduction

SpireBench targets the gap between single-step competence and reliable long-horizon performance, where stored knowledge, reasoning, memory, and stochasticity are difficult to disentangle. It evaluates models in the complete Slay the Spire game and finds that they lag experts as runs lengthen, misjudge risk, and fail to improve through cross-run memories.

1 Introduction

  • Existing agentic benchmarks are mostly short-horizon, while game evaluations often omit player decisions or leave knowledge, reasoning, and luck entangled.Reported memory gains usually rely on external checks of what was learned.Source: §1 · p. 1
  • SpireBench uses the complete Slay the Spire game to combine tactical combat, deck construction, and route planning across 56 floors, separating game knowledge from reasoning.The game presents multiple time horizons: turn actions affect fights, deck choices persist across the run, and route decisions trade safety for rewards.Source: §1 · p. 1, Figure 1 · p. 2
  • Frontier models beat rule-based and symbolic baselines and average human players but remain far from experts; run performance falls behind as the horizon lengthens.The strongest models win Act 1 as often as expert streamers, while their Act 4 win rates are 0–10% versus 46–65% for streamers.Source: §1 · p. 2

2 Background

Slay the Spire requires decisions across combat, deck-building, and route-planning horizons, with early choices shaping later success. Existing game and agent benchmarks cover only parts of this setting; related work also motivates testing memory without external verification and using consistent reskins to probe prior knowledge.

2 Background

  • Combat, deck construction, and route choices compound across a run: cards persist, late threats require early preparation, and early risk can supply growth for rising difficulty.A card can be weak now but useful later, or generally good but mismatched to what lies ahead.Source: §2 · pp. 2–3

2.1 Related work

  • Prior Slay the Spire benchmarks omit parts of the game or use lower difficulty and limited models; SpireBench evaluates complete A20 runs across 20 models against human data.MiniStS tests isolated encounters, Orak delegates non-combat floors to a bot, and AgenticSTS uses the sequel at its lowest difficulty.Source: §2.1 · p. 3
  • Many long-horizon benchmarks use short or fixed action spaces, while deck-building changes the action space available later in play.Prior memory studies often use external checks; without one, models can propagate errors, motivating tests on a long, stochastic task without such checks.Source: §2.1 · p. 3
  • The whole-game reskin keeps mechanics while replacing names with a consistent theme, with random and permuted controls to distinguish familiar-name knowledge from reading unfamiliar terms.The comparison is paired within seed.Source: §2.1 · p. 3

3 Methods

The methods run models through a harness for the released game, with turn-level tool interaction, persistent notes, and a consistent thematic reskin. Evaluation uses held-out A20 Ironclad seeds, human and non-LLM references, and measures of acts cleared and floor reached.

3 Methods

  • The harness exposes the released game through JSON states and text commands, keeps a conversation across each floor, and lets models return multiple turn actions with action effects reported back.The design is compared with other harnesses on actions per call, feedback, context, and card-choice handling.Source: §3 · pp. 3–4, Table 2 · p. 5
  • Models may write, overwrite, or delete floor notes, then consolidate them into a persistent bank that enters later run prompts in seed order.Memory mechanisms are tested through ablations.Source: §3 · p. 5
  • A hand-written, internally consistent tabletop-RPG theme replaces every game-specific name to ablate prior game knowledge while retaining the rules and mechanics.Ablations compare the themed names with the original and randomized rename vocabularies.Source: §3 · p. 5
  • Models are evaluated at their highest supported reasoning settings, and Figure 2 plots mean acts cleared against run cost and the Intelligence Index.The cost comparison spans reasoning-effort levels; the index comparison includes highest- and low-effort points.Source: §3 · p. 5, Figure 2 · p. 6
  • Main results use the Ironclad at Ascension 20 on ten held-out seeds, scoring mean acts cleared and final floor reached; ten seeds support broad trends but not reliable ordering of neighboring models.References include three non-LLM policies, four expert streamers, and released telemetry from 206,082 finished runs; the human runs are not seed-matched.Source: §3 · p. 5

4 Results

At Ascension 20, the strongest models outperform average players but remain far behind experts, with the gap widening after Act 1. Their run-level caution and weaker fight play help explain the shortfall; reskinning also reveals reliance on prior game knowledge, while accumulated memories do not improve performance.

Performance and cost; Comparison with human players

  • The strongest models win 90% of Act 1 runs, like streamers, but only 0–10% of full games, versus 46–65% for streamers.They clear more acts than average players but far fewer than the four streamers; only GPT-5.6 Sol kills the Heart, once.Source: §4 · p. 6

How the models spend a run

  • Models spend runs cautiously, resting more and fighting fewer elites than human players; at the Act 1 boss, top models retain 90% of maximum HP versus 53% for streamers.Act transitions restore 75% of missing HP, and human runs that fight more elites and rest less clear Act 2 in 38% of cases versus 15%. The route explains only about a fifth of the streamer gap; fight play explains the rest.Source: §4 · p. 6, Figure 3 · p. 7

Strong models fight better

  • The top four models take 8.2 damage per Act 1 hallway fight, versus 14.0 for the bottom tier, chiefly by converting energy into damage and avoiding wasted blocks.They deal 7.5 versus 5.8 damage per energy spent on attacks, block when no enemy attacks 6% versus 31% of the time, and never miss a lethal play in hand, versus 8.7%.Source: §4 · p. 7

Harness and benchmark comparisons

  • The benchmark’s harness reaches 3.3 more floors and clears 0.15 more acts than Orak’s on the same models and seeds, though the comparison uses eight mostly weaker models.Across public evaluations, other evaluations predict 50% of SpireBench’s variance for a held-out model, making it one of the least substitutable evaluations reported.Source: §4 · p. 7

Memory does not compound

  • Memory consolidation does not improve acts won; final memory banks contain 17% false memories, and models with more errors clear fewer acts.The association is Spearman ρ = −0.67 (permutation p = 0.002, n = 20); carrying notes forward verbatim does slightly better (+0.06 acts).Source: §4 · pp. 7–8

Prior knowledge is part of the score

  • Themed renaming costs 0.14 acts and 11 Act 1 win-rate points, while nonsense names cost 0.01 acts, indicating that familiar game names contribute to performance.Recognition of the original game or replaced entities did not recover the performance cost.Source: §4 · pp. 8–9

5 Discussion

Models handle short horizons competently but fall behind over acts and runs, with risk calibration worsening as outcomes recede. The authors note scope and measurement limits, including a single character and ten seeds per cohort.

5 Discussion

  • Models win the first act as often as streamers but fall behind from Act 2; boss fights decide where each model stops.They win almost every ordinary fight and forecast the next turn well, yet lose more HP per fight than humans and barely sharpen act-survival forecasts.Source: §5 · p. 9
  • Calibration drops sharply with horizon: every model succeeds on at least 87% of turns but falls below 50% between a fight and a run.For most models, performance is already below 50% at the act horizon.Source: §5 · p. 9
  • The authors relate these findings to delayed, noisy outcomes: unverified memory does not compound, while long contexts cost performance.They state that calibration degrades with horizon when outcomes resolve at several distances.Source: §5 · p. 9
  • The evaluation covers one character in a turn-based, parsed-text game; human comparisons are not seed-matched, and ten seeds cannot order neighboring models.Reskin costs are a lower bound because renaming hides game terms but not familiarity with deck-builders.Source: §5 · p. 9

Appendices

Slay the Spire poses long-horizon decisions because choices shape later options and resource trade-offs, while the game mixes known tactical information with stochastic future events. The harness routes game state and actions between models and symbolic players, with an interface built around the original game.

Appendices

  • The appendices list further material on the harness, model settings, seeds, metrics, costs, baselines, ablations, time horizon, and memory banks.The contents also list public benchmarks, agentic residuals, harness-ended runs, and one-conversation-per-floor analyses.Source: Appendices · p. 15

A Why Slay the Spire

  • A card added early recurs in later fights, so small choices can compound across hundreds of decisions before the run's final win-or-loss verdict.Removing cards is limited, and a poor addition can consume an opportunity to remove a starting Strike.Source: A Why Slay the Spire · p. 17
  • Deck-building defines later action options, but added cards can dilute draws; card value depends on the deck and the fights it must face.The player must adapt a plan as random cards and relics arrive, while different encounters reward different strategies.Source: A Why Slay the Spire · p. 17
  • HP, gold, potions, and deck strength form an irreversible resource pool: elite rewards can pay off later, while resting trades an upgrade for health.Players must price present costs against uncertain future demands and distinguish known enemy actions from random draws and rooms.Source: A Why Slay the Spire · p. 17
  • Knowing act-specific elites and bosses can inform deck-building, making memory across runs relevant to this long, stochastic task.The game is turn-based and text-described, mixes skill with luck, and offers route decisions under uncertainty against a large human-player population.Source: A Why Slay the Spire · pp. 17–18

B Harness

  • The harness turns game states into text, routes game components to a model or symbolic player, and maps validated tool calls back to the game.It can optionally reskin in-game names; the figure also depicts cached prompt layers and model planning, forecasting, and note-writing.Source: Figure 5 · p. 18

B.1 Game interface

  • CommunicationModPlus exchanges JSON states and text commands, with snapshots around enemy turns; each floor shares one conversation across combat, rewards, and reflection.Every run uses the same mod list, and isolated profiles ensure seeds—not profile history—select act bosses.Source: §B.1 · p. 18

B.2 Agent harness

This section describes the harness’s screen-specific information, tools, action handling, persistent notes, and renamed terminology, along with its broader card coverage and greater in-fight control than prior harnesses. It also specifies the experiments’ use of fresh seeds and defined cohorts.

B.2 Agent harness

  • The harness batches a turn’s actions into one response, executes them sequentially, and returns each action’s effects and the enemies’ actions with updated state.Prompts show the current screen without recommending a strategy; at act and floor boundaries, the model plans routes, predicts survival, and may update notes.Source: §B.2 · p. 19, Figure 6 · p. 19
  • One action per response required 3.0 more calls per floor, cost 23% more, and cleared 0.07 fewer acts than multi-action play.On the main seeds, the comparison also found no worse choices by index than by name when using one action per response.Source: §B.2 · p. 20
  • Actions target cards and choices by name to avoid stale hand positions during batches; names also express intended choices, though main-seed tests found no worse choices by index.In preliminary runs, weaker models sometimes reasoned to one option but supplied another option’s index; unmatched names are refused with a suggestion.Source: §B.2 · p. 20

B.3 Notes and memory

  • Run notes are rendered as tagged text, and the memory ablation compares consolidated notes, verbatim carryover, no memory, and compact summaries.The same generic guidance asks models to distill insights and record what went well, what went wrong, and what they wish they had known.Source: §B.3 · p. 20

B.4 Knowledge reskin

  • The reskin renames game entities and harness text using effect-evoking placeholders; its mappings vary from original names to themed, nonsense, or misleading names.Permuted mappings change names while retaining rule words left unrenamed by the themed mapping.Source: §B.4 · p. 20

B.5 Model-facing features

  • The harness displays game state and legal choices, reports action effects, and corrects or refuses calls without making strategic choices for the model.Combat information includes enemy intents and hand details, while non-combat screens list choices and relevant costs or effects.Source: §B.5 · p. 21

B.6 What the model receives

  • The system prompt contains the game’s rules, tips, keywords, cards, relics, and potions, and 17 tools are sent on every call.Before any memory existed, the prompt was 30,715 characters; by the tenth seed it had reached 227,089 characters.Source: §B.6 · p. 24
  • At combat turns, the model receives the previous turn’s action results; at floor end it reflects and forecasts, and at run end it receives the outcome, final state, maps, routes, floor history, and notes.Source: §B.6 · pp. 24–26

B.7 Coverage compared with Orak and Bottled AI

  • Orak delegates map and noncombat choices to a rule bot and limits model control during fights, while this harness puts those choices to the model and interrupts action queues when the state changes.Orak also omits orbs, stance, potions, and card piles from the model’s view, cannot play the Watcher, and never collects the Ruby Key needed to reach Act 4.Source: §B.7 · p. 26
  • Every card is supported by this harness, whereas Orak delegates choices opened by 33 cards to its rule bot and Bottled AI cannot play cards outside its combat simulator.The comparison counts whether a card can be played, not how well it is used.Source: §B.7 · p. 26

C.1 Models and reasoning settings

  • Reasoning settings are kept at the requested level or the nearest supported level above it, with provider-specific handling for models that lack a reasoning setting or use a thinking budget.Across 1,908 finished runs and 274,306 model calls, no response was truncated by the 65,536-token ceiling or a lower model limit.Source: §C.1 · p. 28

C.2 Seeds and cohort selection

  • Every main result, ablation, and difficulty level uses the same ten newly minted seeds, replacing preliminary seeds to reduce the risk of memorization through API traffic.Each table cell uses one checked cohort: runs must match on configuration and seed list, be finished, and use the latest run per seed; memory cohorts must form an unbroken chain.Source: §C.2 · p. 28

C.3 Metrics and statistics

This appendix defines a progression-based performance metric and its statistical and pricing conventions, then specifies the non-LLM policies used as reference baselines.

C.3 Metrics and statistics

  • Mean acts cleared is the main metric because it rewards reaching the next act, while floor statistics describe within-act progress; models are more cautious than humans and win runs less often.Mean acts cleared can still be misaligned with winning the game, but models do not win often enough to rank them on Act 4 win rate alone.Source: §C.3 · p. 28
  • Paired model comparisons use exact sign-flip permutation tests, model correlations use permutation tests, and within-run contrasts use exact sign tests.Analyses group the 20 models into tiers of four, eight, and eight by acts cleared; log-based analyses cover 19 models because Gemma 4 E4B lacks full logs.Source: §C.3 · p. 29

C.4 Cost

  • Hosted-model costs use provider list prices with adjustments for cache writes and time-varying rates; self-hosted tokens are priced using Artificial Analysis rates.OpenAI cache writes are inferred and may be underestimated, while DeepSeek costs use peak rates and are upper bounds.Source: §C.4 · p. 29

C.5 Non-LLM baselines

  • The non-LLM baselines include deterministic leftmost-choice and uniformly random policies, plus Bottled AI’s hand-written symbolic Ironclad player, run with and without its Heart strategy.The simple policies drink every usable potion at the first opportunity; Bottled AI sees no game names.Source: §C.5 · pp. 29–30
  • Bottled AI’s Heart strategy adds key collection and route rewards, while the latest released implementation stops at Act 3 and cannot face the final boss.The Heart strategy derives from an open pull request; it also adds Act 4 combat knowledge and other combat preferences.Source: §C.5 · p. 30
  • At A20, the Heart-seeking and standard Bottled AI variants play identical games on all ten seeds; at lower ascension, the Heart variant performs worse because it forgoes positive boons.Neither variant reaches Act 3 on the ten main seeds.Source: §C.5 · p. 30

C.6 Ablations and analyses

The appendix describes controlled comparisons of harnesses, memory, reskinning, and prompt materials, alongside analyses of cost, public benchmarks, and tool use.

C.6 Ablations and analyses

  • Harness variants are compared on matched model-seed pairs, while efficiency uses like-for-like floors and accounts for runs stopped by harness failures.The variants include one-action interfaces, continuous conversations, and Orak; each ablation otherwise keeps the main-table reasoning setting unless stated otherwise.Source: §C.6 · pp. 30–31

Prompt material.

  • Prompt-material comparisons add enemy statistics, a guide, or streamer transcripts to the cached prompt, while reskin variants separate hiding game names from supplying coherent replacements.The guide and transcripts are renamed along with the rest of the prompt; difficulty-sweep results are reported in Appendix F.Source: Prompt material · p. 31
  • Cost frontiers plot mean acts cleared against run cost, cost per floor, or thinking tokens per floor, with a point for each model and reasoning level.A point is on the Pareto front when no other point clears at least as many acts for no more spend; the front uses cohorts with all ten seeds.Source: Prompt material · p. 31
  • Public-benchmark analysis compares acts cleared with published scores and tests how well other evaluations predict SpireBench for held-out models.The Artificial Analysis Intelligence Index is compared separately; the analysis also reports within-reasoning-setting correlations.Source: Prompt material · p. 31
  • Memory analyses compare consolidation, verbatim note retention, and no persistence, and measure bank growth by the system-prompt length at each new run.Compact arms overwrite one prose summary; the memory bank is measured relative to the empty first-run bank, through run 9.Source: Prompt material · pp. 31–32
  • Tool-call failures are measured per 100 calls across four error types, separately from playing skill, and the harness comparison reports the same rate.The first three error types are reported to the model and do not end a run.Source: Prompt material · p. 32

D Human baselines

The human baselines combine expert run records with population telemetry, which also permits comparisons across play-time bands. The appendix cautions that play time is not a clean measure of skill and introduces later comparisons of agent strategy and play quality.

D Human baselines

  • Expert comparisons draw on four streamers, including 687 automatically logged Ironclad A20 runs for Baalorlord and contiguous archive samples for three others.The streamer samples differ in collection method and period, so the figures do not all represent the same kind of long-run average.Source: D Human baselines · p. 32, Table 8 · p. 33
  • The population cleared Act 1 in 47% of runs and killed the Heart in 3.5%, based on filtered A20 Ironclad telemetry.The analysis retained 206,082 finished runs after excluding other run types, builds, and abandoned runs.Source: D Human baselines · p. 32, Table 9 · p. 33
  • Performance rises slowly with recorded play time: the 50–100-hour bin cleared 0.67 acts and killed the Heart in 2.4% of runs, versus 0.97 acts and 6.8% in the 1,500–2,000-hour bin.Performance barely changes from 25 to 1,000 hours, then rises in the bins above 1,000 hours; save-profile resets can place experienced players in the lowest bin.Source: D Human baselines · pp. 32–33, Table 10 · p. 34, Figure 7 · p. 34
  • Baalorlord’s sample includes “mastery” runs, but including them does not significantly change his overall performance.These runs involve trying to win with two copies of a particular card.Source: D Human baselines · p. 32
  • Play time is a lower bound on experience, measures time rather than skill, and resets with a new save profile; small high-hour bins were merged.The 2,000–3,000-hour and 3,000-hours-and-above bins were combined.Source: D Human baselines · pp. 32–33

E Strategy and play quality

  • The appendix analyzes Table 1 runs to ask why models fall behind human experts and why stronger models outperform weaker ones; model tiers, logged outputs, and tests follow Appx. C.3.Source: E Strategy and play quality · p. 34

E.1 Where agents die: agents against humans

Agents die more often than human players on several reached floors, while their Act 1 choices leave them healthier but with less-developed decks. Across models, route choices do little to distinguish strength; fight performance is more associated with acts cleared.

E.1 Where agents die: agents against humans

  • At the Act 1 boss, 41% of agents died after reaching it, versus 32% of humans; on floor 7 the rates were 12% versus 4%, and on floor 22 they were 20% versus 6%.Act 1 boss deaths account for 27% of agent deaths; Table 12 defines the floor-level death shares and death rates among runs that reached each floor.Source: §E.1 · p. 34, Table 12 · p. 35
  • Agents reached the Act 1 boss at 82% of maximum HP versus 62% for humans, but rested at 68% of campfires versus 23% and fought 1.25 elites versus 1.96.They also took a card after almost every fight, leaving larger decks, but had fewer upgraded cards and relics at death than humans dying on the same floor.Source: §E.1 · pp. 34–35, Table 11 · p. 35
  • These agent-human differences are descriptive rather than matched comparisons: agents used a reskinned game on at most ten held-out seeds, while humans played many other seeds.All human A20 players had previously won at A19 with the Ironclad; several measurements also rely on reconstructed or approximate records.Source: §E.1 · p. 35
  • The strongest models ended Act 1 with a median 90% of maximum HP, while streamers had 53%; models rested at 66–75% of rest sites across visits, versus 15% for streamers.Models commonly rested even below 80% maximum HP, while streamers more often upgraded cards; the streamers’ runs were not matched to model seeds.Source: §E.1 · p. 35
  • In Act 1, models fought fewer elites and rested more often than humans, then fell behind streamers in relics and carried fewer upgrades and larger decks.The grouped route analysis compares models with telemetry and four streamers across acts; Figure 8 specifies the run counts and map-floor measures.Source: §E.1 · pp. 35–36, Figure 8 · p. 36
  • Across 16 models, hallway-fight HP change correlated with acts cleared at ρ=+0.62 (p=0.01), while correlations for elites, shops, resting, and skipped card rewards were weaker.The top four lost 2.4 HP per hallway fight versus 6.5 HP for the bottom eight; the figure plots the choice measures against mean acts cleared.Source: §E.1 · p. 36

E.2 Conservative Act 1 play among humans

Human runs with an Act 1 profile resembling the models’ were less likely to clear later acts, although the authors qualify these comparisons as associations. In the models’ own runs, elite fights were especially dangerous before the Act 1 boss.

E.2 Conservative Act 1 play among humans

  • Among humans who beat the Act 1 boss, model-like runs cleared Act 2 in 15.5% of cases versus 38.0% for runs with two or more elites and fewer rests, and cleared Act 3 in 3.4% versus 18.6%.The model-like profile covered 3.2% of human runs and 36% of model runs; the Act 2 gap persisted across play-time bands.Source: §E.2 · pp. 36–37
  • Within play sessions, each Act 1 elite multiplied the odds of clearing Act 2 by 1.38, while each rest multiplied them by 0.85; model-like runs cleared Act 2 7.3 points less often than other runs in the same session.These are associations: weak runs may skip elites or rest because they are weak, and the telemetry does not record which elites the map offered.Source: §E.2 · p. 37

E.3 Risk in the models' own runs

  • Elite rooms accounted for 37 of 65 pre-boss Act 1 deaths, with a 17.8% death rate per elite fight versus 2.7% per ordinary fight.The rate was higher for 15 of 18 models; the top four died in 0–9% of their elite fights, so avoiding elites can be partly right for models that fight badly.Source: §E.3 · p. 38

E.4 Learning over the chain

Across the chain, memory produced no measurable improvement and tended toward caution; deck preparation and combat play distinguished stronger from weaker models. Forecasting and difficulty analyses also examine performance by decision horizon and run endpoint.

E.4 Learning over the chain

  • Memory had no measurable effect on rest-site resting, Act 1 elites, upgrades, boss HP, floor, or acts; its measured direction was toward greater caution.The added effect was −0.07 on the share of rest sites spent resting and −0.32 Act 1 elites; tests had p≥0.29, with low power.Source: §E.4 · p. 38
  • Of 1,374 final-bank memories, 119 advised caution and 17 advised risk; only four strongest models stated the act heal correctly.Of 209 late-act rest-site decisions reasoning about the heal, models rested in 21 (66%), versus 77% for others.Source: §E.4 · p. 39

E.5 Deck building and boss preparation

  • 13% of the 75-point top–bottom Act 1 boss-win gap was explained by a human-fitted model and 24–29% by a model-fitted one, leaving 52–57 points unexplained.The residual was in the fight or in state information the models omitted.Source: §E.5 · p. 39

E.6 Combat

  • The top four took 6.7 less damage than the bottom tier in paired hallway encounters, in 0.99 fewer turns while dealing 5.3 more damage per turn.On floors 1–6, the gaps were 2.6 HP and 0.82 turns; about half the damage difference came from extra turns and half from damage per turn.Source: §E.6 · p. 39

F Difficulty and where runs end

  • Figure 12 varies pinned Ascension level by separate cohort, while Figure 13 breaks down each agent’s run-ending act, showing why mean floor alone does not describe run outcomes.The figure captions note that agents with similar mean floors can differ in whether they reliably reach the next act or split between early deaths and deep runs.Source: F Difficulty and where runs end · p. 42, Figure 12 · p. 42, Figure 13 · p. 43

G Calibration

  • Models give end-of-floor probabilities for act survival and run wins, plus optional HP forecasts; scores adjust for distance to resolution to account for prediction timing.The scoring standardizes errors to a common mix of distances because stronger models make more predictions close to an act’s end.Source: G Calibration · p. 44

Results.

Success falls sharply between fights and runs, with boss fights marking a major hurdle. Forecasts also vary in how much they improve as outcomes approach.

Results.

  • Fight-end HP forecasts sharpen steeply as the fight ends, while act-survival forecasts barely sharpen before the act resolves; within-floor forecast error tracks play more closely than end-of-floor calibration.The paper notes that calibration, like play, degrades with distance from the outcome.Source: Results · p. 44, Figure 15 · p. 46, Figure 16 · p. 47

H.1 Time horizon

  • 18 of 20 models won 93–100% of ordinary Act 1 fights, but won boss fights only 0–81% of the time; the challenge was a step rather than a gradual decline across task scales.All models succeeded on at least 87% of turns and 77% of fights; human players also showed a step, clearing 43% of acts entered.Source: §H.1 · p. 48

H.2 Public benchmarks and reasoning effort

SpireBench tracks public benchmark performance but measures something those evaluations do not fully capture. The analyses also distinguish benchmark prediction, harness failures, and how much run history models see.

H.2 Public benchmarks and reasoning effort

  • Acts cleared correlated with the Artificial Analysis Intelligence Index at r=0.80 at highest effort and r=0.91 at low effort, while within-model Index changes poorly tracked acts lost (ρ=0.06).The authors conclude SpireBench ranks models much as the public suite does but measures something the suite does not already capture.Source: §H.2 · p. 48

H.3 Agentic residuals

  • Terminal-Bench 4.0 predicted game performance better than Humanity’s Last Exam (r=0.89 versus 0.81) and added information when controlled for (partial r=0.64, p=0.006, n=18).Other reported agentic evaluations predicted less well and added nothing.Source: §H.3 · p. 48

H.4 Runs ended by the harness

  • Orak’s harness ended 22 launches on the tested seeds and its failures ended 11 of 70 seeds; SpireBench’s harness ended none on those arms and 6 of 2,432 launches overall.The paper counts harness-ended runs as losses by the harness and reports them separately from model mistakes.Source: §H.4 · pp. 48–49

H.5 One conversation per floor

  • The conversation comparison holds the task fixed while changing how much history is shown: one conversation spans the run, while the alternative starts a fresh conversation each floor with notes and history carried forward.The fresh-per-floor setup is intended to avoid both a long context used poorly and earlier mistakes kept in view.Source: §H.5 · p. 49

H.6 What the memory banks hold

Consolidating model-written notes did not raise acts cleared, and many stored memories were inaccurate. Reskin results suggest familiar game names contribute to stronger models’ performance, but the cost was not tied to model family.

H.6 What the memory banks hold

  • Consolidating notes into a memory bank did not raise acts cleared; carrying notes forward verbatim gained +0.06 acts, +1.6 floors, and +5 points of Act 1 win rate, suggestively rather than conclusively.17% of tenth-run memories were false; 78% of rules stated without condition were false, and more self-generated errors correlated with fewer acts cleared (ρ=−0.67, p=0.002, n=20).Source: §H.6 · pp. 49–50

H.7 Reskin cost by model family

  • The reskin cost was not detectably associated with model family (p=0.29 for acts; p=0.12 for floors), and it appeared only among models clearing 0.6–0.8 acts with real names.The authors suggest familiar names can help when their usual meaning fits and mislead when it does not.Source: §H.7 · p. 50

H.8 Recognition of the game under the reskin

Models often recognize Slay the Spire beneath the reskin, but recognition does not restore the knowledge tied to the original names. Across runs, their decisions and memories reveal uneven learning and recurring errors.

H.8 Recognition of the game under the reskin

  • 125 of 190 themed runs name Slay the Spire or a replaced entity, but recognition does not predict better performance or recover the cost of reskinning.Recognition is usually in reasoning and often on the first seed; within a model, runs recognizing the game in their first six floors clear no more acts.Source: §H.8 · pp. 50–51

H.9 Deck building does not fail by accumulation

  • Models raise card-skip rates across acts, yet 41% of runs reaching Act 2 never skip a card, and skip-rate adaptation does not predict acts cleared.Model skip rates rise from 0.09 in Act 1 to 0.25 and 0.42 in later acts, while human rates are 0.21, 0.39 and 0.53.Source: §H.9 · p. 51

H.10 Where models go wrong

  • In routine decisions, weaker models spend more reasoning tokens, and Grok 4.6 sometimes calls a different action than its reasoning names.Grok's tool call differs from its stated action in 20 of 68 such decisions; 15 of 20 models sometimes end turns with a different energy amount than declared.Source: §H.10 · p. 52
  • Memory banks accumulate rather than curate: models rewrite or re-save notes, rarely delete them, and sometimes omit or overwrite consequential game facts.Across 200 runs, models create 1,404 memories, rewrite 1,080, re-save 863 unchanged, and delete 30; 15 of 20 never delete one.Source: §H.10 · p. 52, Table 20 · p. 52

I Ablation results in full

The appendix lays out ablations of the harness, reasoning effort, memory, reskin, and prompt material, alongside cost and reasoning-budget views. It specifies pooled and per-model comparisons, but most supplied passages describe their scope rather than their outcomes.

I.1 Harness

  • The harness ablation compares paired agents against condition-specific baselines; Figure 17 shows mean acts cleared and sign-flip tests, while Table 25 gives per-model results.Figure 17 includes only agents with all 10 seeds in both conditions; Orak is shown for four models with all 10 seeds scored.Source: Figure 17 · p. 56, Table 25 · p. 57

I.2 Reasoning effort

  • Higher reasoning effort generally improves performance within a model, but exceptions occur, and thinking-token count shows no clear overall relationship with performance across models.GPT-5.6 Luna peaks at high effort, with reduced performance at xhigh and max; GPT-6 Astra at low effort is the best model when thinking tokens are constrained.Source: §I.2 · p. 57, Figure 18 · p. 58, Table 26 · p. 59

I.3 Memory

  • The memory ablation compares within-run notes and several between-run memory policies on matched runs, with bank growth tracked across a 10-run chain.The pooled comparison uses the same model and seed across conditions; usage comparisons also match room type and occurrence.Source: Figure 19 · p. 60, Table 27 · p. 61, Table 28 · p. 61

I.4 Knowledge reskin

  • The knowledge-reskin ablation compares real names, themed names, nonsense names, and two name-permutation conditions on matched runs.All conditions use XML notes, a read-write memory bank, and generic guidance.Source: Table 29 · p. 62, Table 30 · p. 63

I.5 Prompt material

  • The prompt-material ablation adds an enemy compendium, a renamed Reddit strategy guide, or renamed Jorbs transcripts to the themed-reskin baseline.These arms use XML notes without a memory bank; the evaluation seeds are disjoint from those in Table 1.Source: Table 31 · p. 63, Table 32 · p. 64

J Cost and reasoning

  • The cost appendix presents cost frontiers against reasoning budget and explains how cost estimates are priced.Figure 18 relates acts cleared to thinking tokens per floor; Figures 20 and 21 show named performance points against reasoning effort and cost per floor.Source: J Cost and reasoning · p. 64, §J.1 · p. 64, Figure 20 · p. 65

J.1 Cost

  • The cost figures distinguish whole-run cost from efficiency per floor, with Figures 20 and 21 plotting the same performance points against reasoning effort and cost per floor.Figure 21 uses log-scale cost per floor; self-hosted models use Artificial Analysis headline rates, while peak-rate list prices are upper bounds.Source: §J.1 · p. 64, Figure 20 · p. 65, Figure 21 · p. 66

J.2 Time and cost of a run

  • Run-cost reporting covers wall-clock time, dollars, calls, tokens, and tool-call errors; cohort seeds run in series against one game instance.Wall-clock time also depends on provider throughput on the day.Source: §J.2 · p. 67, Table 33 · p. 67

Related papers

Loading 2610.12596v1…