PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

Yibo Li, Jinhang Qiu, Zhi Zheng, Qianyun Guo, Jiaying Wu, Shuo Ji, Bryan Hooi

arXiv:2610.08215v1cs.AI

TL;DR

Existing evaluations make it difficult to distinguish knowledge acquired through interaction from prior knowledge. Learn2Play Bench tests this with newly designed hidden-rule text games; experiments find that complete interaction histories can support stronger learning than summaries, while top human players achieve higher peak scores than evaluated agents.

  • Problem

    Existing evaluations make it difficult to distinguish newly acquired knowledge from prior knowledge, complicating assessment of whether agents learn through interaction.

    Source: §1 · p. 1, §6 · p. 9

  • Method

    Learn2Play Bench uses newly designed text-based games with hidden rules, deterministic feedback, repeated episodes, and varied instances to evaluate learning through interaction.

    Source: §1 · p. 2

  • Results

    Raw interaction histories can support stronger learning than summarized rules or strategies, while top-performing human players achieve higher peak scores than evaluated agents.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study identifies preserving original interaction evidence and comparing experiences across situations as directions for future agent-learning methods.

    Source: §5 · p. 9

    Agents may stop testing alternatives after finding a partly effective strategy and may fail to plan actions around what later situations require.

    Source: §4.2.3 · p. 7

Abstract

from arXiv · show

Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge. Existing benchmarks have sought to evaluate this ability, but they primarily evaluate tasks whose rules are provided in the instructions or already familiar to pretrained models, making it difficult to distinguish learning from interactions from reasoning with existing knowledge. To address this, we introduce Learn2Play Bench, a benchmark of newly designed text-based games, whose rules are novel or counterintuitive, requiring agents to acquire knowledge through interaction rather than rely solely on pretrained knowledge. These games provide reproducible feedback and automatic scoring, enabling controlled evaluation of learning across repeated attempts. We also vary game instances to test whether agents can apply what they have learned to new situations. Therefore, we evaluate how backbone models, self-evolving methods, and agent harnesses affect agents' learning ability, revealing three findings: (1) Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies. (2) Human agent gap: Top-performing human players achieve higher peak scores than the evaluated agents. Human explore more varied strategies, and repeat actions less. (3) Harness matters: With the backbone fixed, changing the harness can improve performance while reducing estimated inference cost. Together, these findings provide insights into how LLM agents learn from experience and suggest directions for future work to improve their learning ability. Project website: https://liushiliushi.github.io/learn2play-bench-website/

1 INTRODUCTION

Learn2Play Bench evaluates whether LLM agents learn hidden, unfamiliar game rules through interaction and apply that knowledge in later episodes. Its findings compare learning methods and harnesses with human behavior, highlighting both progress and persistent limitations.

  • Existing evaluations can confound learning with pretrained knowledge, and differing models, tasks, and protocols make learning behavior difficult to compare.Source: §1 · pp. 1–2
  • Learn2Play Bench uses newly designed text games with hidden rules, deterministic feedback, and automatic scoring; reshuffled instances test transfer to new situations.Agents discover rules by acting and observing feedback, without updating model weights.Source: §1 · p. 2
  • Raw interaction histories can outperform rule or strategy summaries, while agents may learn rules yet fail to plan effective strategies.In HAUNTED INN, an agent learned to assign guests to safe rooms but left a later guest with nowhere safe to stay.Source: §1 · p. 2
  • Top-performing human players reach higher peak scores than evaluated agents, explore more varied action sequences, and recover more often after setbacks.Source: §1 · p. 2
  • Changing the harness can improve performance and learning while sometimes reducing inference cost; changing instances slows learning more, especially for weaker backbones.The reported cost reductions come from fewer model calls and less repeated context.Source: §1 · p. 2

2 RELATED WORK

Prior work develops agents that adapt from experience and benchmarks that measure repeated or interactive learning. Learn2Play Bench evaluates whether agents can discover hidden rules in newly designed text games, while comparing backbones and learning methods under controlled conditions.

Self-evolving agents

  • Self-evolving agents reuse feedback through reflections, retrieved trajectories, and executable skills, while later approaches evolve workflows, playbooks, prompts, code, or adaptation policies.Examples include Reflexion (Shinn et al., 2023), ExpeL (Zhao et al., 2024), Voyager (Wang et al., 2024), and Meta-TTL.Source: §2 · pp. 2–3
  • The study assesses existing methods rather than proposing a new adaptation method, comparing backbones in a common harness and self-evolving methods with each backbone fixed.Its evaluation focuses on learning without weight updates.Source: §2 · p. 3

Benchmarks for self-evolving agents

  • Repeated-task benchmarks test whether experience improves later behavior, but familiar tasks can confound learning with prior knowledge; interactive alternatives include AutoEnv, J-TTL, and ARC-AGI-3.J-TTL uses Jericho games whose public content may appear in pretraining, while ARC-AGI-3 also depends on visual capabilities (He et al., 2025; Foundation, 2026).Source: §2 · p. 3
  • Learn2Play Bench uses newly designed text-based games with hidden rules to test whether agents discover unknown rules through actions and feedback, including across changed instances.Comparing fixed and changing instances tests whether agents reuse experience when the visible situation changes.Source: §2 · p. 3

3 EVALUATION DESIGN

Learn2Play Bench uses manually designed text-based games with hidden, sometimes counterintuitive rules to test whether agents learn from repeated interaction. Its design supports automatic, reproducible scoring and tests transfer across changed instances.

Rule novelty

  • Novel and sometimes counterintuitive hidden rules require agents to learn which actions succeed from game feedback rather than rely on familiar game knowledge.For example, stealing a colleague’s lunch, rather than working hard, helps the player get promoted in one game.Source: §3 · p. 4

Evaluation protocol

  • Automatic scoring enables large-scale evaluation without human grading, while deterministic outcomes allow replay, matched comparisons, and mistake inspection.The five challenge games use a ten-episode scoring window; the remaining normal games use a five-episode window.Source: §3 · p. 4

Controlled variation across episodes

  • Re-shuffled games vary the visible situation across episodes while preserving the core rule, testing whether agents transfer what they learned.The ten fixed games repeat the same instance across episodes, while the ten re-shuffled games change the instance.Source: §3 · p. 4

Benchmark construction

  • The benchmark contains 20 game templates that generate multiple instances, with ten fixed games and ten re-shuffled games that vary the visible instance while preserving the core rule.Agents receive a goal and available actions, interact in text to discover rules, and receive a final score; they can use experience from earlier episodes.Source: §3 · p. 4

4 EXPERIMENTS

Experiments evaluate agents through repeated black-box episodes, measuring both achieved performance and improvement from experience. They compare model backbones, experience-retention methods, harnesses, humans, transfer to reshuffled instances, and inference costs.

4.2.1 BACKBONE COMPARISON

  • Opus 5 leads backbone Max and Mean, while Qwen 3.8 Max leads LG and LS, separating peak and average performance from improvement across episodes.Interaction logs show Opus 5 and Qwen 3.8 Max test hidden-rule hypotheses and revise them when feedback conflicts, unlike weaker models that may repeat actions despite new observations.Source: §4.2.1 · p. 5, Table 1 · p. 6

4.4 RQ3: HOW DO HUMANS AND AGENTS LEARN DIFFERENTLY?

  • Human Top-1 achieves higher Max than the evaluated models and self-evolving methods, while human episodes show more strategy changes than agent episodes.Human versus agent action-sequence similarity is 0.54 versus 0.66; after a lower-scoring episode, humans reach a new personal best 33% of the time versus 22% for agents. Humans score higher in GEMFORGE and GRAPEVINE, while the strongest OpenCode backbone scores higher in REDEYE and MODEL AUDITOR; some humans reduce or stop testing in later episodes.Source: Table 1 · p. 6, §4.4 · pp. 7–8

4.2.2 SELF-EVOLVING METHOD COMPARISON

  • MEMORY, which retains raw interaction histories, leads all four metrics with Kimi K3, outperforming more elaborate self-evolving methods.The authors argue raw histories preserve evidence for revising interpretations, whereas reflection or memory processing can perpetuate incomplete rules; an EvoTest–Opus 5 run wrongly ruled out a successful GEMFORGE combination.Source: §4.2.2 · p. 6, Table 1 · p. 6

4.2.3 WHAT LIMITS FURTHER IMPROVEMENT?

  • Agents can stop exploring after finding a partly successful strategy, or know relevant rules yet fail to plan actions around future constraints.In GEMFORGE, an Opus 5 run repeats known recipes after concluding higher-value gems are impossible; in HAUNTED INN, it assigns rooms greedily and leaves no safe room for a later guest.Source: §4.2.3 · p. 7

4.3 RQ2: HOW DOES THE AGENT HARNESS AFFECT LEARNING?

  • With the backbone fixed, Claude Code beats OpenCode on all four metrics with Opus 5, and Codex does so with GPT-5.6-SOL.The paper suggests models may perform better on harnesses they were trained on and that are adapted to their capabilities (MacCallum & Fioca, 2026, Shihipar, 2026).Source: Table 2 · p. 7, §4.3 · p. 7

4.5 RQ4: HOW WELL DOES LEARNED KNOWLEDGE TRANSFER TO NEW GAME INSTANCES?

  • When instances change but rules remain fixed, Opus 5 and Kimi K3 show smaller learning-slope drops than Sonnet 5 and the Gemini Flash models.Opus 5 applies a rule learned from earlier SYNERGYCORP bosses to new bosses, whereas Sonnet 5 and Gemini 3.5 Flash test each new boss again; agents with lower reshuffled learning slopes may repeat instance-specific choices such as menu positions.Source: Table 3 · p. 8, §4.5 · p. 8

4.6 RQ5: DO HIGHER INFERENCE COSTS LEAD TO BETTER PERFORMANCE?

  • With OpenCode, Kimi K3 achieves higher performance at lower overall cost than Claude Sonnet 5 despite higher token prices, and better-suited harnesses can improve performance while reducing cost.Kimi K3 uses 0.56M tokens per episode versus Sonnet 5’s 0.73M; OpenCode with Kimi K3 and Claude Code with Opus 5 also outperform the compared self-evolving methods at lower estimated cost. OpenCode reduces repeated input and model calls by executing several actions from one response and removing repeated feedback.Source: §4.6 · p. 8

5 DISCUSSIONS AND FUTURE DIRECTIONS

The discussion identifies exploration, experience storage, and transfer across changing environments as open challenges for improving LLM agents’ learning. It proposes testing new strategies, retaining evidence behind learned rules, and comparing experiences to distinguish shared rules from instance-specific details.

How should agents balance exploration and exploitation?

  • Human Top-1 achieves a higher Max score than every evaluated agent configuration, motivating better balance between exploration and reusing successful strategies.Humans try more varied strategies across episodes, while agents more often reuse strategies that worked before; future work should test new strategies even when current ones score highly.Source: §5 · pp. 8–9

How should agents store learned rules?

  • With Kimi K3, MEMORY outperforms other self-evolving methods overall, and retaining full interaction histories may help agents revise rules drawn from limited evidence.In GEMFORGE, EvoTest’s failure with one gem combination was turned into a rule against testing untried combinations; future methods could link rules to their underlying interactions.Source: §5 · p. 9

How can agents learn what stays constant as environments change?

  • Changing surface details can reduce agents’ learning slopes even when underlying rules remain unchanged, motivating methods that compare situations and separate shared rules from instance-specific details.Future methods could also check when learned rules still apply as surroundings, resources, or people change.Source: §5 · p. 9

6 CONCLUSION

Learn2Play Bench evaluates how LLM agents learn from interaction in unfamiliar environments using newly designed text-based games with hidden rules. Experiments find that elaborate self-evolving methods do not always help, top-performing humans outperform evaluated agents, and harnesses can improve performance while reducing task costs.

Conclusion

  • Learn2Play Bench uses newly designed text-based games with hidden rules that agents discover through interaction without updating model weights.The benchmark aims to distinguish newly acquired knowledge from prior knowledge.Source: §6 · p. 9
  • More elaborate self-evolving methods do not always improve learning.Source: §6 · p. 9
  • Top-performing human players achieve higher peak scores than the evaluated models and self-evolving methods, while exploring more varied strategies.Source: §6 · p. 9
  • Agent harnesses can improve performance while reducing task costs.Source: §6 · p. 9

LIST OF APPENDICES … A.2.1 INSTANCES, EPISODES, AND TRIALS

The appendices define games that test learning rules from feedback and applying them across episodes, then specify how evaluation scores and trial protocols are constructed. They distinguish re-shuffled games, which require transfer to new instances, from fixed games, which allow refinement of a recurring instance.

A.1 GAMES AND LEARNING TASKS

  • Games test whether players discover useful rules from feedback and apply them in later episodes, rather than assigning separate ability scores.Detailed per-game results appear in Appendix D.Source: §A.1 · p. 15
  • Re-shuffled games change visible situations while preserving underlying rules; fixed games repeat an instance so players can refine maps, recipes, or action sequences.Players must transfer learning to new menus, guests, or other instances in re-shuffled games.Source: §A.1 · p. 15

A.2 EVALUATION AND SCORING

  • Scores are computed by normalizing each episode’s raw score, calculating metrics within each game, and averaging across games.Human record selection is described in Appendix B.3, and cost calculations in Appendix B.5.Source: §A.2 · p. 15

A.2.1 INSTANCES, EPISODES, AND TRIALS

  • Each configuration uses pinned game instances and three trials per game; trials contain five episodes for normal games and ten for challenge games, with aggregate results reported alongside trial standard deviations.Appendix C reports aggregate results and Appendix D gives per-game results.Source: §A.2.1 · p. 15

A.2.2 FROM RAW SCORES TO PER-GAME METRICS … B.3.2 CALCULATING SCORES AND LEARNING CURVES

The appendices define how scores, game-level aggregates, and human records are constructed, then supplement the research-question results with learning-curve and interaction examples. The examples illustrate both feedback-based rule testing and ways that incomplete or processed experience can constrain exploration.

A.2.2 FROM RAW SCORES TO PER-GAME METRICS

  • Scores are normalized against each game’s reference, and Max averages trial-level peak terminal scores while Mean, LG, and LS use episode-wise mean scores.The other metrics average scores, compare late with early scores, or measure a linear trend per episode, respectively; formulas are defined in Section 4.1.Source: §A.2.2 · p. 16

A.2.3 AVERAGING ACROSS GAMES AND PLOTTING CURVES; A.2.4 REFERENCE SCORES

  • Challenge games receive weight 3 and other games weight 1, while aggregate curves interpolate each game to ten positions across its scoring window.Both game groups have total weight 15; interpolation applies only to aggregate figures, not Max, Mean, LG, or LS. Reference values are classified as exact, partial, or estimated according to their supporting evidence.Source: §A.2.3 · p. 16, §A.2.4 · p. 16

B ADDITIONAL RESULTS BY RESEARCH QUESTION; B.1 RQ1: HOW EFFECTIVELY DO AGENTS LEARN?; B.1.1 EVIDENCE OF RULE DISCOVERY

  • The RQ1 appendix adds GPT-5 Mini method comparisons, three-backbone averages, four-backbone Memory curves, and per-game rule-discovery records.Discovery labels are retained per game rather than aggregated into overall percentages because their criteria mix complete-rule discovery with successful execution.Source: B ADDITIONAL RESULTS BY RESEARCH… · p. 17, §B.1 · p. 17, Table 7 · p. 17

B.1.2 EXPLORATION AND PLANNING CASES; B.1.3 HOW PROCESSED EXPERIENCE CAN RESTRICT EXPLORATION

  • In GemForge and room-placement cases, agents stop testing after an incomplete belief or revise plans after recognizing that greedy allocation leaves no safe room.The GemForge example shows an untested fusion restriction generalized from one failure; the room-placement scores are 100 and 115, but changing guest lists prevent treating them as a controlled score improvement.Source: §B.1.2 · p. 18, §B.1.3 · p. 18

B.1.4 USING FEEDBACK INSTEAD OF REPEATING MENU POSITIONS

  • OpenCode–GPT-5 Mini repeats one POISONER menu position, while OpenCode–Kimi K3 compares feedback to test which dish properties matter.These recorded cases illustrate different uses of feedback, not how often either behavior occurs across trials.Source: §B.1.4 · p. 18

B.2 RQ2: HOW DOES THE HARNESS AFFECT LEARNING?

  • The harness comparison holds the backbone fixed and separates trials while preserving earlier episode interactions within each trial.Claude Code disables persistent auto-memory, Codex uses --ephemeral, and OpenCode starts a new session; Table 2 reports scores, with curves in Appendix D.Source: §B.2 · p. 18

B.3 RQ3: HOW DO HUMANS AND AGENTS LEARN DIFFERENTLY?; B.3.1 SELECTING HUMAN RECORDS

  • Human participants were college students playing without AI tools or code; per game, Top-1, Top-3, and Top-5 records are selected by highest observed score.There were 21–30 participants per game, with participation pay and score rewards; selected records require a completed first episode, and observed scores take precedence over replay values.Source: §B.3 · pp. 18–19, §B.3.1 · p. 19

B.3.2 CALCULATING SCORES AND LEARNING CURVES

  • Averaging across all eligible participants rather than selecting Top-k records gives Max = 43.6, Mean = 31.2, LG = +10.4, and LS = +2.72.Source: §B.3.2 · p. 19

B.3.3 COMPARING RECORDED BEHAVIOR · B.3.4 CASES OF EXECUTION AND CONTINUED INVESTIGATION · B.4 RQ4: HOW WELL DOES LEARNED KNOWLEDGE TRANSFER TO NEW GAME INSTANCES?

The sections compare human and agent episode behavior, illustrate how agents execute multi-step solutions and continue investigating, and describe a matched-game evaluation of transfer to changing instances. The supplied passages give behavioral statistics and evaluation procedures but no transfer-score values.

B.3.3 COMPARING RECORDED BEHAVIOR

  • The behavioral comparison uses completed human episodes and pooled trials from six Opus 5 methods, excluding prior-best replay values from behavioral statistics.Human Top-3 and agents are compared on fixed games in matching five- or ten-episode scoring windows.Source: §B.3.3 · p. 19
  • Action-sequence similarity averaged 0.5379 across 150 human episode pairs and 0.6575 across 918 agent pairs, measuring repeated actions rather than strategy quality.The comparison pools three trials for each of six Opus 5 methods and uses Human Top-3 on fixed games within the same five- or ten-episode scoring windows.Source: §B.3.3 · p. 19
  • After a score decrease, the next score exceeded the prior record in 13 of 39 human cases (33.3%) and 52 of 238 agent cases (21.8%).These are descriptive, unweighted observations; repeated observations from the same participant or run are not independent, so they are not causal tests of willingness to explore.Source: §B.3.3 · p. 19

B.3.4 CASES OF EXECUTION AND CONTINUED INVESTIGATION

  • In RedEye, an OpenCode–Opus 5 trial combined documentary and character clues into a rescue order, scoring 100 in episodes 9 and 10.The example illustrates executing a multi-step solution within one run.Source: §B.3.4 · p. 20
  • In Model Auditor, an OpenCode–Opus 5 trial used programmatic coefficient search in episode 4 and Gaussian elimination in episode 5, scoring 100 in both.The fitted models are recorded hypotheses, not necessarily the true hidden implementation.Source: §B.3.4 · p. 20
  • The two highest-Max human Model Auditor records reduced probe actions across episodes 1–5, from 40 and 17 initially to zero in later episodes.The records show reduced testing, but do not establish whether effort, confidence, frustration, or another factor caused it; counts exclude resets and predictions.Source: §B.3.4 · p. 20

B.4 RQ4: HOW WELL DOES LEARNED KNOWLEDGE TRANSFER TO NEW GAME INSTANCES?

  • Transfer evaluation compares same-instance controls with changing-instance trials on matched games and rule seeds, using ten-episode learning slopes; Table 9 defines LS drop as Control minus Re-shuf.The Qwen row covers nine games and other rows ten; three independent trials per game and configuration are required, and the same-instance control repeats instance 1.Source: §B.4 · p. 21, Table 9 · p. 21
  • The supplementary budget-sensitivity comparison adds nine extended-budget control trajectories while keeping the games constant; these results do not replace standard-budget estimates.Table 10 compares same-game learning slopes with and without the extended-budget trajectories.Source: §B.4 · p. 21, Table 10 · p. 21

B.4.1 APPLYING A RULE TO A NEW INSTANCE · B.5 RQ5: DO HIGHER INFERENCE COSTS LEAD TO BETTER PERFORMANCE? · B.5.1 USAGE AND PRICING

The cases show agents reusing learned rules or clues on new instances, while estimated inference costs vary across methods and do not uniformly track Max scores. Cost and usage estimates distinguish token types and reflect different memory, update, and calling patterns.

B.4.1 APPLYING A RULE TO A NEW INSTANCE

  • The rule-reuse cases illustrate possible transfer to new instances but do not estimate its frequency or explain aggregate LS differences.Source: §B.4.1 · p. 21
  • OpenCode–Opus 5 repeatedly scored 100 by applying an inferred five-feature condition to changing Poisoner menus.The case illustrates reuse of a condition rather than a fixed dish position and draws on interactions beyond the scoring window.Source: §B.4.1 · p. 21
  • In SYNERGYCORP seed 2, Opus 5 scored 137, Gemini 3.5 Flash 84, and Sonnet 5 67 as they reused clues or probed boss preferences.All three agents began from the same instance with a 30-action budget; using the learned mapping could save actions.Source: §B.4.1 · p. 21

B.5.1 USAGE AND PRICING

  • Cost estimates distinguish fresh input, cached input, and output tokens, and the Kimi K3 breakdown separates input from generation.Prices use the 30 August 2026 snapshot, and the estimates are not billing records.Source: §B.5.1 · p. 22, §B.5 · p. 22
  • Kimi K3 per-episode generation costs were $0.108 for OpenCode, $0.369 for EvoTest, and $0.184 for ReasoningBank.The pricing snapshot used $15 per million generation tokens and $3 per million fresh-input tokens.Source: Table 11 · p. 22, §B.5.1 · p. 22

B.5 RQ5: DO HIGHER INFERENCE COSTS LEAD TO BETTER PERFORMANCE?

  • Reflexion and AWM had lower total estimated costs than OpenCode, alongside lower Max scores.The paper compares estimated costs and Max scores per episode; the estimates are not billing records.Source: §B.5.1 · p. 22

B.5.1 USAGE AND PRICING

  • MEMORY had the largest input volume, while EvoTest updates used 11.5K generated tokens per episode versus OpenCode’s total generation of 7.2K.OpenCode used fewer model calls: one GEMFORGE trace batches nine game actions into one call and returns only the last 60 output lines.Source: §B.5.1 · p. 22

C AGGREGATE RESULTS AND TRIAL VARIABILITY … C.3 RE-SHUFFLED GAMES

The section reports weighted performance and learning estimates with standard deviations derived from independent trials. It describes how trial-level metrics and cross-game variability are calculated, and presents results for all 20 games, fixed games, and re-shuffled games.

C AGGREGATE RESULTS AND TRIAL VARIABILITY

  • Within each game, Max uses a trial’s highest score, Mean its average, LG compares its first and last kg episodes, and LS fits its score trend against original episode indices.Averaging these trial-level metrics reproduces the point estimates apart from rounding of archived episode means.Source: C AGGREGATE RESULTS AND TRIAL… · p. 24
  • The tables report ± sample standard deviation across complete trials, not variation across episodes or game difficulty.The SD is calculated from trial-level metric values.Source: C AGGREGATE RESULTS AND TRIAL… · p. 24
  • Across games, within-game variation is propagated using the paper’s weights under independent runs; the resulting SD is not a standard error or confidence interval for the multi-trial mean.The instance panel and reference scores are held fixed.Source: C AGGREGATE RESULTS AND TRIAL… · p. 24
  • Each of the 580 configuration–game cells contains three complete scored trajectories with no imputed episode scores, limiting the precision of SD estimates.The three-trajectory count applies to every cell in the tables.Source: C AGGREGATE RESULTS AND TRIAL… · p. 24
  • Human Top-k groups are performance-selected descriptive results, so they are not treated as independent repeat trials or assigned agent-trial SDs.Their results remain in Appendix B.3 and are not interpreted as population-level uncertainty.Source: C AGGREGATE RESULTS AND TRIAL… · p. 24

C.1 ALL 20 GAMES

  • Table 13 reports weighted performance and learning over 20 games as point estimates ± standard deviations; per-game results appear alongside game curves and rule evidence in Appendix D.The reporting covers all 29 agent configurations and four metrics.Source: C AGGREGATE RESULTS AND TRIAL… · p. 24, Table 13 · p. 25

C.2 FIXED GAMES

  • Table 14 reports weighted performance and learning over 10 fixed games as point estimates ± standard deviations estimated from within-game trial variability.Source: Table 14 · p. 26

C.3 RE-SHUFFLED GAMES

  • Table 15 reports weighted performance and learning over 10 re-shuffled games as point estimates ± standard deviations estimated from within-game trial variability.Source: Table 15 · p. 27

D PER-GAME RESULTS AND RULE-DISCOVERY EVIDENCE … D.20 MODEL AUDITOR

This appendix pairs task descriptions and scoring windows with per-game performance and learning tables across six games, alongside qualitative assessments of rule discovery from interaction records. These assessments are distinct from scoring-window performance, and complete rule recovery often distinguishes systems more sharply than successful execution.

D PER-GAME RESULTS AND RULE-DISCOVERY EVIDENCE

  • Rule annotations are supplementary interaction-record evidence, not score measurements; terse logs can leave targets unverified, and high scores do not establish every hidden rule.The first five games use complete-rule criteria; later games also include successful execution supported by observed performance. For Haunted Inn and Poisoner, seed-dependent rule and policy labels use the same seed as scored trials.Source: D PER-GAME RESULTS AND RULE-DISCOVERY… · p. 28

D.1 PRIMORDIALSOUP

  • Kimi K3 recovers every PrimordialSoup target in at least one trial, while weaker configurations often miss T6–T7 or the catalyst-state account.Qwen 3.8 Max and several Claude-based systems nearly recover every target; the main separation is recovery of the alternative L2a/L3 branch beyond the common L2b plateau.Source: §D.1 · p. 30, Table 18 · p. 30

D.2 GEMFORGE

  • Claude Opus 5 and Claude Code + Opus 5 each recover every GemForge target in at least one trial, while most other systems leave recipe alternatives unestablished.High scores can come from partial recipe graphs, so complete-rule recovery distinguishes systems more sharply than execution alone.Source: §D.2 · p. 31, Table 21 · p. 31

D.3 CATNIP

  • Claude Code + Opus 5 and Kimi K3 show the broadest Catnip complete-target coverage, while other systems often learn safe actions or reward thresholds without completing the broader model.The broader model includes preference, breed, item, and multiplier rules; strong runs combine local target preferences with long-horizon reward and early-exit tradeoffs.Source: §D.3 · p. 33, Table 24 · p. 33

D.4 DUNGEON

  • Dungeon systems often recover key landmarks and the scoring rule, but every configuration has only partial evidence for the full room graph, and complete hazard coverage is less common than route execution.A successful treasure-and-exit route does not establish discovery of the complete dungeon.Source: §D.4 · p. 35, Table 27 · p. 35

D.5 SYNERGYCORP

  • SynergyCorp systems learn action groups more readily than the temporal rules for a complete promotion policy; taboo, combo, repetition-decay, and drifter-switch conditions rarely become complete in one trial.Several configurations fully recover the action partition in at least one trial.Source: §D.5 · p. 36, Table 30 · p. 36

D.15 PATCH REALITY

D.16 POISONER

  • Poisoner tests inferring taste constraints from accepted and rejected dishes and applying them to new menus on reshuffled seed 2 over five episodes (C = 100).Seed-matched rule evidence covers targets T2, T4, and T5 in the three score-reporting trials, but does not identify when a target was discovered.Source: §D.16 · pp. 53–54, Table 61 · p. 53, Table 63 · p. 54

D.17 ROOMMATES

  • Roommates tests inferring pairwise and nonlinear grouping rules from room-level feedback on reshuffled seed 1 over five episodes (C = 7).One task decomposes harmony sums for nine tenants assigned to three rooms; another uses a hidden five-type ring with adjacent +1, distance-two −1, and same-type 0 scores.Source: §D.17 · pp. 54–55, Table 64 · p. 55, Table 66 · p. 56

D.18 REDEYE

D.19 BUTTERFLY

D.20 MODEL AUDITOR

  • Model Auditor tests probing and predicting a new stateful machine across five episodes on reshuffled seed 1, with method transfer rather than answer transfer.Its machines use ternary inputs and combine GATE thresholds, modulo-3 MIX, bounded-state HOLD, and one-step-delayed ECHO mechanisms.Source: §D.20 · pp. 59–60, Table 73 · p. 60, Table 75 · p. 61

Related papers

Loading 2610.08215v1…