PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition

Kaisen Yang, Qingle Liu, Kejin Wang, Yicheng Zhao, Jieming Li, Shenghan Zheng, Ruize Yang, Bojun Yang, Heng Gong, Xiang Gao, Lanyue Zhang, Kaiyu Zhong, Zhuo Liu, Shaoxuan Li, Chengxi Li, Yong Yan, Weixuan Zhang, Tianwei Luo, Situ Wang, Youjie Zheng, Sihan Zhao, Shengyuan Wang, Huan-ang Gao, Jiazheng Xu, Xiaohui Xie, Wentao Han, Hongning Wang

arXiv:2610.12341v1cs.AIcs.CL

TL;DR

Adapting strategies in adversarial games from limited interaction remains challenging, especially when opponents’ strategies shift the learning target. The paper formalizes AHL and introduces AAArena to evaluate agents that revise executable game policies. Opus5.5 with Claude Code earns six gold medals, while no evaluated configuration tops the other six human ladders.

  • Problem

    Adapting strategies from limited interaction remains challenging in adversarial games, where opponents’ changing strategies shift the learning target.

    Source: §1 · p. 1

  • Method

    The paper formalizes AHL, in which agents revise executable policies and supporting software from experience with fixed model weights, and evaluates them in AAArena’s 12 games and 1,920 archived human programs.

    Source: §1 · p. 2

  • Results

    Opus5.5 with Claude Code earns 6 gold medals, while no evaluated configuration tops the remaining 6 human ladders.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results point to potential for heuristic learning in adversarial games while identifying persistent challenges in game understanding, strategy implementation, and long-horizon planning.

    Source: §7 · p. 14

Abstract

from arXiv · show

Adversarial games have driven advances from heuristic search to reinforcement learning, yet learning and adapting strategies from limited samples remain challenging. AI agents offer an alternative by turning game experience into revisions of executable policies. Building on heuristic learning (HL), we formalize Adversarial Heuristic Learning (AHL), a paradigm that uses AI agents as learning engines to refine game policies and supporting software while keeping model weights fixed. We introduce AAArena, a benchmark comprising 12 authentic adversarial games and 1,920 archived human programs, with an evaluation protocol modeled on real-world game competitions. Agents interpret rules, choose opponents, analyze replays, and revise game agents to achieve their highest ranking within fixed match and evaluation budgets. We evaluate \val{completedmodels} model and harness configurations: Opus5.5 with Claude Code earns 6 gold medals, while no evaluated configuration tops the remaining 6 human ladders. Performance is generally weaker in games with more complex rule specifications. Further experiments show that opponent selection and dense feedback support policy improvement, and that agents learn from both on-policy replays of their own matches and off-policy replays of other players' matches. These results highlight HL's potential in adversarial games and identify persistent challenges in game understanding, strategy implementation, and long-horizon policy development.

1 Introduction

The paper formalizes AHL and introduces AAArena to test whether AI agents can develop competitive game programs from experience. Opus5.5 with Claude Code tops six human ladders, while no evaluated configuration tops the other six.

1 Introduction

  • The motivating challenge is adapting strategies from limited interaction when opponents’ changing strategies shift the learning target.The process also requires translating game understanding into executable policies and planning tests and revisions within a limited budget.Source: §1 · pp. 1–2
  • AI agents’ ability to revise software suggests they could serve as learning engines, reading failures, logs, and replays to improve programs.This builds on heuristic learning, in which agents accumulate experience in editable software structures.Source: §1 · pp. 1–2
  • The Tsinghua University Agent Competition provides the real-world process AAArena emulates: contestants interpret rules, design strategies, and improve programs through freely chosen matches.Contestants compete for leaderboard positions in a new game introduced each year.Source: §1 · p. 2
  • Adversarial Heuristic Learning (AHL) uses AI agents to revise executable game policies and supporting software from game experience while keeping model weights fixed.The game agent executes the policy in competition.Source: §1 · p. 2
  • Opus5.5 with Claude Code earns 6 gold medals, while no evaluated configuration tops the remaining 6 human ladders.The remaining games generally have more complex rules; further experiments highlight opponent selection, dense feedback, and off-policy replays as relevant to policy improvement.Source: §1 · pp. 2–3
  • AAArena combines 12 authentic adversarial games and 1,920 archived human programs in an evaluation modeled on real-world competitions.Its protocol separately budgets selected matches and full-pool evaluations to assess program improvement and movement up human-program ladders.Source: §1 · p. 2

2 Adversarial Heuristic Learning

AHL treats policy development as an iterative process in which an AI agent uses game experience and feedback to revise an executable heuristic system. The base model stays fixed while the game agent competes using the revised policy.

2.1 Preliminaries

  • A game agent executes a code policy from permitted observation history, while an AI agent develops that policy between matches using resources and feedback.Competitive performance of the resulting game agent measures the AI agent’s policy-development ability.Source: §2.1 · p. 3
  • Heuristic learning incorporates experience through direct revisions to a software system, which may include policy logic, state representations, tests, configuration, and memory.Feedback can include rewards, tests, logs, or replays.Source: §2.1 · p. 3

2.2 Heuristic learning in adversarial games

  • AHL applies heuristic learning to adversarial tasks: an AI agent revises a heuristic system to improve competitive performance while the base model’s parameters remain fixed.The heuristic system comprises an executable policy and development state containing tools, tests, experiment records, and memory.Source: §2.2 · pp. 3–4

2 Adversarial Heuristic Learning

  • The AHL loop acquires game experience, interprets feedback, and revises the executable policy and its supporting development state until a stopping criterion is met.The agent chooses how to collect experience and organize revisions; feedback sources, evaluation schedules, and stopping criteria are left to the task setting.Source: Figure 2 · p. 3, §2.2 · p. 4

3 AAArena

AAArena evaluates agents developing game programs across 12 adversarial games and frozen human-program pools. Agents gather targeted replay feedback and full-pool scores while choosing how to spend separate interaction and evaluation budgets.

3.1 Games and opponent pools

  • The 12 games span tactical combat, territory control, resource management, and survival, with verified frozen opponent pools and shared rule, SDK, replay, leaderboard, and reference-program resources.Rule descriptions are characterized by AST-node counts and rule atoms, representation-dependent measures that do not estimate exact Kolmogorov complexity or optimal-play difficulty.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Interaction interfaces

  • Small matches let agents select opponents and inspect detailed replays, while full-pool evaluations return Elo, rank, and per-opponent outcomes against the complete pool.The first channel supports targeted diagnosis; the second supplies a common reference for comparing immutable programs.Source: §3.2 · p. 5

3.3 Evaluation protocol

  • Each run has separate budgets for small-match units and full-pool evaluations, and the agent chooses when to revise, validate locally, or request either feedback channel.Default limits are 128 small-match units and 16 full-pool evaluations; the final result is the highest officially evaluated Elo and its corresponding rank, with the attaining policy retained.Source: §3.3 · p. 5

4 Experiments

Across the 12 games, AI-developed agents top six frozen human ladders, while results vary by game and revisions sometimes regress or plateau. Experiments find that more interaction alone has limited effect, while opponent choice, replay detail, batch size, and replay source shape observed policy progress.

4.1 Performance of AI Agents Developing Game Agents

  • 6 of 12 human ladders are topped; Opus5.5 with Claude Code earns 6 gold medals, and no evaluated configuration reaches rank 1 on the other six.The strongest configuration has the highest or tied-highest Elo on nine games; five of six games with smaller rule descriptions have a rank-1 finish, versus one of six with larger descriptions.Source: Table 2 · p. 5, §4.1 · p. 6

4.2.1 Evolution Patterns Across Games

  • Across games, trajectories show rank-1 successes, overall progress that can include regressions, and persistent plateaus after initial adaptation.Rollman, MoneCraft, Pacman, and AquaWar reach rank 1 and stop early; Generals, Miracle, and LOTA show extended intervals with little improvement.Source: §4.2.1 · pp. 6–7

4.2.2 Case Study: Dorado

  • The Dorado case study tracks full-pool Elo and rank alongside interaction order and token use, with champion milestones described through code-diff excerpts.The presented material frames the analysis as tracing code changes behind Dorado's progress.Source: Figure 4 · p. 7, §4.2.1 · p. 7, Table 3 · p. 8

4.3 Does More Interaction Overcome Performance Plateaus?

  • Tripling the budget leaves two retained champions unchanged and improves two, but none reaches rank 1.Miracle moves from rank 26 to 19 and AntWar2 from rank 9 to 6; Generals and LOTA keep their earlier best scores.Source: §4.3 · p. 8, Table 4 · p. 8

4.4.1 Does Opponent Selection Affect Performance?

  • Opponent-selection results vary by game: the ladder leads on Pacman and AntWar, while model selection leads on Miracle; top-four selection has the lowest Elo throughout.The comparison uses GLM-5.3, dense feedback, and batches of four; model selection outperforms random and top-four selection in all three games.Source: §4.4.1 · p. 10, Figure 6 · p. 10

4.4.2 Does Dense Replay Feedback Affect Performance?

  • Dense replay feedback yields higher retained-champion Elo than binary win/non-win feedback in all three representative games.Detailed replays include state, action, and consequence information for diagnosing decision errors, whereas binary feedback gives the outcome alone.Source: Figure 7 · p. 10, §4.4.2 · p. 10

4.4.3 Does the Number of Opponents per Small Match Matter?

  • Intermediate batches perform best: four opponents on Pacman and Miracle, and two on AntWar.The comparison fixes 128 opponent tests and 16 full-pool evaluations; the observed preference for batches of two to four is limited to these games and settings.Source: Figure 8 · p. 11, §4.4.3 · p. 11

4.5 Can Agents Learn from Off-Policy Replays?

  • All three off-policy runs improve from their first evaluated policy to the retained champion, with Pacman moving from rank 10 to 1, AntWar from 41 to 19, and Miracle from 27 to 15.On Miracle, the off-policy run reaches rank 15 versus rank 26 on-policy; on AntWar, on-policy reaches rank 5 versus 19 off-policy, while both reach rank 1 on Pacman.Source: §4.5 · p. 11, Figure 10 · p. 13

5 Related Work

Related work spans direct game-playing evaluation, competitive program-development benchmarks, and methods that revise programs or learn from experience. AAArena evaluates the agent developing a game-playing program and combines archived human opponents with agent-selected testing.

Competitive program development

  • Competitive program-development benchmarks differ in outputs, opponents, feedback, and revision protocols; AAArena combines 12 custom competition games, archived human-written programs, agent-selected testing, and separate budgets.Other examples span code generation, code repair, tournament logs and peer code, ranked human programs, and single-agent environments with hidden validation.Source: §5 · pp. 13–14

HL and program evolution

  • AHL applies heuristic learning to competitive game-agent development by having an AI agent revise policies and supporting software with a fixed base model.Related work also studies programmatic reinforcement learning, program search, software revision, and trajectory distillation.Source: §5 · pp. 13–14

Direct game-playing evaluation

  • Direct game-playing benchmarks evaluate models' in-match decisions, whereas AAArena evaluates the AI agent developing the autonomous program that plays.The cited examples include interactive-game benchmarks and competitive-player settings.Source: §5 · p. 14

6 Future Work

Future work proposes broadening AAArena, comparing code revision with parameter updates, and studying how agents use replay evidence to plan tests.

Extending AAArena

  • Planned AAArena extensions add games, controlled rule variants, and changing opponent populations to study adaptation to task and competition changes.The proposed evaluation targets include adaptation speed, cross-game transfer, and continual learning.Source: §6 · p. 14

Heuristic and reinforcement learning

  • Future comparisons of code revisions and parameter updates are to account for differences in feedback access and interaction budgets.The paper also proposes hybrid learners that revise policy structure with heuristic learning and train its components with reinforcement learning.Source: §6 · p. 14

From replay use to active learning

  • Proposed active-learning work would study how agents allocate replay views and matches to resolve uncertainty, including using external replays to form hypotheses before testing opponents.Validated strategies could then be turned into reusable policy modules.Source: §6 · p. 14

7 Conclusion

AAArena evaluates whether agents can develop executable game policies across 12 real-world competition games and archived human programs. Opus5.5 with Claude Code earns six gold medals, while no evaluated configuration tops the other six ladders; the games span distinct strategic settings.

7 Conclusion

  • Opus5.5 with Claude Code earns 6 gold medals, while no evaluated configuration tops the remaining 6 human ladders.The conclusion also reports policy improvement from dense feedback and from both on-policy and off-policy replays, alongside persistent challenges in game understanding, strategy implementation, and long-horizon planning.Source: §7 · p. 14
  • AAArena combines 12 real-world competition games, archived human programs, and a unified two-budget evaluation protocol.Its games cover distinct decision problems, including tactical combat, resource management, territory control, and survival.Source: §7 · p. 14, A The evaluated games · p. 17

A The evaluated games

  • The game descriptions include resource-and-army development in Dorado, drafting and tactical coordination in Miracle, and lane-based combat in LOTA.Dorado couples combat and harvesting with resource needs; Miracle adds heterogeneous units and artifacts; LOTA combines immediate fights with pressure on defensive structures.Source: A The evaluated games · p. 18
  • Other games test sequence-dependent tower-defense decisions, asymmetric pursuit, territory growth, mining, survival, hidden-identity inference, and combined economic and tactical control.Examples include AntWar2, Rollman, SnakeGo, MoneCraft, LostSpace, AquaWar, AntWar, and Generals.Source: A The evaluated games · pp. 17–18

B Arena resources and evaluation details

The appendix specifies the frozen arena resources, agent workflow, scoring procedures, rule-complexity measures, and conventions for reading policy-development plots. It also presents complete trajectory plots for runs classified as success, progress, or stagnation.

B.1 AI agent workflow

  • Agents use a persistent workspace to read rules, edit code, test selected opponents, inspect replays, and submit full-pool evaluations within match and evaluation budgets.Small-match requests cost one unit per opponent tested, while each full-pool evaluation costs one large unit; source edits, model calls, tokens, and wall-clock time have no separate limits.Source: §B.1 · p. 18, B Arena resources and evaluation details · p. 18

B.2 Elo and rank

  • Full-pool Elo estimates use match outcomes against a frozen opponent pool with fixed ratings, and rank counts opponents rated above the candidate.A virtual draw against the pool’s mean rating keeps the estimate finite for perfect win or loss records.Source: §B.2 · pp. 18–19

B.3 Gold medals

  • A gold medal marks a rank-1 finish under the rating protocol, not a direct head-to-head win against the human leader or optimal play.The reported results also do not measure generalization to unseen opponents because full-pool evaluations provide optimization feedback.Source: §B.3 · p. 19

B.4 Rule complexity: AST and RA

  • Rule complexity is measured with AST node counts and rule atoms, which count independently changeable propositions across seven rule classes.Both measures depend on the chosen description language and capture rule-description size, not state-space size or difficulty of optimal play.Source: §B.4 · p. 19

B.5 Reading the trajectory plots

  • Trajectory plots show snapshot Elo with rank annotations, interaction order across budget slots, and token use at each full-pool evaluation.The retained champion is reported separately from potentially regressing Elo curves; paired replay-source plots distinguish own-policy and other-policy replays.Source: §B.5 · pp. 19–20

B.6 Complete Policy Evolution Trajectories

  • Complete trajectories for twelve GLM-5.3 runs are presented in success, progress, and stagnation groups.The accompanying figures show full-pool Elo and ranks, interaction order, and token use.Source: §B.6 · p. 20, Figure 11 · p. 21, Figure 12 · p. 22

C Detailed ablation results

The appendix provides three GLM-5.3 ablation tables reporting Elo and rank under the 128/16 budget.

Opponent selection ablation

  • The opponent-selection ablation table reports Elo and rank for GLM-5.3 under the 128/16 budget.The supplied table caption does not state the cell values or the direction of any comparison.Source: Table 6 · p. 24

Replay feedback ablation

  • The replay-feedback ablation table reports Elo and rank for GLM-5.3 under the 128/16 budget.The supplied table caption does not state the cell values or the direction of any comparison.Source: Table 7 · p. 24

Small-match batch size ablation

  • The small-match batch-size ablation table reports Elo and rank for GLM-5.3 under the 128/16 budget.The supplied table caption does not state the cell values or the direction of any comparison.Source: Table 8 · p. 24

D Token consumption

Token expenditure varies substantially despite equal interaction budgets, both across configurations and within a configuration. The paper reports token use alongside competitive performance.

D Token consumption

  • 260M tokens for GPT6-sol versus 1,454M for Qwen 3.8 yields more than a fivefold difference in total use.Qwen 3.8 uses 1.6 times GLM-5.3’s tokens but earns three gold medals versus four; GPT6-sol also earns four with less than a third of GLM-5.3’s tokens.Source: D Token consumption · p. 24
  • 737M tokens are consumed by the leading configuration, Opus5.5 with Claude Code.All runs share the 128/16 interaction budget.Source: D Token consumption · p. 24, Table 9 · p. 24
  • Kimi K3 uses 1.5M tokens on AquaWar and 89M on Generals, showing substantial within-configuration variation.The feedback ceilings do not impose equal computational expenditure, so token use is reported alongside competitive performance.Source: D Token consumption · p. 24

E Algorithm families in the champion policies

Retained champions rely mainly on rules and heuristic scoring, but algorithm-family shares vary across games and models. These code-composition patterns are descriptive and do not establish which changes caused policy improvements.

E Algorithm families in the champion policies

Related papers

Loading 2610.12341v1…