PaperlayerView exact revision on arXiv

Source-linked AI summary

The PokeAgent Challenge: Competitive and Long-Context Learning at Scale

Seth Karten, Jake Grigsby, Tersoo Upaa, Junik Bae, Seonghun Hong, Hyunyoung Jeong, Jaeyoon Jung, Kun Kerdthaisong, Gyungbo Kim, Hyeokgi Kim, Yujin Kim, Eunju Kwon, Dongyu Liu, Patrick Mariglia, Sangyeon Park, Benedikt Schink, Xianwei Shi, Anthony Sistilli, Joseph Twin, Arian Urdu, Matin Urdu, Qiao Wang, Ling Wu, Wenli Zhang, Kunsheng Zhou, Stephanie Milani, Kiran Vodrahalli, Amy Zhang, Fei Fang, Yuke Zhu, Chi Jin

arXiv:2603.15563v2cs.LGcs.AI

TL;DR

Existing benchmarks rarely combine partial observability, adversarial reasoning, and long-horizon planning under realistic conditions. The paper introduces a two-track Pokémon benchmark with standardized resources and orchestration infrastructure; its competition reveals persistent gaps between specialist RL, LLM systems, and human performance, motivating a living benchmark.

  • Problem

    Few existing benchmarks jointly stress partial observability, game-theoretic reasoning, and long-horizon planning under realistic conditions.

  • Method

    The PokéAgent Challenge pairs standardized competitive battling and Pokémon Emerald speedrunning with large battle datasets, diverse baselines, harness evaluation, and open-source orchestration infrastructure.

  • Results

    Specialist RL and search methods dominated LLM approaches in battling, while no raw frontier model achieved non-trivial speedrunning progress without a sophisticated harness.

  • Takeaways & Limitations

    The challenge transitions into a living benchmark with a Battling leaderboard and self-contained Speedrunning evaluation for reproducible future comparisons.

  • Takeaways & Limitations

    The best speedrunning agent remains 2.2× slower than human speedrunners, and closing the gap requires advances in navigation, obstacle avoidance, and objective sequencing.

Abstract

from arXiv · show

We present the PokeAgent Challenge, a large-scale benchmark for decision-making research built on Pokemon's multi-agent battle system and expansive role-playing game (RPG) environment. Partial observability, game-theoretic reasoning, and long-horizon planning remain open problems for frontier AI, yet few benchmarks stress all three simultaneously under realistic conditions. PokeAgent targets these limitations at scale through two complementary tracks: our Battling Track, which calls for strategic reasoning and generalization under partial observability in competitive Pokemon battles, and our Speedrunning Track, which requires long-horizon planning and sequential decision-making in the Pokemon RPG. Our Battling Track supplies a dataset of 20M+ battle trajectories alongside a suite of heuristic, RL, and LLM-based baselines capable of high-level competitive play. Our Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons of harness-based LLM approaches. Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokemon, with over 100 teams competing across both tracks and winning solutions detailed in our paper. Participant submissions and our baselines reveal considerable gaps between generalist (LLM), specialist (RL), and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokemon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing suites and positioning Pokemon as an unsolved benchmark that can drive RL and LLM research forward. We transition to a living benchmark with a live leaderboard for Battling and self-contained evaluation for Speedrunning at https://pokeagentchallenge.com.

1 Introduction

The paper introduces PokéAgent Challenge, a standardized benchmark targeting partial observability, game-theoretic reasoning, and long-horizon planning through competitive battling and RPG speedrunning. Its large-scale resources and competition expose substantial gaps among LLM, RL, and human performance.

  • Motivation: Few existing benchmarks stress partial observability, game-theoretic reasoning, and long-horizon planning simultaneously under realistic conditions.Pokémon combines hidden information and strategic opponents in battles with exploration, resource management, and extended decision sequences in campaigns.
  • Motivation: Prior Pokémon evaluations used different games, harnesses, and criteria, preventing meaningful comparison of model capability.Earlier demonstrations included Claude, Gemini, and GPT systems, but their differing setups conflated harness and model effects.
  • Benchmark: The PokéAgent Challenge standardizes two complementary tracks: competitive battling for strategic reasoning under partial observability and RPG speedrunning for long-horizon planning.Battling uses two-player competitive Pokémon, while Speedrunning evaluates completing Pokémon Emerald as quickly as possible.
  • Competition Results: 100+ teams competed in NeurIPS 2025, while specialist RL and search methods dominated LLM approaches in battling and raw frontier models made no non-trivial speedrunning progress.The competition also produced methods including Scripted Policy Distillation and iterative offline RL with dynamic data weighting.
  • Resources: The benchmark releases 4M human demonstrations, 18M synthetic battles, 200K+ curated teams, heuristic and learning baselines, and open-source RPG orchestration infrastructure.These resources support standardized comparisons across RL, LLM, and hybrid approaches.

2 Related Work

Prior game benchmarks advance individual capabilities such as cooperation, resource management, or open-world planning, but do not combine adversarial partial observability with long-horizon planning at living-ecosystem scale. Prior Pokémon systems likewise lacked reusable, standardized infrastructure for fair cross-paradigm comparison.

  • Game AI Benchmarks: Traditional game benchmarks have driven progress across board games, poker, StarCraft II, Diplomacy, and other strategic settings.Performance margins shrink in stochastic, partially observable environments, where LLM agents often lag specialist RL and search systems.
  • NeurIPS Game AI Competitions: Neural MMO, Lux AI, and MineRL each target specific axes, but none combines adversarial partial observability with long-horizon planning in a living competitive ecosystem.PokéAgent addresses this gap through its dual-track design.
  • Prior Work on Pokémon AI: PokéChamp, Metamon, Puffer, Claude, Gemini, and GPT demonstrated individual Pokémon systems, but none established standardized evaluation, public leaderboards, or multi-track infrastructure.The challenge reframes these efforts as reusable benchmark resources for fair cross-paradigm comparison.

3 Competitive Battling Track

The Battling Track evaluates agents in Pokémon Showdown, an imperfect-information stochastic game with diverse teams, long matches, and evolving rules. It provides standardized data, baselines, evaluation infrastructure, and metrics for comparing RL, LLM, and hybrid systems.

  • Battling Environment Design: Pokémon battles are two-player, zero-sum, stochastic games with imperfect information, approximately 9 actions per turn, and matches lasting 20–100 turns.Hidden team composition, rare stochastic events, diverse movesets, and evolving rules make progress difficult to evaluate.
  • Evaluation Criteria: The track addresses noisy, non-stationary ladder evaluation by releasing standardized baselines and datasets alongside a dedicated AI leaderboard.Matches run on a separate modified Showdown server against community submissions and organizer-maintained baselines.
  • Evaluation Criteria: Full-History Bradley–Terry is the primary skill metric, fitted over all battle results with bootstrapped uncertainty and a minimum sample size.Glicko-1 and GXE are also reported, but the authors characterize online metrics as less suitable for the small, dense, fixed-policy agent pool.
  • Rulesets: The benchmark focuses results on Gen 1 OU and Gen 9 OU, which stress different combinations of hidden information, state-space size, and available demonstrations.An Extended Timer variant permits nearly unlimited deliberation for LLMs and test-time reasoning.
  • Baselines and Data: The released resources include reconstructed player-perspective trajectories, 200K+ inferred teams, LLM harnesses, and 30 RL checkpoints spanning the competitive skill ladder.LLM baselines use structured state representations and configurable minimax search, while RL baselines train on human and self-play battles.
  • Baseline Performance: The strongest baselines are competitive with top human players, although the current upper bound remains below superhuman performance.Figure 3 compares selected RL and LLM agents both against one another and against human players.

4 Long-Context Speedrunning Track

The Speedrunning Track formalizes Pokémon Emerald as a long-horizon planning task evaluated through standardized milestones and completion time. Its orchestration baseline combines specialized agents and tools, yet frontier-model performance remains slower, costlier, or less reliable than human play.

  • Speedrunning Environment Design: Speedrunning supplies a clear optimization objective, milestone-based credit assignment, and a task requiring perception, planning, memory, navigation, and combat.The environment is formalized as an episodic MDP with button inputs as actions.
  • Speedrunning Environment Design: Agents receive visual frames and limited party information, while puzzle states, obstacles, items, and movesets remain hidden.This preserves perception and state-inference challenges during gameplay.
  • Speedrunning Route: The early-game route spans milestones from Littleroot Town to defeating Roxanne and requires exploration, backtracking, branching navigation, and nonlinear objective dependencies.Human world-record splits provide an upper bound for route progression.
  • Evaluation: Evaluation uses milestone completion percentage, completion time for fully completed runs, and action count as the tiebreaker; the initial scope ends at the first gym.Even this segment requires thousands of steps and millions of reasoning tokens across hours of play.
  • Orchestration Baseline: The open-source baseline coordinates pathfinding, button input, retrieval, battle strategy, reflection, puzzle solving, and objective verification through a central orchestrator.Automatic context compaction manages the thousands of reasoning steps required for long-horizon play.
  • Baseline Results: Gemini 3 Flash completes the route fastest at approximately 2:24 mean, while Claude Sonnet 4.5 completes all milestones with high variance; the best organizer baseline is approximately 1.8× slower than average human play.The figure compares cumulative time, actions, tokens, and cost across five frontier models.

5 NeurIPS 2025 Competition

The NeurIPS 2025 competition tested Battling and Speedrunning agents at scale, exposing performance gaps among specialist methods, generalist LLMs, and human-level systems. Results also showed that evaluation outcomes depend on both architectural support and the metric used.

  • Competition Overview: 100+ teams competed across both tracks, producing new methods and revealing a capability hierarchy between specialist and generalist approaches.The competition also attracted 650+ researchers and generated 150+ submissions.
  • Battling Track: 13 of 16 qualifying Battling tournament spots went to teams extending public RL baselines, while independent RL and search methods won the remaining three.The two #1 seeds, PA-Agent in Gen 1 OU and FoulPlay in Gen 9 OU, won their tournament brackets.
  • Speedrunning Track: 6 of 22 valid Speedrunning submissions completed all 15 milestones, with Heatz winning in 40:13 using Scripted Policy Distillation.SPD combines LLM-generated subgoals and scripted policies with imitation learning and reinforcement learning.
  • Speedrunning Track: 649 steps made Deepest the most action-efficient finisher despite ranking 5th by wall-clock time, exposing a tradeoff between inference speed and sample efficiency.Heatz completed in 1,608 steps, but its RL-distilled policy executed substantially faster per step.
  • Long-Context Agents: Raw frontier VLMs achieved effectively 0% task completion without a harness, while domain-specific perception, memory, planning, and action modules enabled progress.Common CLI-agent architectures also struggled to maintain coherent state across thousands of sequential decisions.
  • Cross-Track Insights: Pokémon battling measured capabilities largely missed by standard LLM benchmarks: standard benchmark structure explained 91% of variance but only 27% of GXE variance.No existing benchmark correlated strongly with GXE; the maximum Spearman correlation was 0.77.

6 Conclusion: From Competition to Living Benchmark

The challenge transitions from a time-bounded competition into a living benchmark with persistent Battling evaluation and reproducible local Speedrunning evaluation. The paper identifies remaining capability gaps and open research problems for maintaining difficulty and broadening access.

  • Living Benchmark: The Battling Track now provides a live Showdown leaderboard with organizer baselines and the full history of submissions for ongoing evaluation.The Speedrunning Track instead uses standardized local emulation and milestone evaluation.
  • Open Challenges: VLM-SLAM remains a bottleneck because Speedrunning agents struggle with localization, action-distance estimation, and objective detection.The paper frames consistent spatial grounding through language-vision interfaces as an open challenge for RPG play.
  • Open Challenges: Specialist RL agents still outpace harness LLM agents in battling, leaving performance parity or effective hybridization as an open problem.The paper also identifies full-game completion with open-source models and human-level speedrunning times as unresolved goals.
  • Competition Transition: The competition timeline ran from July through November 2025, with a September hackathon, October qualifying rounds, and November best-of-99 finals.The schedule supported iterative improvement while preserving evaluation integrity through held-out final stages.
  • Infrastructure: The challenge supplied a dedicated Showdown server, a retrieval-augmented Discord chatbot, cloud credits, and a streamed hackathon to support participants.These resources combined evaluation infrastructure with participant-facing technical assistance.

A.2 Organizational Outcomes

The competition achieved substantial participation and community engagement across both tracks, attracting researchers from RL, game AI, long-context reasoning, and language-agent communities. Supporting infrastructure and scheduling choices were associated with sustained participant involvement.

  • Participation: 100+ active teams registered submissions across both tracks, while 650+ Discord members engaged in technical discussions.The competition also generated more than 100K battles on its Showdown server.
  • Speedrunning Participation: 22 valid Speedrunning submissions produced 6 complete runs, with all 15 milestones achieved by those finishers.This completion rate provided a common basis for comparing long-horizon RPG agents.
  • Cross-Community Engagement: The dual-track structure attracted both RL and LLM research communities, with several teams developing architectures for combat and exploration.The Battling Track drew multi-agent learning researchers, while Speedrunning attracted long-context reasoning and language-agent communities.
  • Community Infrastructure: The @pokeagent retrieval-augmented Discord bot answered setup, baseline, and submission questions using organizer-curated documentation.The paper reports that this reduced organizer burden while providing continuous assistance.
  • Timing: A September–November competition window aligned peak participation with the Fall university semester and encouraged student involvement.Participants could integrate the challenge into course projects and independent studies.
  • Competition Structure: The Battling Track used Practice, Qualifying, and Tournament stages across Gen 1 OU and Gen 9 OU rulesets.The practice and qualifying stages used ranked battles against participants and organizer-hosted agents.

B.2 Qualifying Stage

The Qualifying Stage combined extended ladder play with multiple skill-rating metrics before selecting teams for best-of-99 tournaments. Results favored RL and search-based systems and motivated closer scrutiny of rating uncertainty and time-pressure effects.

  • Qualifying Results: 35K battles were played during the two-week qualifying period, with strong alignment across the reported skill-rating metrics.The leaderboard was also compared using alternative metrics including Elo, GXE, and win rate.
  • Baseline Comparisons: Participants competitive with private organizer baselines separated from the broader field, while many others clustered slightly above the best public baseline.Organizer baselines provided reference points grounded in known training details and human-relative performance.
  • Method Comparison: All top-performing submissions used RL or search rather than pure LLM approaches, while time pressure further deflated LLM ladder performance.The organizers added a long-timer leaderboard to separate reasoning ability from inference speed.
  • Tournament Stage: The top eight teams in each format advanced to head-to-head brackets, where PA-Agent won Gen 1 OU and FoulPlay won Gen 9 OU.The finals were best-of-99 matches, with PA-Agent defeating 4thLesson 50–28 and FoulPlay defeating Q 50–14.
  • Rating Analysis: Top-3 agents converged across four rating systems, but ranks 4+ disagreed systematically, with Elo diverging from Bradley–Terry.Glicko-1 was preferred because it supports online updates, uncertainty quantification, and convergence matching batch maximum likelihood estimation.
  • Judge’s Choice: Porygon2AI received Judge’s Choice recognition for league training inspired by AlphaStar’s diverse opponent modeling, while August was recognized for a pure LLM method.August used chain-of-thought reasoning for move selection without learned components.

C Speedrunning Track: Full Competition Results

The Speedrunning Track evaluated Pokémon Emerald completion using time, actions, and completion status, while also highlighting the joint role of model and harness capabilities. Heatz won the completed runs, whereas Deepest achieved the fewest actions and received recognition for efficiency.

  • Competition Results: 6 of 22 submitted teams achieved 100% completion, with Heatz winning in 40:13 using Scripted Policy Distillation.The standings reported both wall-clock time and step-count views.
  • Evaluation: Speedrunning evaluation combines completion percentage, completion time, and action count, with action count breaking ties among fully completed runs.The supplied passages identify these evaluation dimensions but do not provide the full qualifying-metrics interpretation.
  • Harness Components: Successful harnesses decompose speedrunning into perception, memory, planning, and action components.These components cover state understanding, long-horizon coherence, route optimization, and low-level control.
  • Attribution: Benchmark results reflect joint model-and-harness capability, so model differences alone cannot be interpreted as raw model capability.The harness includes the surrounding system used to represent state, maintain context, plan, and act.
  • Efficiency: Deepest received the Judge’s Choice for completing the run with the fewest total actions despite slower wall-clock performance.This recognition emphasizes action efficiency as distinct from elapsed-time ranking.

D Baseline Architecture Details

The baseline ecosystem combines LLM, RL, search, and multi-agent systems for competitive battling and RPG play. These architectures emphasize scalable training or search, orchestration, and adaptations for partial observability, while results show strong competition and differing efficiency trade-offs.

  • Baseline Foundations: The challenge builds on upgraded PokéChamp and Metamon baselines spanning LLM minimax search, offline RL, and multi-agent RPG orchestration.PokéChamp uses LLM position evaluation with approximate state transitions, while Metamon trains Transformer policies on replay-derived data.
  • RL Baselines: PA-Agent achieved 80.35% GXE in Gen 1 OU qualifying through iterative offline RL and tournament-based team selection.The approach targets stochastic, partially observable battles and diverse opponent strategies.
  • Search Baselines: Foul Play uses root-parallelized MCTS with DUCT and damage-roll grouping to search simultaneous moves efficiently.Its engine supports tree depths of 10 or more turns on promising lines while pruning less useful branches.
  • Search Results: Foul Play exceeded 90% GXE in Generation 9 Random Battles, exceeded 80% GXE in Generation 9 OU, and won Gen 9 OU with a 50–14 finals victory.The passage reports peak ratings of 2341 in Random Battles and 1879 in OU.
  • Training Modifications: The Kron optimizer replaces AdamW to improve gradient stability during model scaling, at higher computational cost.The architecture changes also include replacing Leaky ReLU with AID, which is described as mitigating plasticity loss.
  • Curriculum Learning: Team Q separates mechanics learning from strategic fine-tuning, and this separation significantly accelerates convergence.The largest performance jump occurred when moving from random or heuristic initialization to curriculum-based initialization.
  • Speedrunning Baselines: SPD achieved a 40:13 Pokémon Emerald run up to the first gym after scripted policy generation, distillation, and reinforcement-learning fine-tuning.The resulting policy sometimes synthesized BFS routines for short navigation paths and later executed them more efficiently.
  • Speedrunning Architecture: A milestone vector acts as both memory and goal-conditioning, improving stability under Pokémon Emerald’s randomness and nondeterministic inputs and outputs.The vector records sub-milestones, locations, and PokéCenters reached during the run.

E.2.4 Deepest (Speedrunning Track Judge’s Choice: Most Efficient)

Deepest’s training-free speedrunning system uses partial visual observations, guidebook-based goals, auxiliary tools, and a frontier vision-language model. It achieved a fifth-place result while receiving the Judge’s Award for sample efficiency.

  • Result: Deepest achieved 5th place with a 02:04:29 completion time for the first-gym milestone and received the Judge’s Award for sample efficiency.The approach used Gemini 2.5 Flash without data collection or human feedback.
  • Perception Constraint: The agent operates without privileged game state, computing navigation from partially observed tiles explored during gameplay.This design aims to make planning and control observation-driven rather than dependent on internal game mechanics.
  • Goal Specification: A Guidebook supplies milestone-specific knowledge, while image goals help the agent recognize visually indistinct destinations.The guidebook includes route, rival, and starter-selection information for high-level strategy.
  • Auxiliary Tools: Auxiliary pathfinding, memory, and thinking tools let the agent plan over long horizons and adapt its computation beyond low-level action tokens.The tools support navigation, explicit state persistence, and variable reasoning budgets.

F.1 RPG System Comparison

This section contrasts heterogeneous Pokémon RPG systems with PokéAgent’s standardized evaluation and documents the benchmark’s configuration-space foundations. It also describes the infrastructure, runtime, cost, and combinatorial assumptions used for comparison.

  • System comparison: The (S, T, M, F, Φ) framework characterizes RPG systems by state representation, tools, memory, feedback, and fine-tuning.Table 2 warns that heterogeneity across these dimensions makes direct performance comparison methodologically unsound without standardization.
  • Feedback and reflection: PokéAgent’s multi-agent self-reflection evaluates action outcomes, detects suboptimal play, and triggers strategy revision for error recovery.This contrasts with observe-act systems that lack explicit decision evaluation or recovery mechanisms.
  • Execution model: The Speedrunning Track measures wall-clock time as a primary metric, unlike systems that freeze the emulator and allow unlimited deliberation per action.Prior playthroughs commonly required hundreds of hours, whereas standardized timing makes comparisons operationally meaningful.
  • Performance and cost: LLM baseline costs vary by over 70×, from $1.247 per game for GPT-5.2 to roughly $0.015 for DeepSeek V3 and Qwen3.5 Plus.Output token counts also differ substantially, with MiniMax producing ∼6.2K completion tokens per turn and DeepSeek V3 and Qwen3.5 Plus fewer than 10.
  • Configuration space: The EV count uses nonnegative 4-EV units with per-stat and total caps of 63 and 127 units, respectively.Inclusion-exclusion subtracts 119,877,472 configurations per capped stat; two simultaneously capped stats are impossible because 128 exceeds the total budget of 127.
  • Configuration space: The Gen 9 configuration derivation combines exact factors with upper bounds, including Mew’s 375-move movepool, three abilities, 21 functional natures, 248 held items, and 19 Terastallization types.The moveset factor is an upper bound, while the final log10 rounding is the only approximation.

G.4 Battle State Spaces

The battle-state analysis constructs large combinatorial spaces by multiplying team, active-state, field-condition, and status factors across formats. It explicitly marks approximations and shows that Gen 9 doubles adds substantial state complexity.

  • Gen 9 OU: Gen 9 OU state counting uses a representative maximum HP of 300, yielding 301 possible HP states per Pokémon.This is the derivation’s only approximation; actual maximum HP ranges from 1 to ∼714, but 300 is considered representative of competitive play.
  • Gen 9 OU: Gen 9 field conditions include 36 weather states and 33 terrain states, with turn counters for temporary effects.These factors are multiplied with hazards, screens, status, and other battlefield variables.
  • Gen 9 OU: Gen 9 singles models active Pokémon, stat stages, volatile conditions, item and ability changes, statuses, slot conditions, and Terastallization.The derivation omits move PP because depletion is uncommon in typical competitive games, although binary PP tracking would increase totals by ∼14 orders of magnitude.
  • Gen 9 VGC: Gen 9 VGC uses two active Pokémon per side, position-sensitive targeting, and a hidden choice of four Pokémon from each six-member team.Ordered active pairs from four brought Pokémon yield 144 arrangements across both sides.
  • Gen 9 VGC: VGC adds doubles-specific conditions including Pledge effects, Quick Guard, Wide Guard, Crafty Shield, Mat Block, redirection, and Helping Hand.The resulting per-side factor is 12,597,120,000, giving approximately 10^20.2 states across both sides.
  • Effective space: Competitive usage substantially reduces the effective team space because many legal options are dominated and rarely selected.The effective-space table counts only species, moves, items, abilities, and EV/nature spreads appearing above specified usage thresholds.

H.1 Additional Findings

Additional findings identify vision and model-specific reasoning pathologies as major sources of failure, while harnesses can narrow capability gaps and synchronized reasoning traces can diagnose otherwise hidden behavior.

  • Limitations and failure modes: Vision-related errors remained a dominant failure mode across Gemini generations despite improvements in reasoning capability.The paper therefore suggests that game-playing benchmarks will continue challenging AI until vision-language integration improves substantially.
  • Model comparisons: Frontier models reach 71% GXE versus 29% for Gemma3-1B without a harness, while the harness raises Gemma3-1B to 53% GXE.The comparison shows that architectural support partially compensates for raw model capability.
  • Failure modes: Claude models show memory corruption cascades, Gemini models roadblock behavior, GPT models excessive plan commitment, and Qwen models computational paralysis.These distinct patterns indicate that model families fail through different sequential-decision pathologies.
  • Failure modes: Qwen’s recursive damage calculations consume the decision window while the battle state evolves, making extended deliberation itself harmful in high-stakes sequential play.The live ladder visualization makes this pathology identifiable rather than reducing it to generic weak play.
  • Cross-domain relevance: Modular harness techniques developed for Pokémon agents have influenced autonomous coding agents through persistent memory, hierarchical planning, and structured perception.This is presented as an unexpected transfer of harness ideas to another domain.
Loading 2603.15563v2…