Porimon: An LLM-Based Pok\'emon Battle Agent Enhanced by Long/Short-Term Knowledge Augmented Generation

I. INTRODUCTION

• Pokémon Battles test opponent-aware planning; prior LLM agents retrieve game facts and use within-battle turns, but do not reuse experience summaries from completed battles.Offline RL alternatives require substantial training data and are vulnerable to unseen cases; LLM agents can address unfamiliar tasks without fine-tuning but struggle with hallucination and accurate damage calculation.Source: §I · p. 1
• Porimon builds on PokéLLMon with detailed game retrieval, API-based damage calculation, and LSTKAG for reasoning over battle turns and accumulating experience across battles.The design is intended to use prior-task information without additional training or fine-tuning.Source: §I · pp. 1–2
• After hyperparameter optimization, Porimon-based players statistically significantly outperform PokéLLMon-based and heuristic players; retrieval extension contributes, while Long-Term KAG’s contribution remains inconclusive.Source: §I · p. 2
• Tournament experiments comprise 15,000 battles across hyperparameter optimization, ablations, and comparisons, evaluated with Bradley–Terry modeling and corrected Wald tests.Source: §I · p. 2

A. Game Information Retrieval

• The retrieval layer draws on Smogon data about Pokémon and moves, type effectiveness, abilities and items, and API-calculated damage ranges for available moves.Source: A. Game Information Retrieval · p. 2

B. Short-Term Knowledge Augmented Generation

• Battle events are converted to natural-language segments separated by [sep], and a sliding window of size T retains up to T previous turns.Long-Term KAG summarizes the last 12,000 log characters; once 10 experiences are stored, it retrieves the three highest-scoring items and includes their summaries and tactical lessons in the prompt.Source: B. Short-Term Knowledge Augmented… · p. 2, C. Long-Term Knowledge Augmented… · p. 3

C. Long-Term Knowledge Augmented Generation

• After each battle, Short-Term KAG’s log supports summaries of the battle, tactical lessons, Pokémon importance scores, and exact team information for reuse.One example records a Rattata’s Focus Sash, Endeavor, and Quick Attack strategy as a tactical lesson and assigns it high importance.Source: C. Long-Term Knowledge Augmented… · p. 2
• Pokémon similarity combines Euclidean distance over numerical attributes with Jaccard similarity over categorical attributes, balanced by hyperparameter α.The representation includes base stats and remaining HP as a vector, and type, ability, and status condition as a set.Source: C. Long-Term Knowledge Augmented… · p. 2
• Team similarity treats Pokémon order as irrelevant and averages both directional comparisons because the directional measure is not always symmetric.The importance-weighted team score uses Pokémon ratings from memory and normalizes by total importance.Source: C. Long-Term Knowledge Augmented… · pp. 2–3
• Long-Term KAG scores stored experiences by similarity to the current battle state, weighting the opponent’s team by λ = 0.7 and offering an importance-weighted alternative based on Pokémon ratings.Because ratings exist only for stored experiences, the weighted score is averaged with the unweighted current-state score to keep it approximately symmetric.Source: C. Long-Term Knowledge Augmented… · pp. 2–3

D. Exception Handling

• If the LLM response cannot be parsed or gives an invalid action, the agent retries up to manually set Tr attempts, then chooses the move with the highest base power; replacement Pokémon are selected randomly when switching is required.A valid response is executed as generated.Source: D. Exception Handling · p. 3

A. Environment Configuration

• The evaluation uses Gen8 Random Battle, where Pokémon are randomly assigned to both players to reduce team-composition influence, and disables Dynamax to simplify the environment and remain consistent with PokéLLMon.Dynamax temporarily increases maximum and current HP and grants access to Max Moves for three turns.Source: A. Environment Configuration · p. 3

• All LLM-based players use GPT-5.6 Luna with default sampling parameters and reasoning effort set to none because of budget limits.The configuration also sets Tr = 1 to reduce LLM requests.Source: B. Player Configuration · pp. 3–4
• The agent uses a locally deployed Smogon Damage Calculation API for precise damage calculation.Source: B. Player Configuration · p. 4
• Player-name prefixes identify Porimon with or without Long-Term KAG, PokéLLMon, and heuristic players.For lt0 players, v disables the game-information retrieval extension and w keeps only that extension; lt1 suffixes uniform and weighted identify memory retrieval strategies.Source: B. Player Configuration · p. 4
• T and α suffixes encode their parameter values, and suffixes override the corresponding inherited winner configuration; the default T is 5.Source: B. Player Configuration · p. 4

• The three stages optimize Porimon’s α, then all LLM players’ T, and finally compare heuristic, p, lt0, wa, and wb while running ablations.The Long-Term KAG ablation also compares lt0 v against p and lt0.Source: C. Multistage Tournament Evaluation · p. 4
• An additional round-robin among lt0, lt0 v, and lt0 w tests the game-information retrieval extension.Source: C. Multistage Tournament Evaluation · p. 4
• Each of 30 matchups runs 500 battles, totaling 15,000; Long-Term KAG players start each matchup with empty, matchup-specific memory.This prevents information leakage across matchups.Source: C. Multistage Tournament Evaluation · p. 4

• Hyperparameter selection uses the highest average win rate in each round-robin, since pairwise outcomes may be nontransitive.Source: D. Statistical Methods · p. 4
• The Bradley–Terry model estimates player performance using the heuristic player as reference, followed by pairwise and linear-contrast Wald tests.Holm–Bonferroni correction controls the test error rate.Source: D. Statistical Methods · p. 4
• Additional round-robin comparisons use corrected binomial tests and Wilson-score 95% confidence intervals.The Wilson interval is used for its robustness in edge cases.Source: D. Statistical Methods · p. 4

A. Hyperparameter Optimization

• For α, no evaluated agent significantly outperforms another; the best tested values are 0.7 for uniform Porimon and 0.3 for weighted Porimon.Source: A. Hyperparameter Optimization · p. 4
• T = 10 is optimal for both Porimon variants and PokéLLMon; only PokéLLMon’s T = 10 player significantly outperforms its T = 5 player (padj ≈0.025).Source: A. Hyperparameter Optimization · p. 4

B. Comparison Test

• Uniform and weighted Porimon significantly outperform PokéLLMon (padj ≈0.003, βdiff ≈0.196; padj ≈0.008, βdiff ≈0.180, respectively).Source: B. Comparison Test · p. 5
• All Porimon- and PokéLLMon-based agents significantly outperform heuristic (padj < 5 × 10−5, βdiff ≥0.499).The two Porimon variants do not significantly differ (padj ≈0.775, βdiff ≈0.016).Source: B. Comparison Test · p. 5

C. Ablation Study

• In the multistage tournament, extended game-information retrieval increases performance without statistical significance (padj ≈1.000, βdiff ≈0.062).Long-Term KAG also does not significantly outperform its counterpart without the feature: uniform padj ≈0.772, βdiff ≈0.080; weighted padj ≈1.000, βdiff ≈0.063.Source: C. Ablation Study · p. 5
• In the additional round-robin, lt0 significantly outperforms lt0 v (padj ≈4 × 10−4), while comparisons involving lt0 w are nonsignificant.The authors note the difference from the multistage result may reflect randomness or the number of hypotheses in the Holm–Bonferroni correction.Source: C. Ablation Study · p. 5
• Adding components or optimizing a hyperparameter yields monotonically increasing Porimon performance across evaluated configurations, but not significantly (padj ≈0.169 for both variants).Source: C. Ablation Study · p. 5

V. DISCUSSION AND CONCLUSION

• After hyperparameter optimization, Porimon agents significantly outperform both PokéLLMon-based agents and the rule-based heuristic player.The observed Long-Term KAG improvements were small and not statistically significant, so its contribution remains inconclusive.Source: §V · p. 5
• The authors identify potentially insufficient battles per match-up and indefinite retention of Long-Term KAG experience items as implementation and experiment limitations.They also note that retrieval uses similarity alone, regardless of whether an experience item is useful.Source: §V · p. 5
• Future work could evaluate Porimon in other games requiring opponent-aware planning, adapting game-specific retrieval and state-similarity metrics while retaining LSTKAG.Source: §V · p. 5