Source-linked AI summary
Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks
Xiyang Wu, Zongxia Li, Guangyao Shi, Alexander Duffy, Tyler Marques, Matthew Lyle Olson, Tianyi Zhou, Dinesh Manocha
TL;DR
Long-horizon game agents need to discover and reuse structured skills across episodes, but LLM decision agents lack a mechanism for doing so consistently. COS-PLAY couples skill-guided LLM decision-making with an agent-managed pipeline that learns from unlabeled rollouts, achieving strong single-player gains while remaining competitive in multiplayer games.
Problem
Long-horizon tasks require multi-step reasoning and reusable skills, while LLM agents lack mechanisms to discover, retain, and reuse structured skills across episodes.
Method
COS-PLAY co-evolves an LLM decision agent that retrieves skills with a skill-bank pipeline that discovers, contracts, refines, and updates skills from unlabeled trajectories.
Results
25.1% average improvement over GPT-5.4 on single-player games was achieved over 16 runs, while COS-PLAY remained competitive on multiplayer social reasoning tasks.
Takeaways & Limitations
Reusable skill abstractions provide compact guidance for action selection, while continual refinement supports stable co-evolution and efficient adaptation.
Takeaways & Limitations
COS-PLAY relies on compact text-based state summaries, limiting grounded multimodal understanding and potentially missing evidence in visual observations or long temporal dependencies.
Abstract
from arXiv · showhide
Long horizon interactive environments are a testbed for evaluating agents skill usage abilities. These environments demand multi step reasoning, the chaining of multiple skills over many timesteps, and robust decision making under delayed rewards and partial observability. Games are a good testbed for evaluating agent skill usage in environments. Large Language Models (LLMs) offer a promising alternative as game playing agents, but they often struggle with consistent long horizon decision making because they lack a mechanism to discover, retain, and reuse structured skills across episodes. We present COSPLAY, a co evolution framework in which an LLM decision agent retrieves skills from a learnable skill bank to guide action taking, while an agent managed skill pipeline discovers reusable skills from the agents unlabeled rollouts to form a skill bank. Our framework improves both the decision agent to learn better skill retrieval and action generation, while the skill bank agent continually extracts, refines, and updates skills together with their contracts. Experiments across six game environments show that COSPLAY with an 8B base model achieves over 25.1 percent average reward improvement against four frontier LLM baselines on single player game benchmarks while remaining competitive on multi player social reasoning games.
1 Introduction
COS-PLAY addresses the interdependence between skill banks and LLM decision agents in long-horizon games through joint co-evolution. It combines skill-guided action with unsupervised skill discovery and reports gains across six game environments.
- External skill banks package reusable procedural knowledge with execution guidance and applicability constraints.
- The skill bank and decision agent are interdependent: effective skill selection requires useful skills, while decision quality depends on available skills.
- Games expose long-horizon requirements including multi-skill sequences, recurring subgoals, delayed rewards, compositional strategies, and limited demonstrations.
- COS-PLAY jointly trains an LLM decision agent and a skill bank agent that discovers, contracts, and refines reusable skills from unlabeled rollouts.
- 25.1% average gain over four frontier LLM baselines is achieved by the 8B COS-PLAY model on single-player games, while remaining competitive on multiplayer social reasoning tasks.
2 Related Work
Prior work establishes games as demanding benchmarks for long-horizon reasoning and highlights memory and skill reuse as mechanisms for self-improving agents. COS-PLAY focuses on extracting, refining, and feeding reusable skills back into decision-making across diverse games.
- Game agents and benchmarks test memory, planning, and real-time action under delayed rewards and complex dynamics.
- COS-PLAY studies extracting reusable skills from trajectories, refining them over time, and feeding them back into long-horizon decisions across games.
- Memory- and skill-augmented agents reuse past experience and executable skills instead of solving each step from scratch in complex tasks.
3 Problem Formulation
COS-PLAY represents interactive gameplay as trajectories with observations, actions, rewards, transitions, and intentions, then organizes reusable skills in a continuously refined bank. The decision agent retrieves skills and updates intentions before acting, while accumulated rollouts drive bank updates.
- An interactive game environment is defined by observation and action spaces, transition dynamics, rewards, discounting, and a finite horizon.
- The latent intention z_t represents the agent’s strategic interpretation and may guide either the next action or behavior over several steps.
- A skill is a reusable, temporally extended behavior abstraction stored with a summary, pre-condition, plan, termination criteria, and state-change contract.
- The skill bank is continuously refined with new trajectory evidence by segmenting rollouts, refining existing skills, and storing emerging skills.
- At each timestep, the decision agent retrieves candidate skills, selects one using observation and intention, updates intention, and executes an action.
- The framework couples gameplay and skill learning through a decision agent, a skill bank agent, and a skill bank updated after episodes.
- The accumulated rollout set is passed to a skill-bank update pipeline that continually aligns reusable skills with the evolving decision policy.
4 Methodology
COSPLAY combines a skill-augmented LLM decision agent with a skill-bank agent that discovers, verifies, and maintains reusable skills from rollouts. The agents co-evolve through a closed loop in which retrieved skills guide action and new trajectory evidence updates the bank.
- Decision Agent: The decision agent maintains a state summary, intention, and active skill while retrieving or switching skills and executing primitive actions.Skill plans guide multi-step behavior but do not replace low-level action control.
- Skill Bank Agent: The skill-bank agent extracts reusable skills from decision-agent rollouts through boundary proposal, segmentation, contract learning, and bank maintenance.Its maintenance operations include refinement, materialization, merging, splitting, and retirement.
- Skill Discovery: Boundary proposal scores timestep-level transition signals and merges nearby high-scoring candidates before later disambiguation by segmentation.Signals include predicate flips, intention changes, reward or event spikes, surprisal peaks, and transitions between primitive actions and skill selection.
- Skill Discovery: Segmentation matches observed segment effects against learned skill contracts, labels matching segments with existing skills, and marks unmatched behavior as new skills.New-skill segments remain provisional until multiple instances support a reusable effect contract.
- Contract Learning: Contract learning retains consensus state changes across decoded skill segments and treats infrequent effects as noise.An LLM summarization module may enrich contracts when sufficient evidence exists, followed by contract verification.
- Co-Evolution: The decision and skill-bank agents form a mutually dependent closed loop: the bank guides interaction, while resulting trajectories support skill extraction, contract inference, and refinement.Both agents are updated through the co-evolution process described for COSPLAY.
5 Experiments
COS-PLAY is evaluated across six game environments using structured textual states and discrete text actions. It improves single-player performance with few-shot adaptation, remains competitive in social games, and co-evolution supports aligned, reusable skills.
- Evaluation Setup: COS-PLAY evaluates 2048, Candy Crush, Tetris, Super Mario Bros., Avalon, and Diplomacy under a unified game-playing setup.Single-player tasks use native rewards; Avalon uses team win rate, and Diplomacy uses occupied supply centers.
- Main Results: 25.1% average improvement over GPT-5.4 is achieved on single-player games across 16 evaluation runs.The framework requires at most 25 adaptation iterations per game.
- Main Results: In multi-player games, COS-PLAY trails comparable models by 1% in Avalon win rate and outperforms GEMINI-3.1-PRO by 8.8% in Diplomacy.These results concern social reasoning games evaluated against GPT-5.4 as the opponent model.
- Skill Reusability Analysis: The Diplomacy skill bank expands strategic diversity while staying compact: about 55–70 skills remain active, although 121 are discovered overall.Periodic merge-and-split curation removes 53 redundant skills.
- Ablation Study: Co-evolution outperforms isolated decision-agent or skill-bank variants because mismatched banks become less aligned with changing policies and state distributions.SFT without skills improves formatting but lacks long-horizon structure, while GRPO without skills remains unstable under sparse rewards.
- Discussion: COS-PLAY largely preserves general reasoning while reaching strong game performance within 25 co-evolution steps.Math-500 and MMLU-Pro show small performance drops after adaptation.
6 Conclusion
COS-PLAY co-evolves an LLM decision agent and an agent-managed skill pipeline, using unlabeled rollouts to build and refine reusable skills. Across games, it improves single-player performance and remains competitive in multi-player social reasoning, but text-based state summaries limit multimodal grounding.
- Conclusion: COS-PLAY jointly trains skill retrieval, action generation, skill extraction, contract refinement, and skill-bank updates.The decision agent retrieves skills while the skill pipeline discovers reusable skills from unlabeled rollouts.
- Conclusion: The framework improves long-horizon control by keeping the policy and skill bank aligned over time.Experiments span diverse game environments and compare COS-PLAY with frontier LLM baselines.
- Limitations: Compact text-based state summaries limit grounded multimodal understanding and may miss visual evidence or long temporal dependencies.Summarization errors can accumulate over long trajectories and reduce skill relevance.
A Reproducibility
COS-PLAY specifies reproducible environments, training settings, and evaluation protocols across six games, with textual observations and discrete actions. The protocol also documents rollout counts, hyperparameters, and opponent-model choices.
- Evaluation Protocol: Single-player results use 16 evaluation rollouts, while multi-player results use 10 rollouts per player; GPT-5.4 serves as the multi-player opponent model.Exact numerical replication may vary because multiplayer opponents use black-box LLM APIs.
- Environment Settings: COS-PLAY evaluates six games using textual or natural-language observations and discrete text actions.The environments span puzzle solving, platform control, and multi-agent social reasoning.
- Environment Settings: 2048 uses a 4 × 4 board, four directional actions, a 200-step cap, and merge-score rewards.
- Environment Settings: Candy Crush uses an 8 × 8 board, dynamic valid swaps, a 50-move cap, and rewards from matches and cascades.
- Environment Settings: Tetris uses macro placement actions on a 10 × 20 board and caps episodes at 200 steps.The state includes the current piece, upcoming pieces, and board statistics.
- Environment Settings: Avalon and Diplomacy expose natural-language social-game states with phase-dependent actions and multi-player interaction.Avalon has up to five quests, while Diplomacy cycles through movement, retreat, and adjustment phases.
D Breakdown of Role-wise Performance on Multi-player Games
COS-PLAY remains competitive in multi-player social games, matching strong baselines in Avalon and outperforming GEMINI-3.1-PRO in Diplomacy. The breakdowns report role-wise Avalon win rates and power-wise Diplomacy supply centers.
- Avalon: Avalon performance is comparable to GEMINI-3.1-PRO and GPT-OSS-120B, trailing by 1% in win rate.The method uses structured states and phase-specific skills for multi-round reasoning.
- Diplomacy: Diplomacy performance is at least 8.8% better than GEMINI-3.1-PRO.The comparison uses mean occupied supply centers across powers.
- Avalon: Avalon’s role-wise breakdown reports 95% confidence win rates for Merlin, Servant, Assassin, and Minion.
- Diplomacy: Diplomacy’s power-wise breakdown reports mean supply centers for each power and the overall mean with 95% confidence intervals.GPT-5.4 is evaluated in self-play, while other models face GPT-5.4.
- Interpretation: The results are attributed to persistent state tracking, cross-turn memory, iterative skill-bank refinement, and skill-guided decision-making.
E Necessity of Using Different LoRA
COS-PLAY assigns separate LoRA adapters to distinct decision and skill-bank functions. A Candy Crush ablation reports that merging these functional adapters worsens training performance.
- Adapter Design: The decision agent uses two LoRAs for skill retrieval and action generation, while the skill-bank agent uses three for segmentation, contracts, and maintenance.
- Ablation: The merged-LoRA variant performs worse during Candy Crush training than stage-specific LoRAs.The comparison uses one LoRA for each agent rather than separate adapters for each functional stage.
- Training: Function-specific adapters receive dedicated reward functions designed to reinforce the capabilities they govern.Both agents are trained with GRPO, and reward computation is CPU-only without additional LLM inference.
- Decision-Agent Rewards: Action-taking combines environment reward with skill-following shaping and a switching cost.The skill-following term rewards progress through active-skill predicates, while switching is penalized with λ_f = 0.1.
- Skill-Bank Rewards: Skill retrieval is trained using delayed rewards for success, temporal efficiency, contract completion, applicability, and retrieval confidence.
- Skill-Bank Rewards: Skill-bank training separately rewards trajectory segmentation, contract quality, and curator decisions about materializing, merging, splitting, or retiring skills.
G Generalization onto General LLM Reasoning Tasks
COS-PLAY largely preserves the base model’s general reasoning ability after adaptation. On Math-500 and MMLU-Pro, performance remains close to QWEN3-8B despite small absolute drops.
- General Reasoning Benchmarks: COS-PLAY stays close to QWEN3-8B, with absolute drops of 0.8% on MMLU-Pro and 1.8% on Math-500.Math-500 measures mathematical reasoning, while MMLU-Pro evaluates broad knowledge and reasoning.
H Training Curve
COS-PLAY’s joint decision-agent and skill-bank training improves single-player performance, while self-play rewards in multiplayer games remain near equilibrium. Its general reasoning performance remains comparable to the base model.
- General reasoning: COS-PLAY remains comparable to the base model on both general reasoning benchmarks.
- Single-player games: Single-player rewards increase steadily across 2048, Candy Crush, Tetris, and Super Mario during co-evolution.The paper attributes these gains to joint optimization of the decision agent and skill bank.
- Multiplayer games: Multiplayer self-play rewards remain flatter because stronger ego agents are matched by equally stronger opponents.The resulting rewards converge near the game-theoretic equilibrium rather than rising monotonically.
- Multiplayer games: Diplomacy shows tight variance bands throughout training, indicating stable learning dynamics in the seven-player negotiation setting.The paper notes that external-opponent evaluation, rather than self-play reward, reveals policy improvement in competitive games.
I Qualitative Analysis
Qualitative analyses compare COS-PLAY with GPT-5.4 across staged play in Candy Crush and Diplomacy, examine failures, and illustrate skill retrieval as structured exploration. The analyses connect co-evolution to trajectory-level decisions and distinct failure modes.
- Step-level case studies: Step-level case studies compare COS-PLAY and GPT-5.4 across early, mid, and late stages in Candy Crush and Diplomacy.Figures 5–8 include strategy annotations and brief justifications for key decisions.
- Training dynamics: Single-player co-evolution curves show steady gains, while multiplayer self-play remains flat as all players improve symmetrically toward equilibrium.
- Failure analysis: In Diplomacy, COS-PLAY fails through stagnation in 5/28 episodes plateauing at 3 SC, whereas GPT-5.4 collapses in 16/60 episodes to 1–2 SC.The comparison characterizes stagnation as the safer failure mode.
- Skill retrieval: Skills act as a curriculum schedule for Diplomacy exploration by imposing temporal structure, broadening the action distribution, and establishing a safety floor.