Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models

Daniel Hennes; Zun Li; John Schultz; Marc Lanctot

Source-linked AI summary

Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models

Daniel Hennes, Zun Li, John Schultz, Marc Lanctot

arXiv:2603.10098v1Mar 10, 2026cs.GTcs.AIcs.LG

TL;DR

Deep-RL response oracles produce opaque policies that are difficult to interpret, trust, or debug. CSRO replaces them with LLM-generated executable code, extending generation with refinement and context abstraction. The framework achieves competitive performance and low-exploitability behavior while retaining interpretable policies, though scalability and LLM dependence remain limitations.

Problem
PSRO’s deep-RL oracles produce opaque neural-network policies, limiting strategy verification and interpretability in approximate equilibrium computation.
Method
CSRO replaces the deep-RL oracle with an LLM that synthesizes executable policy code, using opponent-context abstraction and zero-shot, iterative, or evolutionary refinement.
Results
CSRO achieves competitive performance and convergence to low-exploitability equilibria while producing fully interpretable policies represented by commented source code.
Takeaways & Limitations
CSRO demonstrates that code-generation oracles can compete with mature baselines while shifting policy construction from opaque parameter optimization toward interpretable algorithmic behavior.
Takeaways & Limitations
CSRO depends on LLM capability and prompt quality, incurs repeated LLM-call costs, and its scalability to vast high-dimensional games remains open.

Abstract

from arXiv · show

Recent advances in multi-agent reinforcement learning, particularly Policy-Space Response Oracles (PSRO), have enabled the computation of approximate game-theoretic equilibria in increasingly complex domains. However, these methods rely on deep reinforcement learning oracles that produce `black-box' neural network policies, making them difficult to interpret, trust or debug. We introduce Code-Space Response Oracles (CSRO), a novel framework that addresses this challenge by replacing RL oracles with Large Language Models (LLMs). CSRO reframes the best response computation as a code generation task, prompting an LLM to generate policies directly as human-readable code. This approach not only yields inherently interpretable policies but also leverages the LLM's pretrained knowledge to discover complex, human-like strategies. We explore multiple ways to construct and enhance an LLM-based oracle: zero-shot prompting, iterative refinement and \emph{AlphaEvolve}, a distributed LLM-based evolutionary system. We demonstrate that CSRO achieves performance competitive with baselines while producing a diverse set of explainable policies. Our work presents a new perspective on multi-agent learning, shifting the focus from optimizing opaque policy parameters to synthesizing interpretable algorithmic behavior.

1. Introduction

CSRO addresses the interpretability limits of PSRO by replacing deep-RL oracles with LLMs that generate executable policy code. It extends LLM-based response oracles with refinement, context abstraction, and external benchmarking.

PSRO computes approximate equilibria by iteratively adding best responses, but deep-RL oracles produce opaque neural-network policies that hinder strategy verification.
CSRO reframes best-response computation as program synthesis, prompting an LLM to generate executable, human-readable policies from game and opponent information.
Pretrained LLM knowledge of logic, planning, and strategy supports sophisticated, human-like policies while reducing costly exploration.
CSRO extends LLM-PSRO with iterative refinement and context abstraction, using strategy summaries and filtering to scale beyond full opponent source code.
Evolutionary refinement enhances zero-shot strategies, while standardized external populations and game-theoretic solvers provide validation beyond internal comparisons.

2. Code Space Response Oracles

CSRO preserves PSRO’s iterative equilibrium-search structure while replacing its deep-RL oracle with an LLM that synthesizes stateful policies as executable code. Prompts provide game, API, and opponent-strategy context, with generated policies added to the evolving policy set.

PSRO preliminaries: PSRO repeatedly solves an equilibrium mixture over its policy set and adds a best response to that mixture.
Code policies: Standard PSRO uses deep reinforcement learning to produce opaque neural-network policies, whereas CSRO generates stateful code policies mapping observations to actions.
Prompt construction: CSRO prompts include game rules, objectives, an executable policy API, and descriptions or source code for opponents’ current meta-strategy.
Prompt construction: Opponent context can be summarized or filtered to keep prompts manageable when many policies receive nonzero equilibrium support.
Algorithm: Each iteration generates a new programmatic policy and adds it to the policy set, enriching the metagame with increasingly sophisticated strategies.
Policy improvement: CSRO uses complementary in-context mechanisms operating on different timescales for immediate tactical corrections and longer-term strategic guidance.

16 end

CSRO improves generated policies through opponent-context analysis and inner feedback loops. Its refinement variants range from direct zero-shot generation to sequential or distributed evolutionary program search.

The oracle analyzes active opponent policies in the equilibrium mixture, including their source code or LLM-generated strategic summaries.
An inner feedback loop evaluates each candidate against the current meta-game and refines it to produce a more robust best response.
Refinement variants: ZeroShot directly generates a program without entering the refinement stage.
Refinement variants: LinearRefinement regenerates losing policies using evaluation feedback until utility is nonnegative or the refinement budget is exhausted.
Refinement variants: AlphaEvolve mutates programs across parallel threads, evolves clustered subpopulations independently, and uses expected utility against the meta-strategy as its score.

3. Experiments

The experiments test CSRO’s equilibrium quality, zero- or few-shot strategy generation, and policy interpretability across repeated Rock-Paper-Scissors and repeated Leduc hold’em. Evaluation uses diverse heuristic populations, Nash strategies, and a deep-RL PSRO baseline.

Research questions: The experiments ask whether CSRO reaches low exploitability, generates effective non-trivial strategies with limited prompting, and is more interpretable than neural policies.
Environments: CSRO is evaluated on repeated Rock-Paper-Scissors and repeated Leduc hold’em, implemented using OpenSpiel.
Environments: Repeated Rock-Paper-Scissors runs for 1000 rounds and tests opponent modeling against patterns that uniform random play cannot exploit.
Environments: Repeated Leduc hold’em runs for 100 hands and tests stateful reasoning with private and public information across multiple decision points.
Evaluation populations: RRPS evaluation uses 43 hand-coded competition strategies spanning random, fixed-sequence, memory-based, switching, neural-network-like, and cognitive architectures.
Evaluation populations: Leduc policies are tested against a CFR+-computed single-hand Nash strategy and heuristic AlwaysCall and AlwaysFold opponents.
Metrics: Population Return, Within Population Exploitability, and Aggregate Score measure average performance, approximate exploitability, and the balance between average- and worst-case outcomes.
Baselines: The conventional baseline is PSRO with an IMPALA-trained deep LSTM best-response oracle, while additional RRPS baselines include an LLM sequence predictor, tabular Q-learning, and Contextual Regret Minimization.

4. Results

Across repeated Rock-Paper-Scissors and Leduc poker, CSRO variants achieve competitive performance while generating interpretable, strategically diverse code policies. Evolutionary refinement and opponent-aware inputs improve robustness, exploitability, and adaptation to predictable opponents.

Repeated Rock-Paper-Scissors: AlphaEvolve achieves population exploitability of 25.2 ± 20.3 in repeated Rock-Paper-Scissors, making it the strongest CSRO variant on this measure.The oracle prioritizes robustness against strong strategies in the meta-game equilibrium.
Prompt and filtering ablations: Description-based ZeroShot reaches an aggregate score of 63.5 ± 11.4, versus −54.3 ± 118.7 for code-based ZeroShot.The no-opponent-input variant instead attains PopReturn of 135.3 ± 10.2 but PopExpl of 614.2 ± 60.8 and aggregate score of −478.9 ± 70.2.
Prompt and filtering ablations: The Top 5 opponent filter generally outperforms Min support by preserving a more diverse opponent set and reducing brittleness from overfitting to one counter-policy.Min support may expose the oracle to only the highest-supported opponent strategy.
Repeated Leduc hold’em poker: In repeated Leduc poker, AlphaEvolve obtains exploitability of 4.4 ± 0.6 and the highest PopReturn and AggScore among the reported CSRO variants.This performance is reported against an evaluation population containing CFR+, AlwaysCall, and AlwaysFold.
Repeated Leduc hold’em poker: Against AlwaysCall, CSRO-AlphaEvolve achieves average return of 110.3 ± 9.7, exceeding PSRO-IMPALA at 57.7 ± 3.3 and CFR+ at 62.1 ± 0.8.LinearRefinement instead performs best against AlwaysFold, with 57.3 ± 8.8.
Interpretable policy analysis: The best LinearRefinement strategy achieves PopReturn of 238.3 through a commented ensemble of 32 predictors spanning Markov, reactive, and heuristic models.Its expert votes are weighted by the fifth power of their scores for rapid adaptation.
Interpretable policy analysis: The generated policies are compositions of interpretable modules that model opponents and expose strategic reasoning, unlike opaque black-box policies.In Leduc, EV combines hand equity with estimated opponent folding tendencies to determine betting behavior.

5. Related Work

The paper builds on PSRO and related LLM-based game-theoretic methods, distinguishing CSRO by generating executable code and adding iterative refinement and external evaluation.

PSRO: PSRO computes approximate equilibria by iteratively adding best responses to a meta-strategy, but standard implementations rely on deep-RL neural-network oracles.These advances have achieved strong performance in difficult games, while retaining the opacity of learned policies.
LLM-based game-theoretic methods: LLM-based game-theoretic methods have used solvers to steer dialogue through strategic prompts, whereas CSRO produces executable policies as code.The distinction is between natural-language prompts for game-playing LLMs and executable code outputs.
LLM-PSRO: CSRO most closely extends LLM-PSRO by adding an inner feedback loop for iterative refinement and evaluating beyond self-play populations.LLM-PSRO demonstrated code-generating best responses but lacked iterative candidate refinement and external-population or baseline evaluation.

6. Conclusion and Discussion

CSRO replaces opaque deep-RL oracles with LLM-generated executable code to compute approximate equilibria, achieving competitive low-exploitability performance while retaining interpretability. The approach remains dependent on model and prompt quality, incurs repeated API-call costs, and has unresolved scalability challenges in high-dimensional games.

Conclusion: CSRO reframes best-response computation as program synthesis, prompting an LLM to generate human-readable, executable policies instead of black-box deep-RL policies.The framework targets approximate equilibria in multi-agent games while making policies inspectable as source code.
Results: CSRO achieves competitive convergence to low-exploitability equilibria, particularly in complex games such as multi-hand Leduc hold’em poker.The conclusion also reports a computational-efficiency contrast with turn-level LLM baselines in RRPS, where LinearRefinement generates a reusable policy.
Interpretability and reasoning: The method's strategic success requires synthesizing best responses to dynamically generated mixtures of programmatic opponents, beyond retrieving strategies for a classic game.This capability is presented as in-context strategic reasoning and code generation.
Limitations: CSRO performance depends on the underlying LLM and prompt quality, while repeated API calls can be costly and scalability to vast observation spaces remains open.Encoding complex states and opponent strategies within current context limits is identified as a substantial engineering challenge.

A.1.1. PSRO-IMPALA

The PSRO-IMPALA section presents an expert-ensemble strategy for repeated Rock-Paper-Scissors, combining many predictive models with adaptive weighting, meta-prediction, and randomized tie-breaking. The implementation is tuned for long repeated games and uses explicit model-update and prediction components.

Adaptive weighting: Expert votes are weighted by the fifth power of their scores, allowing successful predictors to dominate while suppressing weaker models.The scoring scheme is described as aggressive and high-confidence adaptation that locks onto successful models and reduces noise from less-successful experts.
Meta-prediction: The strategy models both the opponent and the opponent’s predictions of its own behavior, then counters the anticipated move.Its meta-predictors include models of the agent’s own Markov and reaction patterns, including a higher-level imitation component.
Strategic randomization: Random selection among tied best options makes the agent’s behavior harder to predict and exploit.Randomization is applied when predictive models or the final vote tally produce ties.

A.2. Prompts and Initial Strategies

The appendix specifies prompts, rules, observations, and interfaces for initial strategies in repeated Rock-Paper-Scissors and repeated Leduc Poker. It also defines how agents act, receive outcomes, and restart across games.

A.2.1. Repeated Rock-Paper-Scissors: Repeated Rock-Paper-Scissors asks an Agent class to represent a strategy for a 1000-round game using ROCK, PAPER, and SCISSORS.The objective is to maximize total wins under the standard move hierarchy.
A.2.1. Repeated Rock-Paper-Scissors: The Rock-Paper-Scissors interface requires an act method that receives an observation dictionary containing the previous moves of both players.Both move fields default to None in the first round.
A.2.2. Repeated Leduc Poker: The Leduc prompt asks for iterative bot improvement across games with randomly permuted player positions and evaluation metrics where larger values are better.The bot is instructed to act as an expert in game algorithms, opponent modeling, and multiagent learning.
A.2.2. Repeated Leduc Poker: Repeated Leduc Poker is presented as a two-player, six-card game with private cards, a shared public card, blinds, and imperfect information.The deck contains two Jacks, two Queens, and two Kings.
A.2.2. Repeated Leduc Poker: Leduc betting uses fixed raise sizes of 2 units pre-flop and 4 units post-flop, with a maximum of two raises per betting round.Player 1 acts first in both betting rounds.
A.2.2. Repeated Leduc Poker: The rules define legal responses across pre-flop and post-flop play, including fold, call, raise, and zero-cost calls when no bet is outstanding.The implementation uses Call instead of Check, and the betting cap restricts later actions.
A.2.2. Repeated Leduc Poker: After pre-flop betting, a shared public card is revealed before post-flop betting, followed by showdown when neither player folds.A hand combines a private card with the public card; pairs rank above high cards, and equal hands split the pot.
A.2.2. Repeated Leduc Poker: The bot API exposes restart, act, and receive_outcome methods for position assignment, observation-based actions, and learning from completed games.Observations include player state, public state, action history, and game results, with actions drawn from FOLD, CALL, and RAISE.