PaperlayerView exact revision on arXiv

Source-linked AI summary

PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordination

Paul Teiletche, Kanghyun Ryu, Kyungjoon Park, Dhruva Tirumala, Markus Wulfmeier, Negar Mehr

arXiv:2610.15969v1cs.AIcs.MA

TL;DR

Zero-shot coordination is difficult because an agent must cooperate with unseen partners whose strategies and conventions are unknown, while trajectory-based diversity may not capture strategy differences. PopGen uses LLMs to generate semantic behavior descriptions and executable policies; cooperators trained against its populations perform best across four collaborative environments among population-play baselines.

  • Problem

    Zero-shot coordination requires cooperation with unseen partners, and trajectory-based diversity measures may not capture strategically distinct behaviors.

    Source: §1 · p. 1

  • Method

    PopGen uses LLMs to generate semantically diverse behavior descriptions and translate them into executable policies refined with structured advisor feedback.

    Source: §1 · p. 2

  • Results

    PopGen-trained cooperators achieve the best performance across four collaborative environments against other population-play baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that semantic diversity grows with population size and that policy code yields more faithful behaviors and stronger cooperators than LLM-generated reward functions.

    Source: §1 · p. 2

    Evaluation uses synthetic partners only; a human study is still needed, and LLM-based generation may vary across runs.

    Source: §6 · p. 9

Abstract

from arXiv · show

In multi-agent reinforcement learning, exposing a cooperator to partners with diverse strategies during training is key to better collaboration at test time. However, existing methods typically rely on trajectory-level proxies to measure diversity, which may label two distinct trajectories of the same strategy as diverse. Instead, our key insight is to leverage LLMs' ability to describe behaviorally distinct agents in environments, similar to how humans can recognize high-level strategies. To this end, we introduce PopGen, a framework that synthesizes semantically diverse agents from scratch using LLMs. PopGen first generates a diverse set of textual descriptions of agent behaviors, then translates each behavior into a code policy that can be executed within the environment. We show that cooperators trained against PopGen populations achieve robust zero-shot cooperation with unseen partners, outperforming population-play baselines across four collaborative environments. We further show that semantic diversity scales with PopGen population size, improving the cooperator's worst-case performance, and we identify design choices that preserve population quality while maximizing diversity. Our code is available at https://github.com/labicon/PopGen.

1 INTRODUCTION

Zero-shot coordination requires cooperation with unseen partners whose strategies and conventions may differ. PopGen uses LLMs to generate semantically diverse behaviors and executable policies; cooperators trained against its populations perform strongly across collaborative environments.

1 INTRODUCTION

  • Zero-shot coordination asks a cooperator to work with unseen partners whose conventions, skill levels, and strategies are unknown at test time.Cooperative tasks can have equally effective but mutually incompatible conventions, so training around one convention may fail with a partner using another.Source: §1 · p. 1
  • Trajectory-based diversity measures can distinguish different paths taken under the same strategy, leaving coordination-relevant strategies uncovered.For example, chefs may take different kitchen trajectories while both only carry onions.Source: §1 · p. 1
  • PopGen generates semantically distinct behavior descriptions with LLMs, then translates them into executable policies refined through advisor feedback.The framework uses natural language to express high-level behavioral differences.Source: §1 · pp. 1–2
  • The authors report that semantic diversity grows with population size and that policy code yields more faithful behaviors and stronger cooperators than LLM-generated reward functions.They also describe ablations linking semantic diversity to downstream cooperation.Source: §1 · p. 2
  • PopGen-trained cooperators achieve the best performance across four collaborative environments against other population-play baselines.The paper also reports robust cooperation with novel partners.Source: §1 · p. 2

2 RELATED WORK

Prior population-generation methods use trajectory, action, or latent-space diversity, while PopGen uses text-level semantics to represent strategy differences. For policy instantiation, it uses executable decision logic rather than reward-generated RL policies or costly LLM action calls.

Generating diverse populations for ZSC

  • Population-generation methods range from pooling checkpoints to optimizing diversity over trajectories or actions and sampling from generative models of human play.These approaches use trajectory, action, or latent spaces, whose distances may not reflect strategic distinctions humans recognize.Source: §2 · p. 2
  • PopGen instead uses semantic diversity captured at the text level to generate diverse agent populations.This represents differences in terms of strategies humans might recognize.Source: §2 · pp. 2–3

LLM-generated agents

  • LLM-generated agents can act directly, generate rewards for RL, or encode decision logic as executable policy code.Direct LLM action calls are costly, while reward optimization may fail to produce the intended behavior; the paper reports that policy code more reliably realizes specified behaviors and improves cooperator performance.Source: §2 · p. 3

3 PROBLEM FORMULATION

The paper formulates coordination in a Dec-POMDP and studies symmetric two-player population play. A partner population is associated with natural-language behavior descriptions, and the cooperator is trained to maximize expected return across that population.

3 PROBLEM FORMULATION

  • The Dec-POMDP formulation specifies agents, states, actions, observations, transitions, a shared reward, and a discount factor.Each agent acts from its observation history, and the transition function gives the next-state distribution from the current state and joint action.Source: §3 · p. 3
  • In symmetric two-player population play, a population of diverse partner policies is constructed before training the cooperator.The agents are exchangeable; for notation, agent 1 is the cooperator and agent 2 the partner.Source: §3 · p. 3
  • The cooperator is trained to maximize expected return averaged over partners sampled from the constructed population.Each partner policy is associated with a natural-language behavior description, and the descriptions form a behavior set.Source: §3 · p. 3

4 POPGEN: GENERATING DIVERSE AGENT POPULATIONS

POPGEN generates semantically distinct behavior descriptions, turns them into executable policies, and uses them as partner populations for zero-shot coordination. It measures and filters behavior quality, then checks policy adherence through repeated rollouts and iterative refinement.

4 POPGEN: GENERATING DIVERSE AGENT POPULATIONS

  • POPGEN synthesizes partner populations in two stages, first generating behavior descriptions and then translating them into policies for zero-shot cooperation.The framework is intended to supply a population for training a cooperator.Source: §4 · p. 3

4.1 STAGE 1: GENERATING SEMANTICALLY DIVERSE BEHAVIORS

  • Stage 1 measures semantic diversity as the average pairwise distance between unit-normalized embeddings of behavior descriptions.The method uses textual descriptions because different actions can reflect the same strategy.Source: §4.1 · pp. 3–4
  • Behavior quality is scored on realism, relevance, and specificity, each graded from 0 to 5 by an LLM and summed out of 15.POPGEN oversamples candidate descriptions, then queries the LLM again to select the highest-quality candidates.Source: §4.1 · p. 4

4.2 STAGE 2: INSTANTIATING BEHAVIORS AS AGENTS

  • Stage 2 uses an LLM to implement each behavior as executable Python policy code, avoiding an RL training run for every behavior.The policies are evaluated for adherence by a VLM that classifies whether rollout trajectories align with the intended behavior across multiple initial states.Source: §4.2 · p. 4

A POPGEN population can serve directly as the partner population for population play. We call this POPPLAY, which trains a cooperator π1

  • POPPLAY achieves the highest mean return in all four environments against held-out partners, exceeding CoMeDi in PMR and overcooked with overlapping error bars.Figure 2 summarizes average team reward across held-out partners; the table reports per-environment evaluation rewards.Source: Figure 2 · p. 5, A POPGEN population can serve directly… · p. 5

4.2 STAGE 2: INSTANTIATING BEHAVIORS AS AGENTS

  • When adherence is below target, an LLM iteratively revises a candidate policy using feedback from a separate advisor until it reaches the target or attempts are exhausted.The procedure relies on the adherence measure from rollout evaluation.Source: §4.2 · pp. 4–5

A POPGEN population can serve directly as the partner population for population play. We call this POPPLAY, which trains a cooperator π1

  • POPPLAY trains a cooperator with PPO against frozen POPGEN policies, sampling harder-to-coordinate partners more often through rank-based prioritization.Executable partner policies add negligible training-loop overhead compared with learned generative models or live populations of RL agents.Source: A POPGEN population can serve directly… · p. 5

5 EXPERIMENTS

The experiments test POPGEN populations for zero-shot coordination with held-out partners across four environments, then examine population size, behavior quality and diversity, and policy-code modeling. POPPLAY generalizes robustly, while larger and more diverse populations and executable policy-code partners show further benefits.

5.1 EXPERIMENTAL SETUP

  • Across four environments, the setup compares POPPLAY with four population-play baselines using held-out scripted partners whose behaviors were not given to the LLMs.The environments are Point-Mass Rendezvous, Level-Based Foraging, Predator-Prey, and overcooked.Source: §5.1 · pp. 5–6

5.2 TRAINING COOPERATORS WITH POPGEN POPULATIONS

  • POPPLAY achieves the highest mean return in every environment and maintains the highest worst-case rewards against held-out partners.Self-Play and FCP fall to zero reward with their worst partners in PMR and overcooked.Source: §5.2 · pp. 6–7, Figure 3 · p. 6
  • In overcooked, POPPLAY shifts among subtasks to complement its partner, while other methods often focus on a subset of actions.Figure 3 illustrates role adaptation; Table 8 breaks down subtask counts by held-out partner.Source: Figure 3 · p. 6, §5.2 · p. 7

5.3 GENERATING DIVERSE POPULATIONS FOR COOPERATOR TRAINING

  • Larger POPGEN populations increase semantic diversity; average team reward gains plateau beyond n = 4, while worst-case reward jumps from near zero at n ≤ 2 to up to 39 on average from n = 3.The fixed training budget uses eight partner slots, replicating the n distinct agents as needed.Source: §5.3 · pp. 7–8, Figure 4 · p. 7
  • For 30 behaviors, POPGEN reaches diversity 0.728, near VS at 0.726, while quality is 12.74 vs. 12.03; DIR has quality 14.45 and diversity 0.446.Compared with SemDiv, POPGEN gives up 6% of quality for a 30% gain in diversity.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 MODELING AGENT BEHAVIORS AS POLICY CODE FOR POPULATION PLAY

  • With the same textual behaviors, policy-code partners yield higher test-time reward in all four environments than reward-function partners.Policy-code adherence averages 0.66 ± 0.07 vs. 0.43 ± 0.07; the gap is widest in LBF (0.61 vs. 0.13) and overcooked (0.63 vs. 0.41).Source: §5.4 · pp. 8–9, Figure 6 · p. 9

6 CONCLUSION

The paper concludes that semantic diversity can support robust cooperation with unseen partners, and that executable policy code improves behavior adherence and cooperator robustness. Its evaluation uses synthetic partners, so whether the approach improves human-AI coordination remains unconfirmed.

6 CONCLUSION

  • POPGEN’s text-to-policy pipeline produces diverse populations, and cooperators trained against them outperform population-play baselines in all four studied environments.The paper links semantically diverse training partners to better worst-case performance and reports more faithful behavior adherence with executable policy code than with LLM-generated reward functions.Source: §6 · p. 9
  • Evaluation is limited to synthetic partners; a human study is still needed to confirm improved human-AI coordination.Generated populations and cooperator performance may also vary across runs because both behavior and policy generation rely on LLMs.Source: §6 · p. 9
  • Future work could generate behaviors online to target current cooperator weaknesses, while refinement quality is bounded by the context provided to the LLM.Source: §6 · p. 9

AI USE STATEMENT

The statement describes the authors’ uses of generative AI tools and says AI-assisted work was reviewed by the authors.

AI USE STATEMENT

  • Generative AI supported methodology, experiments, implementation, analysis, writing, figures, and code; authors reviewed the AI-assisted work and verified generated code and figures.The statement excludes AI assistance for developing theoretical models, conceptual frameworks, and mathematical claims or proof ingredients.Source: AI USE STATEMENT · p. 10

A.1 ENVIRONMENTS

The section describes four cooperative environments, an iterative policy-refinement loop, and how population-play baselines construct partner populations.

A.1 ENVIRONMENTS

  • PMR, PP, LBF, and Overcooked test landmark convergence, prey capture, pooled carrying capacity, and sequential soup preparation, respectively.The environments span discrete and continuous action spaces; PP prey follow fixed policies, so only the ego team is generated and trained.Source: Figure 7 · p. 14, Table 1 · p. 14

A.2 ITERATIVE REFINEMENT FOR ADHERENCE

  • The refinement loop tests candidate policies over rollouts, accepts one when its adherence score reaches a threshold, and otherwise regenerates it using targeted feedback.If no attempt reaches the threshold within the iteration limit, the highest-adherence policy is returned.Source: §A.2 · p. 15

B.1 BASELINES

  • Population-play baselines first build partner populations, then train a cooperator against them; they mainly differ in how they generate partners.POPGEN replaces the partner-generation stage with language-guided policy code generation.Source: §B.1 · p. 15
  • Self-Play diversity comes only from initialization and sampling noise, while FCP adds intermediate checkpoints to include partners of lower skill.FCP retains checkpoints from each self-play run.Source: §B.1 · p. 16
  • MEP jointly trains partners with an entropy bonus on their mean policy, encouraging population-level diversity.The bonus decomposes into individual and collective terms.Source: §B.1 · p. 16
  • CoMeDi seeks high self-play return and low return with the nearest existing convention, while mixed-play return discourages deliberate failure against that partner.Mixed-play switches from pairing the candidate with its nearest neighbor to candidate self-play at a random step.Source: §B.1 · p. 16

B.2 HELD-OUT EVALUATION PARTNERS

The evaluation uses designed partners held out until testing and specifies cooperator-training procedures and environment-specific frame budgets.

B.2 HELD-OUT EVALUATION PARTNERS

  • Cooperators are evaluated against four designed partners per environment that they never see during training.The partners are paired with cooperators only at evaluation time.Source: §B.2 · p. 17, Table 2 · p. 17

B.3 HYPERPARAMETERS

  • Cooperators are trained with PPO using recurrent actors and critics, while partner policies remain frozen.Training populations have 4 partners for POPPLAY, Self-Play, MEP, and CoMeDi, and 12 for FCP.Source: §B.3 · p. 17
  • Training budgets are 4M frames for LBF and Overcooked, 30M for PMR, and 80M for PP.Source: §B.3 · p. 18

B.4 RESULTS TABLE

POPPLAY leads evaluation reward in every environment and is selected by most held-out partners, while analyses show convention collapse in Self-Play populations.

B.4 RESULTS TABLE

  • POPPLAY attains the highest evaluation reward in every environment.Table 4 reports average team reward against held-out evaluation partners and training partners; the eval–train reward difference measures reward lost to out-of-distribution partners.Source: Table 4 · p. 18, §B.4 · p. 18
  • 14 of 16 held-out partners pick POPPLAY as the most cooperative agent; the remaining two, both in Overcooked, pick CoMeDi.Table 5 breaks down evaluation reward by individual partner.Source: §B.4 · p. 18, Table 5 · p. 19

B.5 MODE COLLAPSE IN SELF-PLAY POPULATIONS

  • In PMR, a four-policy Self-Play population covers an average of 2.8 of four landmarks, and all four policies in Run 5 settle on West.Different seeds can converge on different landmarks, but each policy commits to one landmark.Source: §B.5 · p. 19, Table 6 · p. 19
  • Overcooked Self-Play pairs mostly put onions into the pot and rarely fetch a dish or scoop soup; FCP checkpoints follow the same behaviors at different skill levels.The table reports each pipeline stage’s share of all stage actions performed by both agents.Source: §B.5 · p. 19, Table 7 · p. 20

B.6 TASK EVENT COUNTS IN OVE R C O O K E D

The section analyzes Overcooked task events and compares behavior-generation strategies, including the diversity–quality trade-off of oversampling variants.

B.6 TASK EVENT COUNTS IN OVE R C O O K E D

  • Against held-out Overcooked partners, POPPLAY shifts toward subtasks they leave open, while Self-Play, FCP, and MEP remain focused on plating.The per-partner breakdown tracks counts of cooperator task events alongside partner counts.Source: §B.6 · p. 20, Table 8 · p. 20

C DIVERSE BEHAVIOR GENERATION

  • The generation experiments also include Football and SMACv2 because the experiments were cheaper to run in those environments.Football tests role-switching team play, while SMACv2 tests allied combat under partial observability.Source: C DIVERSE BEHAVIOR GENERATION · p. 21
  • Behavior-generation ablations compare instance-, list-, and distribution-level methods; VS-OS oversamples verbalized-probability candidates, then merges and deduplicates them.The method is designed to separate diversity exploration from quality control.Source: C DIVERSE BEHAVIOR GENERATION · p. 21
  • Sequential ×2, the variant used as VS-OS, achieves the best overall diversity–quality trade-off across environments.Diversity is average pairwise embedding distance, and quality is total score.Source: Table 10 · p. 22

C.1 OVERSAMPLING TO PRESERVE QUALITY IN BEHAVIOR GENERATION

Oversampling followed by merging preserves semantic diversity better than multi-turn refinement and scales more effectively with a stronger generator, at the cost of latency. The paper also evaluates how LLM-generated policies adhere to their behavior descriptions on development and held-out initializations.

C.1 OVERSAMPLING TO PRESERVE QUALITY IN BEHAVIOR GENERATION

  • Sequential ×2 achieves the best average quality (13.00) while remaining within 0.005 of the highest average diversity (0.696 versus 0.701), so it is the default VS-OS configuration.All oversampling variants improve quality over vanilla VS while maintaining comparable diversity; increasing Sequential to ×4 lowers diversity.Source: §C.1 · p. 22
  • Football and SMACv2 are difficult to include in the rest of the paper because long rollouts make training take substantially longer.The passage cites football’s long rollouts and the slow StarCraft backend in SMACv2.Source: §C.1 · p. 22
  • VS-OS gains both quality (+0.57) and diversity (+0.04) when moving from Flash to Pro, while VS-MT gains quality (+1.00) but loses diversity (−0.06).VS-OS also recovers worst-case distance and coverage relative to VS-MT; Figure 9 contrasts the variants and their scaling behavior.Source: §C.1 · p. 22, Figure 9 · p. 23
  • Verbalized Sampling’s diversity gains transfer from creative writing: prompting the model to verbalize a distribution improves diversity across VS variants.SimpleStrat falls below the Pareto front at 13.07 quality and 0.532 diversity; the passage attributes this to difficulty identifying meaningful diversity axes for agent strategies.Source: §C.1 · p. 23

D MODELING BEHAVIORS IN AGENTS WITH LLMS

D.1 BEHAVIOR MODELING METHODS

Policy Code achieves the highest reported average adherence and has low generation and rollout latency. The section also reports gains from iterative refinement and describes an advisor, an adherence evaluator, and a behavior-quality judge.

D.1 BEHAVIOR MODELING METHODS

  • Policy Code achieves the highest average adherence (81.3% dev; 63.8% test) and requires 8.1s for generation and 9.8s per rollout on average.The latency comparison reports 2878.6s generation for RF and 110.1s rollout for LLM-A; SMACv2 is excluded from the latency table because engine overhead dominates.Source: §D.1 · pp. 24–25, Table 13 · p. 24

D.2 IMPACT OF THE REFINEMENT LOOP ON POPGEN AGENTS’ ADHERENCE

  • Refinement raises average adherence by 23.1 points on dev and from 56.7% to 66.3% (+9.6 points) on test; PP starts near ceiling at 95.0% dev.The reported policies are the best of five attempts, compared with first-attempt policies.Source: §D.2 · p. 25, Table 14 · p. 25
  • With the advisor, LBF refinements make coordination-oriented fixes; without it, the edits patch symptoms and leave teammate-motion information unused.The examples include correcting an inverted direction mapping, selecting food using teammate motion, and choosing an adjacent free support cell.Source: §D.2 · p. 25

E.1 ADVISOR

  • The advisor diagnoses rollout failures against the intended behavior and environment, then gives actionable feedback for the next code attempt.Its prompt directs attention to the most important failure patterns.Source: §E.1 · p. 27

E.2 ADHERENCE EVALUATOR

  • The adherence evaluator focuses on whether the ego agent follows the described behavior, using serialized states as primary evidence and images as support.It considers other agents only when relevant to following the behavior or affecting the ego agent’s ability to demonstrate it.Source: §E.2 · p. 28

E.3 BEHAVIOR QUALITY JUDGE

  • The behavior-quality judge scores descriptions on relevance, realism, and specificity, each using a 0–5 rubric.The rubrics assess environmental fit, plausibility under environment constraints, and actionable detail.Source: §E.3 · p. 30
Loading 2610.15969v1…