PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-LLM Collaborative Alignment via Stackelberg Games

Christina Hahn, Shangbin Feng, Dean Light, Swastik Roy, Hila Gonen, Yulia Tsvetkov

arXiv:2609.39076v1cs.AIcs.LG

TL;DR

Collaborative training methods commonly sample instructions uniformly, although their usefulness changes as models improve. STACKELBERG ALIGNMENT uses an EXP3 leader to adapt instruction selection while models duel, judge, and learn from peer signals. It achieves the highest macro-average across three heterogeneous pools, improving over training-based baselines by up to 7.4%.

  • Problem

    Prior collaborative methods typically sample instructions uniformly, despite instruction informativeness varying with difficulty, model disagreement, and improvement over training.

    Source: §1 · pp. 1–2

  • Method

    STACKELBERG ALIGNMENT uses an EXP3 leader to adapt instruction sampling from duel informativeness while models generate responses, judge peers, and learn from preference signals.

    Source: §1 · p. 2

  • Results

    STACKELBERG ALIGNMENT achieves the highest macro-average across three heterogeneous model pools and 12 benchmarks, improving over training-based baselines by up to 7.4%.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Ablations confirm that the adaptive curriculum, reputation-weighted judgment, and reputation-based matching each contribute to gains in multi-LLM evolution.

    Source: §6 · p. 10

    The framework requires all pool models to run simultaneously during combat and judgment, increasing memory and compute requirements relative to single-model fine-tuning.

    Source: I LIMITATIONS · p. 21

Abstract

from arXiv · show

A pool of language models can collaborate and improve collectively by learning from one another's responses. These interactions depend on the instructions used during training. Existing methods typically sample instructions uniformly, even though their usefulness may change as the models improve: an instruction on which models' responses once differed in quality may later be answered equally well, while a previously difficult instruction may begin to provide a useful learning signal. We propose Stackelberg Alignment, a game-theory-inspired leader-follower framework that turns instruction selection into an adaptive curriculum. An EXP3 bandit acts as the leader, allocating a fixed sampling budget across instructions and updating its sampling distribution using a reward that combines instruction difficulty and response discriminability. The language models act as followers: they respond to the selected instructions, evaluate one another's responses, and learn from the resulting preference signals through DPO or GRPO. The framework uses Elo-style reputation-weighted peer judgment and reputation-based opponent matching to support reliable and competitive model interactions. Experiments across three heterogeneous model pools and 12 benchmarks spanning scientific discovery, reasoning, code, instruction following, and knowledge show that Stackelberg Alignment achieves the highest macro-average across three diverse model pools, outperforming the strongest training-time baseline by up to 7.4% and the best static inference baseline by 12-25%. Analysis confirms that the adaptive leader concentrates duels on the most informative instructions, and ablations show that both reputation-weighted judgment and reputation-based matching improve the effectiveness of multi-LLM evolution.

1 INTRODUCTION

STACKELBERG ALIGNMENT adaptively selects instructions for collaborative model training, targeting the changing frontier of disagreement. Across three heterogeneous model pools, it achieves the highest macro-average and beats training-time baselines by up to 7.4%.

1 INTRODUCTION

  • The paper identifies adaptive instruction selection as an unexplored component of prior collaborative methods, which typically sample uniformly throughout training.The described training-time methods use model comparisons and competition to produce preference-learning signals.Source: §1 · p. 1
  • Uniform sampling can waste training compute because instruction informativeness shifts as models improve, making once-useful tasks trivial and harder tasks newly learnable.Instructions where all models agree or fail provide little discriminative preference signal; disagreement at the capability frontier is more informative.Source: §1 · pp. 1–2
  • The method casts collaborative alignment as a Stackelberg game, using an EXP3 leader to adapt instruction selection as follower models duel and learn.Its composite reward captures instruction difficulty and duel discriminability; reputation weights peer judgments and guides opponent matching.Source: §1 · p. 2
  • STACKELBERG ALIGNMENT achieves the highest macro-average across three model pools, improving over training-based algorithms by up to 7.4% and the best static inference baseline by 12–25%.Evaluations span 12 benchmarks in scientific discovery, reasoning, code, instruction following, and knowledge.Source: §1 · p. 2

2 METHOD

STACKELBERG ALIGNMENT makes instruction selection an adaptive leader strategy in a collaborative training loop. Reputation-based matching and peer judgment generate signals that followers use for DPO or GRPO updates.

2 METHOD

  • Each iteration draws duels with replacement from the leader’s instruction distribution, so not every instruction is guaranteed to be sampled; the distribution retains a nonzero floor for each instruction.Models respond to selected instructions, peer judges score responses, and the resulting preference pairs support reinforcement-learning fine-tuning.Source: §2 · p. 3
  • The Stackelberg formulation has a leader commit to an instruction distribution before followers respond, while prior paradigms arise as special cases when that distribution is fixed to uniform.With one model, the framework recovers Self-Rewarding and SPIN; with multiple models, uniform sampling recovers SPARTA ALIGNMENT.Source: §2 · p. 3

2.1 STACKELBERG LEADER: ADAPTIVE INSTRUCTION SELECTION

  • EXP3 updates instruction weights from a composite reward that favors learnable difficulty and preference gaps near a scheduled target, which decays over training.The reward penalizes tasks below a minimum score threshold and comparisons that are too easy or too difficult; EXP3 retains exploration while favoring high-weight instructions.Source: §2.1 · pp. 3–4

2.2 SCHEDULED FOLLOWER MATCHING

  • Reputation-based soft matching shifts its target from mismatched opponents early toward near-peers later, moving from clear preference pairs toward finer distinctions.The schedule anneals the target reputation gap from 1 to 0 over training.Source: §2.2 · p. 5

2.3 PEER JUDGMENT AND REPUTATION SYSTEM

  • Non-competing models judge both responses, and a reputation-weighted mean gives greater influence to judges with stronger track records.Reputations are updated after each duel with an adapted Elo rule; the update cools as reputations stabilize and tracks recent volatility.Source: §2.3 · p. 5

2.4 FOLLOWER TRAINING

  • DPO trains on non-tied preference pairs collected across duels, whereas GRPO uses online peer scores as per-completion rewards for assigned prompts.DPO uses the previous iteration’s checkpoint as its reference policy; GRPO derives reputation updates from logged rewards after training.Source: §2.4 · pp. 5–6

3 EXPERIMENT SETTINGS

The experiments compare three heterogeneous model pools and two Stackelberg training variants against static-inference and training-based baselines across 12 datasets. The settings specify pool composition, benchmark coverage, and implementation choices.

Models

  • Three pools cover nine heterogeneous models, eight academic-project models, and four general-purpose models.Pool 1 combines Qwen2.5-7B variants with AgentFlow-Planner-7B, Llama-3.1-8B-Instruct, Aya-Expanse-8B, and Pangea-7B.Source: §3 · p. 6

Baselines

  • The comparison includes Stackelberg (DPO) and Stackelberg (GRPO) alongside eight static-inference and training-based baselines.Training baselines include Sparta Alignment, Multiagent Fine-tuning, Trained Router, and AggLM; inference baselines include Majority Vote, MoA, Multiagent Debate, and Heterogeneous Swarms.Source: §3 · p. 6

Hyperparameters

  • Training uses eight iterations of 64 duels, with EXP3 exploration rate γ = 0.2 and difficulty and preference-quality reward weights λ1 = 0.3 and λ2 = 0.7.DPO and GRPO use learning rate 1 × 10^-6 and one epoch per iteration; experiments run on NVIDIA H100 GPUs.Source: §3 · p. 6

Evaluation

  • Table 1 reports performance by pool and dataset, with Avg as the macro-average over available datasets and AlpacaEval min-max normalized to [0, 1].Bold and underline mark the best and second-best result per column within each pool.Source: Table 1 · p. 7

Datasets

  • The 12 benchmarks span scientific discovery, reasoning, code, instruction following, and knowledge and truthfulness.The datasets include BixBench, LabBench, SMDD, AssayBench, GPQA-Diamond, MATH, HumanEval, MBPP, AlpacaEval, IFEval, TruthfulQA, and CulturalBench-Hard.Source: §3 · p. 7

4 RESULTS

Stackelberg Alignment achieves the highest macro-average across all three model pools, with GRPO outperforming the strongest training baseline by up to 7.4%. Results also show gains across open-ended tasks and broad reasoning, code, and knowledge domains, with differences between GRPO and DPO and across pool heterogeneity.

Overall performance

  • 7.4%, 2.1%, and 5.2% are Stackelberg (GRPO) macro-average improvements over Sparta Alignment in Pools 1–3, respectively.GRPO has Avg scores of 0.563, 0.544, and 0.609 in Pools 1–3; the best static inference baseline gap exceeds 20% in Pools 1 and 3 and 12% in Pool 2.Source: §4 · p. 7

Consistent gains on open-ended tasks

  • Stackelberg (GRPO) and (DPO) rank first and second on SMDD across all three pools; Stackelberg also leads AlpacaEval reward-model scores in Pools 1 and 3 at 8.081 and 7.138.Stackelberg (DPO) ranks first on IFEval in Pool 1, and Stackelberg (GRPO) ranks first in Pool 3.Source: §4 · pp. 7–8

Reputation trajectories

  • Pool 2 reputation scores diverge and stabilize by iterations 4–5, with a different leading model on TruthfulQA than on HumanEval.This pattern indicates task-specific competitive structure in the model pool.Source: Figure 3 · p. 8

Gains transfer broadly

  • Stackelberg gains extend across reasoning, code, and knowledge tasks, including GRPO first-place GPQA-Diamond results in Pools 2 and 3 at 0.423 and 0.424.GRPO is first on MATH in Pool 1 at 0.877, DPO leads in Pool 2 at 0.889, and GRPO ties for first on HumanEval across all pools; LabBench and AssayBench are exceptions.Source: §4 · p. 8

GRPO and DPO show complementary strengths

  • GRPO beats DPO on Avg in all pools and on most scientific and reasoning columns, while DPO is more competitive on several instruction-following and cultural-knowledge tasks.The exception to Stackelberg’s consistent advantage is Pool 3, where DPO scores 0.576 versus Sparta’s 0.579.Source: §4 · p. 8

Margins vary with pool heterogeneity

  • The adaptive-leader margin over Sparta ranges from 2.1% to 7.4%, with the narrowest margin in Pool 2 and the largest GRPO margin in Pool 1.Pool 3 has the largest gap over static inference, at 25.1% above MoA.Source: §4 · p. 8

5 ANALYSIS

The analysis links model-pool heterogeneity to task-specific competitive structure and adaptive opponent selection. It also finds that the leader’s instruction weights sharpen into a curriculum during training.

Pool heterogeneity and competitive stratification

  • No Pool 2 model dominates every task: M4 leads on 5/10 tasks but loses to M0 on reasoning and coding and to M1 on TruthfulQA.The per-model evaluation scores are shown at each model’s best iteration, with rows normalized and green/red marking best/worst per task.Source: §5 · p. 8, Figure 2 · p. 8
  • Reputation scores diverge from equal starting values, stabilize by iterations 4–5, and identify different leading models across TruthfulQA and HumanEval.This supports task-specific competitive structure being resolved early and maintained during training.Source: §5 · p. 9

Adaptive opponent selection

  • By iterations 4–5, the leader gives the highest-evaluation model less than 0.5× uniform opponent weight, while its most-selected opponent differs by task.M7 receives weight 0.176 on math at iteration 5 but 0.062 on TruthfulQA at iteration 7.Source: §5 · p. 9, Figure 4 · p. 9

Emergent difficulty curriculum over instructions

  • By iteration 5, instruction weights sharpen into an implicit difficulty curriculum without explicit difficulty labels.On MATH in Pool 1, the top five instructions rise to about 1.6× uniform by iteration 7, while the bottom five fall to about 0.75× uniform.Source: §5 · p. 9, Figure 5 · p. 9

6 CONCLUSION

Stackelberg Alignment makes instruction selection an adaptive curriculum and reports the highest macro-average across three heterogeneous pools and 12 benchmarks. Its results are situated among related collaboration, curriculum, and game-theoretic approaches, with statistical significance assessed per task.

6 CONCLUSION

  • Across three pools and 12 benchmarks, Stackelberg (GRPO) achieves the highest macro-average in every pool, improving over training-based baselines by up to 7.4% and over the best static inference baseline by 12–25%.Ablations confirm contributions from the adaptive curriculum, reputation-weighted judgment, and reputation-based matching.Source: §6 · pp. 9–10

A RELATED WORK

  • Stackelberg Alignment extends collaborative training, self-play, curriculum learning, and game-theoretic alignment by making instruction selection an adaptive leader decision in a competitive multi-LLM pool.Related work distinguishes its disagreement-driven curriculum from predefined categories, gradient heuristics, and symmetric simultaneous games.Source: A RELATED WORK · p. 14

B.1 STATISTICAL SIGNIFICANCE

  • Stackelberg (GRPO) earns at least one ∗∗ mark in every pool and at least one ∗ mark on 7–8 of the 10 reported tasks; SMDD and AssayBench are excluded from the confidence-interval comparison.The table reports per-task 95% confidence intervals; ∗∗ means the interval’s lower bound exceeds every baseline, while small test sets make the omitted tasks’ intervals uninformative.Source: §B.1 · p. 15, Table 2 · p. 15

B.2 CROSS-TASK TRANSFER

Competitive pool training on mathematical reasoning transfers to coding, while opponent-selection strategy, pool size, and maintained response diversity shape the reported MATH results. The appendix compares these findings across transfer tests and training ablations.

B.2 CROSS-TASK TRANSFER

  • +18 pp on HumanEval and +33 pp on MBPP follow MATH training, while same-task coding training yields no improvement over the best untrained model.HumanEval training also transfers to MBPP (+32 pp); the reported comparisons use the best untrained pool model as the per-task baseline.Source: §B.2 · pp. 15–16, Table 3 · p. 16

B.3 ABLATION: OPPONENT SELECTION STRATEGY

  • 0.878 for schedule increasing exceeds 0.865 for schedule decreasing on MATH, while highest diff scores 0.861 and performs worst among the strategies.All four strategies beat the 0.857 single-model baseline; ablation variants ran for two iterations versus eight for the default, so their final scores are not directly comparable.Source: §B.3 · p. 16, Figure 7 · p. 17

B.4 ABLATION: POOL SIZE

  • 0.880 for Pool 6 slightly exceeds Pool 4 (0.875) and Pool 8 (0.865), but Pool 4 and Pool 6 confidence intervals overlap.All three pool sizes beat the 0.857 single-model baseline, and the larger pool does not monotonically improve performance.Source: §B.4 · pp. 16–17, Figure 8 · p. 17

B.5 RESPONSE DIVERSITY DURING TRAINING

  • Across eight MATH training iterations, lexical diversity has no downward trend and response length stays within 100–135 words, while pre-trained-model stylistic differences persist.The analysis interprets maintained behavioral heterogeneity as a prerequisite for meaningful adaptive selection.Source: Figure 9 · p. 17, §B.5 · p. 17

B.6 ABLATION: LLM LEADER VS. PROBABILISTIC LEADER

The leader-strategy ablation compares EXP3 with a Gemini instruction selector on four Pool 1 tasks. EXP3 performs better on several reported tasks and avoids per-round LLM-inference latency and cost.

B.6 ABLATION: LLM LEADER VS. PROBABILISTIC LEADER

  • 8.08 vs. 1.65 on AlpacaEval and +10.1% absolute on GPQA favor the probabilistic leader over Gemini; MBPP gains are +6.2% for DPO and +7.2% for GRPO.The probabilistic leader essentially matches Gemini on TruthfulQA; EXP3 updates instruction sampling from follower win-rates.Source: §B.6 · p. 17, Table 4 · p. 18
  • EXP3 avoids the latency and cost of running LLM inference at each training round, while GRPO edges out DPO on MBPP and TruthfulQA and DPO leads on GPQA.The comparison reports complementary strengths between the two probabilistic training variants.Source: §B.6 · p. 18

C IMPLEMENTATION DETAILS

The implementation specifies the EXP3 instruction-sampling reward, reputation and opponent matching, training configurations, generation and evaluation settings, and model-pool configurations.

C IMPLEMENTATION DETAILS

  • The leader runs for eight iterations over 64 instructions, mixing exploration and exploitation at γ=0.2 and using reward 0.3 rdiff + 0.7 rpref.The ideal preference gap is annealed from 0.6 to 0.15; the difficulty reward uses threshold 3.0 and the preference reward uses bandwidth 0.15.Source: C IMPLEMENTATION DETAILS · p. 18
  • Reputation updates use a decaying K factor and the sample standard deviation of recent reputation deltas; opponent matching decreases the target reputation gap over training.Matching uses uniformly random opponents for 20% of duels and draws each combatant from three candidate opponents.Source: C IMPLEMENTATION DETAILS · p. 18
  • DPO and GRPO both use LoRA and a learning rate of 10^-6, while their batch, rollout, and objective settings differ.DPO uses β=0.1 and effective batch size 16; GRPO uses four rollout generations per prompt, β=0.0, and clip ratio 0.2.Source: C IMPLEMENTATION DETAILS · p. 18
  • Responses use temperature 0.7 and top-p 0.9, while peer judges score greedily; multiple-choice benchmarks use exact match and AlpacaEval uses a reward model.The supplied implementation passage also points to domain-specific metrics for open-ended scientific tasks.Source: C IMPLEMENTATION DETAILS · pp. 18–19

D MODEL POOL CONFIGURATIONS

E DATASET STATISTICS

The appendix specifies how benchmark instructions are split, how baselines are configured, and how peer judges score and aggregate responses.

E DATASET STATISTICS

  • Each benchmark’s development set supplies candidate training instructions, with 80% used for the pool and 20% held out for validation; test data is reserved for final evaluation.Source: E DATASET STATISTICS · p. 19, Table 6 · p. 19

F BASELINE CONFIGURATIONS

  • Training-based baselines receive equal gradient updates, and Sparta shares the duel loop, opponent schedule, and reputation system but samples instructions uniformly.Other baselines use debate-style distillation, an RL aggregator, or supervised routing; inference baselines use response synthesis, iterative refinement, critique, or voting.Source: F BASELINE CONFIGURATIONS · pp. 19–20

G JUDGE PROMPT

  • Peer judges score the given question and response on a 1-to-10 scale, with JSON scores parsed and regex extraction as fallback; parse failures default to 5.0.Source: G JUDGE PROMPT · p. 20
  • Judge scores are aggregated as a reputation-weighted mean, using each judge’s current reputation score.Source: G JUDGE PROMPT · p. 20

H COMPUTATIONAL COMPLEXITY ANALYSIS

The analysis compares per-iteration model-call and training costs across Stackelberg Alignment and baselines. Stackelberg Alignment adds no model calls relative to Sparta, while some training baselines process the full dataset and inference-only methods shift cost to evaluation.

H COMPUTATIONAL COMPLEXITY ANALYSIS

  • Stackelberg Alignment and Sparta both require O(D · m) forward passes and m LoRA training steps, while EXP3 adds only an O(K) update per duel.The analysis states that Stackelberg Alignment introduces zero additional model calls relative to Sparta; gains come from reallocating the fixed duel budget, not extra compute.Source: H COMPUTATIONAL COMPLEXITY ANALYSIS · p. 20
  • Multiagent FT, AggLM, and Trained Router require O(N · m) forward passes by processing all N instructions with all m models before each update.Source: H COMPUTATIONAL COMPLEXITY ANALYSIS · p. 20
  • Inference-only methods have no training steps, but Swarms and Debate incur repeated forward passes at inference time across refinement rounds.Source: H COMPUTATIONAL COMPLEXITY ANALYSIS · p. 20, Table 7 · p. 20
  • Stackelberg Alignment uses at most 576 model calls per iteration (D=64, m≤9), versus up to 8,604 for full-dataset sweeps (N=956, m=9).The comparison is for the MATH example in the supplied complexity discussion.Source: H COMPUTATIONAL COMPLEXITY ANALYSIS · pp. 20–21

I LIMITATIONS

The approach requires simultaneous execution of the model pool, depends on instruction-pool coverage of the capability frontier, and may need several iterations to distinguish model strengths on imbalanced pools.

I LIMITATIONS

  • Running all pool models simultaneously during combat and judgment increases memory and compute needs relative to single-model fine-tuning.This constraint is shared by training-time multi-LLM collaboration methods.Source: I LIMITATIONS · p. 21
  • An instruction pool that does not span the model pool’s capability frontier limits the learning signal EXP3 can identify.Source: I LIMITATIONS · p. 21
  • Equal reputation initialization may take several iterations to differentiate strengths on highly imbalanced pools; warm-starting from measured benchmark performance is a proposed extension.Source: I LIMITATIONS · p. 21

Related papers

Loading 2609.39076v1…