PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning

Shangbin Feng, Yulia Tsvetkov, Jan Kautz, David Acuna

arXiv:2610.17456v1cs.CLcs.AIcs.LG

TL;DR

Solo benchmarks do not measure how much a model contributes to a collaborative system. The paper formalizes and estimates that contribution, then trains policies inside collaborative systems; compositional strength is largely uncorrelated with individual strength, while C-RL performs best in most tested settings.

  • Problem

    Models are commonly evaluated alone even as they are deployed in collaborative systems, leaving their contribution to those systems unmeasured.

    Source: §1 · p. 1

  • Method

    The paper formalizes compositional strength as a Shapley value, estimates it with a leave-one-out marginal, and trains policies with collaborative-system outcomes as rewards.

    Source: §1 · p. 2

  • Results

    C-RL produced the strongest collaborator in 21 of 24 settings, while individual and compositional strength showed no significant relationship in 12 of 15 settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Standalone benchmark scores are mostly uninformative about how much a model contributes to collaborative systems.

    Source: §2.3 · p. 4

    The authors identify a dual-use risk: C-RL could help a malicious model inject harmful intent into a multi-model system while remaining individually inconspicuous.

    Source: ETHICS STATEMENT · pp. 13–14

Abstract

from arXiv · show

Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained. For a model inside a system, what matters most is not how well it performs on its own, but how much the system gains from its presence. We call this a model's compositional strength and formalize it using the Shapley value of a cooperative game over coalitions of a model pool, and use a model's leave-one-out contribution to the full coalition as a tractable proxy, requiring n+1 rather than 2^n system evaluations. Evaluating individual vs. compositional strengths across five model pools, twelve collaboration methods, and twelve datasets, we find them largely uncorrelated: in 12 of 15 (pool, method-family) settings, individual strength does not translate into compositional advantages, and the coupling is weakest among general-purpose models that practitioners are most likely to deploy. Standalone benchmarks are therefore often poor proxies for models' value in collaborative systems. We then propose compositional reinforcement learning (C-RL), which trains a policy inside a live multi-model system and rewards the outcome produced by the system as a whole, rather than the policy's correctness in isolation. The leave-one-out baseline in Shapley values cancels under a group policy gradient, so this needs no additional rollout cost beyond RL on the collaborative system's outcome and never requires running a coalition without the policy. Across three collaboration protocols and eight tasks spanning reasoning, coding, and scientific discovery, C-RL produces the strongest collaborator in 21 of 24 settings, lifting collaborative performance by +4.0 to +5.4 pts on average (~3x the gain of baselines), up to +13.1 pts.

1 INTRODUCTION

The paper argues that solo evaluation misses how much a model contributes inside a collaborative system, and introduces ways to evaluate and train for that contribution. It finds individual and compositional strength largely uncorrelated and reports that C-RL improves collaborative performance across its tested settings.

1 INTRODUCTION

  • Compositional strength is formalized as a model’s Shapley value in a cooperative game and estimated by its leave-one-out marginal using n + 1 runs.The estimate tracks the exact value closely.Source: §1 · pp. 1–2
  • In 12 of 15 pool–method-family settings, individual and compositional strength were not significantly related; significant correlations were weak (|ρ| ≤0.29).No setting among capable general-purpose model pools showed a significant relationship.Source: §1 · p. 2
  • C-RL trains a policy inside a live collaborative system and rewards it with the system outcome rather than its solo correctness, without extra rollout cost.The pool of frozen collaborators and collaboration protocol form the policy’s environment.Source: §1 · p. 2
  • C-RL produced the strongest collaborator in 21 of 24 settings and lifted collaborative performance by +4.0 to +5.4 points on average under each protocol.It improved performance by as much as +13.1 points on a single task.Source: §1 · p. 2

2 EVALUATING LANGUAGE MODELS BY COMPOSITIONAL STRENGTH

The section defines a model’s value in a collaborative system through its compositional strength and evaluates how that value relates to solo performance across model pools, methods, and tasks. It finds solo scores mostly uninformative about collaborative contributions, while collaboration outcomes vary by method and task domain.

2.1 METHOD

  • Compositional strength is defined by a model’s Shapley value in a cooperative game, and estimated by the performance drop when that model is removed from the full coalition.The leave-one-out estimate requires n + 1 coalition evaluations rather than 2^n for exact Shapley; it tracks exact Shapley and Banzhaf estimates empirically.Source: §2.1 · p. 3

2.2 EXPERIMENT SETTINGS

  • The evaluation spans five model pools, 12 datasets across five task categories, and 12 collaboration methods in routing, agentic, and training families.For each valid pool–dataset–method combination, the protocol measures solo scores and leave-one-out compositional strengths.Source: §2.2 · pp. 3–4

2.3 RESULTS

  • In 12 of 15 pool–method-family settings, individual and compositional strength are not significantly related; the remaining correlations are weak (|ρ| ≤0.29).Across the three significant cells, the correlations are weak despite large samples; all six panels for >100B Open and Proprietary API models are nonsignificant.Source: §2.3 · p. 4, Figure 1 · p. 5

2.4 ANALYSIS

  • Sparta Alignment and Multiagent Debate most often rank first, while Coding is consistently collaboration-helpful and QA is weakest for four of five pools.Figure 2 ranks methods by full-coalition performance; Figure 3 shows Coding is strongest for three pools and never weakest, while QA is weakest for four.Source: §2.4 · p. 6, Figure 2 · p. 6, Figure 3 · p. 6
  • Collaboration methods differ in their frequency of top finishes, and the strongest compositional-strength gains vary across pool–domain pairs.Diverse Academic Models have the largest single pool–domain Reasoning peak; that peak is not shared by the other pools.Source: §2.4 · p. 6, Figure 3 · p. 6

3 TRAINING FOR COMPOSITIONAL STRENGTH

Compositional RL (C-RL) trains a model within a collaborative system to improve the system’s outcome, using group-relative policy gradients to avoid extra coalition rollouts. Across the reported results, it improves collaborative performance and transfers to held-out datasets and collaboration protocols.

3.1 METHOD

  • C-RL optimizes system performance by training a policy inside a live collaboration environment and rewarding the system’s outcome rather than the policy’s solo correctness.Group-relative policy gradients cancel the leave-one-out baseline, so training needs no additional rollout of the collaborators without the policy.Source: §3.1 · p. 7
  • Each policy generation receives the final system reward, and training applies policy-gradient updates only to the policy’s own sampled generations.The number of policy calls can vary by rollout; the paper uses uniform credit assignment and notes that differentiated credit assignment remains future work.Source: §3.1 · p. 8

3.2 EXPERIMENT SETTINGS

  • Experiments train Qwen 3.6 27B with three frozen, larger collaborators across LLM-Blender, Multiagent Debate, and SLM-MUX, using eight tasks and individual and collaborative evaluation modes.The five baselines include an untrained policy, two solo-reward RL methods, prompt nudge, and collaborative RAFT.Source: §3.2 · pp. 8–9

3.3 RESULTS

  • C-RL achieves the best score in 21 of 24 cells, improves on Base in all 24, and adds +4.8 pts under LLM-Blender, +4.0 pts under Multiagent Debate, and +5.4 pts under SLM-MUX.It exceeds the best baseline in each protocol block by +3.2 pts, +1.1 pts, and +3.6 pts, respectively.Source: Table 1 · p. 9, §3.3 · pp. 9–10
  • Across 24 cells, GRPO-indiv and DAPO-indiv improve on Base in 19 and 21, respectively, but individual-reward RL can improve solo correctness while reducing collaborative performance.On MMLU-ProX under SLM-MUX, DAPO-indiv gains +6.2 pts alone yet costs the system 3.6 pts; GRPO-indiv gains +2.6 pts alone while costing the system 2.6 pts.Source: §3.3 · p. 10
  • Relative gains average +27.0% on execution-verified and reference-ranking tasks, versus +7.4% on answer-matching tasks.The paper suggests graded rewards for open-ended artifacts and low starting scores as possible reasons these tasks show larger gains.Source: §3.3 · p. 10

3.4 ANALYSIS

  • On held-out APEX-Shortlist and BixBench, C-RL is the only method improving on Base on both datasets, by +7.5 pts and +4.8 pts, respectively.On unseen protocols, C-RL is best in three of four cells and improves on Base everywhere: +9.6 pts on KernelBench under Mixture-of-Agents, +6.3 pts on KernelBench under Heterogeneous Swarms, and +2.1 pts on AssayBench under Heterogeneous Swarms.Source: Figure 4 · p. 10, §3.4 · p. 11, Figure 5 · p. 11
  • With GPT-5.5, Gemini 3.5 Flash, and Claude Opus 4.8 as collaborators, C-RL reaches 0.254 on KernelBench (+8.0 pts over Base) and 0.055 on AssayBench (+1.1 pts).On AssayBench, every baseline performs below Base, whereas C-RL improves on it.Source: Table 2 · p. 11, §3.4 · pp. 11–12

4 RELATED WORK

Prior work studies collaboration strategies, system-level evaluation, contribution attribution, and reinforcement learning in multi-model systems. This paper measures model contributions across pools, tasks, and collaboration methods, then trains a policy to improve its collaborative contribution.

4 RELATED WORK

  • The paper studies pools of genuinely distinct models and training-time collaboration, extending work that often reuses one model with different prompts or optimizes the collaboration strategy itself.It distinguishes this focus from inference-time composition of frozen models and from prior work optimizing strategies rather than models as collaborators.Source: §4 · p. 12
  • Prior system-level evaluation examines interactive settings, agent scaffolds, and harnesses, while this paper measures whether individual benchmark scores predict contribution across pools, tasks, and methods.Related work also studies when collaboration is worthwhile and model complementarity, often for one ensemble or task at a time.Source: §4 · p. 12
  • Contribution attribution has been applied to data, prompts, workflow modules, and agents; this paper treats model marginal contribution as a property measured across systems and optimized directly.Prior multi-model RL includes co-training participants, self-play, distillation, and training routers or aggregators; here collaborators are neither trained nor policy copies, and reward is the system outcome itself.Source: §4 · pp. 12–13

5 CONCLUSION

The paper defines compositional strength as a model’s contribution to a multi-model system and finds it largely uncorrelated with individual strength. It proposes C-RL to optimize that contribution, while noting reproduction and safety boundaries.

5 CONCLUSION

  • C-RL produces the strongest collaborator in 21 of 24 settings, never leaves a system worse off than the untrained policy, and transfers to held-out datasets, unseen protocols, and stronger proprietary pools.The paper frames compositional strength as a step toward system-level intelligence, evaluating and training models for what they make possible together.Source: §5 · p. 13
  • Exact reproduction may be limited because proprietary collaborators are served through APIs whose updates the authors do not control, so results involving them may shift.The authors say they will release code and configurations for reported runs upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 13
  • The authors identify a dual-use risk: C-RL could help a malicious model inject harmful intent into a multi-model system while remaining individually inconspicuous.They name safety against such malicious contributions as future work.Source: ETHICS STATEMENT · pp. 13–14

A ADDITIONAL RESULTS AND ANALYSIS

The appendix compares methods’ solo scores with their collaborative standing, showing that the two rankings can diverge substantially.

A.1 INDIVIDUAL PERFORMANCE

  • Table 3 scores methods with the policy answering alone; the five baselines are reported once because solo scores do not depend on the later evaluation protocol.Only C-RL is trained separately per protocol.Source: §A.1 · p. 20
  • Collaborative RAFT is among the strongest individually (average 0.524, essentially tied with GRPO-indiv at 0.526) but is the weakest collaborator overall, improving on Base in only 14 of 24 cells.It finishes last or second to last under all three protocols.Source: §A.1 · p. 20
  • Prompt nudge is near the bottom individually (0.498, against 0.484 for Base) yet is the best baseline under LLM-Blender and second best under SLM-MUX.C-RL leads in individual mode in 13 of 24 cells, compared with 21 of 24 in collaborative mode.Source: §A.1 · p. 20

A.2 CONFIDENCE INTERVALS

The appendix reports confidence intervals for C-RL scores and explains their scope as a measure of finite-test-set precision, not as hypothesis tests.

A.2 CONFIDENCE INTERVALS

  • Table 5 compares base-to-C-RL outcomes using rescue and drag fractions; drag falls under all three protocols.Cells marked -- lack a usable log.Source: Table 5 · p. 21
  • Table 4 reports Base beside C-RL for each protocol, with 95% confidence intervals on C-RL scores; the lower bound exceeds Base in every aggregate column.The intervals use each task’s true test-set size, from 99 items on GPQA-Diamond to 613 on CulturalBench-Hard; individual-dataset exceptions cluster on the two smallest, noisiest tasks.Source: §A.2 · p. 21, Table 4 · p. 21
  • The intervals describe precision from finite test sets rather than serving as hypothesis tests; the authors base their claim on C-RL winning 21 of 24 cells and never falling below Base.No single interval speaks to the pattern across cells.Source: §A.2 · p. 21

A.3 RESPONSE-LEVEL ANALYSIS

The response-level analysis shows that C-RL helps correct policy answers survive collaboration by reducing cases where the system is dragged into error. Illustrative examples show that the policy’s framing can shape the answer adopted by the system.

A.3 RESPONSE-LEVEL ANALYSIS

  • Drag falls under all three protocols, while rescue remains 11–18% of items after training, showing that collaborators still carry the policy on many questions.Drag declines from 4.7% to 4.0% under LLM-Blender, 5.1% to 3.6% under Multiagent Debate, and 4.9% to 3.7% under SLM-MUX.Source: §A.3 · pp. 21–22
  • In two ChemBench examples, C-RL changes the policy’s candidate so the fuser’s answer changes from wrong to correct.One example shows the policy’s error overturning three correct collaborators; after C-RL, the policy agrees with them and the correct answer stands.Source: Table 6 · p. 22, §A.3 · p. 22
  • The analysis attributes these changes to how the policy frames its contribution for collaborators, and whether the system’s mechanism adopts that framing.The same collaborators and protocol are used in both systems; the candidate supplied by the policy differs.Source: §A.3 · pp. 22–23
  • Two LLM-Blender examples show C-RL systems correcting interpretive errors on TheoremQA and GPQA-Diamond, rather than arithmetic errors.The examples turn on interpreting symmetry in BANANA and recognizing the effect of aqueous workup in ozonolysis.Source: §A.3 · p. 22, Table 7 · p. 23

A.4 COLLABORATION EFFECTIVENESS BY POOL AND DOMAIN

  • Table 8 reports leave-one-out and full-coalition scores and gains for the best dataset–method pair in each of 25 model-pool and task-category cells.The table provides the values underlying the collaboration-effectiveness claim.Source: §A.4 · p. 23

A.5 PER-DATASET GAIN CONSISTENCY

Per-dataset results show that collaboration gains vary within domains: Coding gains are consistently positive, while Science is bimodal, with SciRIFF standing apart from AssayBench and KernelBench.

A.5 PER-DATASET GAIN CONSISTENCY

  • Coding datasets HumanEval and MBPP each show consistently positive gains, not merely a positive category average.The per-dataset pattern supports the conclusion that Coding’s gains are not an artifact of aggregating its two datasets.Source: §A.5 · p. 23
  • The mean gain across Table 8’s 25 model-pool and task-category values is +11.5, with a 95% CI of [+8.7, +14.7].Each value uses the best-performing collaboration method for its cell.Source: Table 8 · p. 24
  • SciRIFF gains reach 0.142 and 0.146 for the Specialized Expert and >100B Open pools, while AssayBench stays at or below 0.018 and KernelBench at or below 0.030 across pools.The domain-level Science result is therefore driven almost entirely by SciRIFF, rather than indicating uniform gains or losses across scientific tasks.Source: §A.5 · pp. 23–24, Figure 6 · p. 24

A.6 THE SHAPLEY VALUE: DERIVATION AND AXIOMS

The paper defines compositional strength using the Shapley value, averaging each model’s marginal contribution across all possible orders in which it can join a coalition.

A.6 THE SHAPLEY VALUE: DERIVATION AND AXIOMS

  • The Shapley value averages a model’s marginal contribution across all n! orders in which the models can join the coalition.For each order, the model’s contribution is the change in coalition value when it joins.Source: §A.6 · p. 24
  • The contribution for a preceding set S is v(S ∪ {i}) − v(S), weighted by the fraction of orders in which exactly S precedes model i.The weight |S|!(n−|S|−1)!/n! counts the relevant orderings as a share of all n! orders.Source: §A.6 · p. 24
  • The Shapley value uniquely satisfies efficiency, symmetry, the null-player property, and additivity, making it the paper’s definition of compositional strength.Under this definition, strength is expected usefulness averaged over every coalition a model might join.Source: §A.6 · p. 24

A.7 COMPARING CONTRIBUTION ESTIMATORS

On the Specialized Expert pool, leave-one-out strength is positively associated with both Banzhaf and exact Shapley values, though it agrees more in magnitude than in model ranking and reads systematically lower.

A.7 COMPARING CONTRIBUTION ESTIMATORS

  • The comparison uses 64 model–dataset–method observations from four datasets and four methods in the four-model Specialized Expert pool.All three estimators are computed from the same cached coalition values, with 16 coalitions per cell.Source: §A.7 · p. 25
  • Leave-one-out strength correlates with Banzhaf at Pearson r = 0.61 (p = 6.8 × 10⁻⁸) and with exact Shapley at r = 0.66 (p = 2.8 × 10⁻⁹).Figure 7 plots each model–dataset–method cell against the two estimators.Source: Figure 7 · p. 25, §A.7 · p. 25
  • Agreement is weaker in rank, with Spearman correlations of 0.31 against Banzhaf and 0.42 against exact Shapley, and leave-one-out values are usually lower.The paper attributes the downward difference to remaining models compensating when one model is removed, reducing credit for duplicated competence.Source: §A.7 · p. 25

B.1 EVALUATION SETUP DETAILS

The evaluation varies model-pool scale and training heterogeneity, and measures coalition performance on twelve datasets using rule-based or execution-based scoring. It compares twelve collaboration methods across five pools.

B.1 EVALUATION SETUP DETAILS

  • Coalition values use each dataset’s full test split, with rule-based or execution-based scoring and no model acting as judge.The twelve datasets and their categories and test-item counts are listed in Table 10.Source: §B.1 · p. 25, Table 10 · p. 26
  • Five pools span participant scale and training heterogeneity, from four specialized sibling 7B models to eight independently post-trained research models.Each pool defines a separate cooperative game; compositional strengths are computed within a pool and not compared across pools.Source: §B.1 · p. 25, Table 9 · p. 26
  • Twelve collaboration methods run through MoCo, spanning routing, debate and feedback, swarms, candidate ranking and fusion, and collaborative training.Configurations include three-round debate, one feedback-and-refine cycle, five-member swarms, and methods with fixed Qwen2.5-7B selectors, routers, rankers, or fusers.Source: §B.1 · p. 26

B.2 C-RL IMPLEMENTATION DETAILS

C-RL uses collaborative-protocol rollouts to train a fully fine-tuned policy, while baselines either optimize solo correctness or use collaborative transcripts. The implementation specifies dataset splits, execution-based scoring, policy-token attribution, and checkpoint selection.

B.2 C-RL IMPLEMENTATION DETAILS

  • Eight datasets train and validate C-RL using an 8:2 split of MoCo development data, while APEX-Shortlist and BixBench are held out for generalization.Table 11 lists the ten datasets; the two held-out datasets are sized by their test splits.Source: §B.2 · p. 26, Table 11 · p. 27
  • KernelBench rewards correct kernels and measured speedups, SMDD-Bench checks molecular constraints or pharmacophore performance, and AssayBench scores gene rankings with Adjusted nDCG@100.KernelBench uses an H100 GPU; SMDD-Bench scoring depends on problem type, and unmatched or hallucinated genes receive zero relevance in AssayBench.Source: §B.2 · pp. 26–27
  • Each optimizer step runs 64 protocol executions; pooled-generation advantages update a fully fine-tuned policy with a clipped policy-gradient loss and no KL anchor.Runs use up to 100 rollout steps and learning rate 10^-6; a complete run takes roughly 30 hours on three nodes of eight H100 GPUs.Source: §B.2 · p. 27
  • A shim records the policy’s prompts, sampled tokens, and log-probabilities, so loss applies to its own generations while collaborator text remains context.The rollout reward is broadcast to every recorded policy entry tagged to that rollout.Source: §B.2 · pp. 27–28
  • Training uses two collaboration rounds; LLM-Blender ranks listwise with a fixed gpt-oss-120b ranker, while SLM-MUX draws three samples per model.Every three steps, a checkpoint is scored on 16 development questions; early stopping follows 25 evaluations without improvement, and the best checkpoint is reported on the full test split.Source: §B.2 · p. 28
  • GRPO and DAPO optimize solo correctness, Collaborative RAFT fine-tunes on high-scoring collaborative transcripts, and prompt nudge changes only the system prompt.GRPO and DAPO use single-model rollouts; RAFT trains on the policy’s own generations from selected transcripts.Source: §B.2 · p. 28

Related papers

Loading 2610.17456v1…