PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat

Yuru Jiang, Wenxuan Ding, Shangbin Feng, Greg Durrett, Yulia Tsvetkov

arXiv:2506.04721v3cs.CL

TL;DR

Single-model self-alignment can suffer from biased self-judgment and limited response diversity. SPARTA ALIGNMENT has multiple LLMs compete, judge peers, and learn from the resulting preferences. It reports state-of-the-art results on 10 of 12 datasets and a 7% average improvement on reasoning and instruction-following tasks.

  • Problem

    Single-model self-alignment can reinforce self-judgment bias and produce homogeneous responses, limiting the diverse, distinguishable preferences available for learning.

    Source: §1 · p. 2

  • Method

    SPARTA ALIGNMENT pairs models to answer instructions, uses reputation-weighted peer judgments to form preference pairs, and trains the models on those preferences.

    Source: §1 · p. 2, §2 · pp. 3–4

  • Results

    SPARTA ALIGNMENT achieves state-of-the-art results on 10 of 12 datasets, with a 7% average advantage over second-best baselines on reasoning and instruction-following tasks.

    Source: §4 · p. 5

  • Takeaways & Limitations

    The approach collectively aligns multiple LLMs through competition, with the reported gains spanning several evaluation domains.

    Source: §1 · p. 2, §7 · p. 9

    The authors discuss limitations in Section A, but the cited passages do not specify their scope.

    Source: §2 · p. 18

Abstract

from arXiv · show

We propose SPARTA ALIGNMENT, an algorithm to collectively align multiple LLMs through competition and combat. To complement a single model's lack of diversity in generation and biases in evaluation, multiple LLMs form a "sparta tribe" to compete against each other in fulfilling instructions while serving as judges for the competition of others. For each iteration, one instruction and two models are selected for a duel, the other models evaluate the two responses, and their evaluation scores are aggregated through a adapted elo-ranking based reputation system, where winners/losers of combat gain/lose weight in evaluating others. The peer-evaluated combat results then become preference pairs where the winning response is preferred over the losing one, and all models learn from these preferences at the end of each iteration. SPARTA ALIGNMENT enables the self-evolution of multiple LLMs in an iterative and collective competition process. Extensive experiments demonstrate that SPARTA ALIGNMENT outperforms initial models and 4 self-alignment baselines across 10 out of 12 tasks and datasets with 7.0% average improvement. Further analysis reveals that SPARTA ALIGNMENT generalizes more effectively to unseen tasks and leverages the expertise diversity of participating models to produce more logical, direct and informative outputs.

1 Introduction

SPARTA ALIGNMENT addresses single-model self-judgment bias and limited response diversity by having multiple LLMs compete, judge peers, and learn from resulting preferences. Its experiments report improvements across most evaluated tasks.

1 Introduction

  • Single-model self-alignment can reinforce self-judgment bias and produce homogeneous responses, limiting the diversity and distinguishability needed for preference learning.The cited work notes these concerns may be especially pronounced on tasks involving human values and cultural implications.Source: §1 · pp. 1–2
  • SPARTA ALIGNMENT pairs two models to answer a sampled instruction, has the remaining models judge them, and trains all models on the resulting preference pairs.Peer judgments are aggregated using judge reputations, which increase for winners and decrease for losers.Source: §1 · p. 2, §2 · p. 3
  • 10 of 12 datasets show state-of-the-art results, with a 7% average improvement over the second-best baselines on reasoning and instruction-following tasks.The introduction also reports highest generalization accuracy in two of three challenging mathematical benchmark categories.Source: §1 · p. 2

2 Methodology

  • Match-making selects the first model randomly, then chooses its opponent either randomly with probability α or from the top-k models closest in reputation.The strategy balances exploration of diverse models with competition among models of similar reputation.Source: §2 · p. 3
  • Reputation-weighted aggregation combines peer scores, while reputation updates reflect score gaps, recent reputation deviation, and relative opponent strength.Stable reputations receive smaller updates and unstable ones larger updates; the update scaling is capped by tanh(σ_i).Source: §2 · p. 4

3 Experiment Settings

Experiments use a diverse pool of independently fine-tuned Qwen models and compare SPARTA ALIGNMENT with initial-model and self-alignment baselines. Evaluation spans multiple task domains, with a reported comparable or lower training inference cost than other self-alignment methods.

Models and Implementation

  • The model pool contains 10 Qwen2.5-7B-Instruct models independently fine-tuned on different Tulu-v2 data domains.This initialization is intended to provide diverse skills and expertise for collective alignment.Source: §3 · p. 4

Baselines

  • The study compares SPARTA ALIGNMENT with the best-performing initial model and four self-alignment algorithms, including SELF-REWARD, META-REWARD, SPIN, and SPPO.SPIN assumes access to gold training labels, which SPARTA ALIGNMENT does not require.Source: §3 · p. 5

Evaluation

  • SPARTA ALIGNMENT maintains comparable or lower training inference cost than other self-alignment methods by processing each instruction only twice per iteration.The comparison holds despite using more models.Source: §3 · p. 5

Dataset

  • Evaluation covers 8 tasks and 12 datasets across domain-specific question answering, reasoning, instruction following, and safety.Benchmarks include MedQA and Normad, GSM8K, KCross, COM2 and MATH, plus Alpaca and TruthfulQA.Source: §3 · p. 5

4 Results

SPARTA achieves state-of-the-art results on 10 of 12 datasets, with a 7% average advantage over second-best baselines on reasoning and instruction-following tasks. Results span domain-specific QA, reasoning, instruction following, and truthfulness, while the authors identify limited dataset scale as a constraint on KCross.

4 Results

  • SPARTA achieves state-of-the-art results on 10 of 12 datasets and beats second-best baselines by 7% on average across reasoning and instruction-following tasks.Table 1 reports results across 12 datasets and marks significant results with an asterisk.Source: §4 · p. 5, Table 1 · p. 6
  • On Normad-Value, SPARTA improves 2% over SELF-REWARD, and on MedQA it leads in clinical informativeness and logical consistency.The authors attribute these domain-specific results to greater response diversity than self-alignment methods.Source: §4 · p. 6
  • On GSM8K, SPARTA improves 4.5% over initialization and 3.6% over SPPO; on MATH, gains average 4.0% over initialization and 1.9% over prior best; on COM2, gains are 20.5% over initialization and 9.6% over SELF-REWARD.The authors report strongest results across knowledge, mathematical, and commonsense reasoning tasks.Source: §4 · p. 6
  • The authors attribute SPARTA’s limited gains on KCross to the dataset’s small scale, which may constrain robust generalization on structural-knowledge reasoning.They suggest performance may improve with more diverse and extensive unsupervised data.Source: §4 · p. 6
  • On Alpaca, SPARTA outperforms the best initial model by 32.8% and SELF-REWARD by 28.1%; on TruthfulQA, it scores 0.424 versus 0.410 for the best initial model and 0.421 for SPPO.The paper attributes weaker single-model-feedback baselines to limited capture of nuanced human preferences and principles in instruction-following and safety tasks.Source: §4 · p. 6
  • An initially underperforming model can become the strongest in the pool through iterative collaborative alignment.The authors describe weaker models as improving through diverse interactions with stronger peers.Source: §4 · p. 6

5 Analysis

The analysis finds that SPARTA ALIGNMENT generalizes across mathematical difficulty levels and benefits from larger, more diverse model pools. It also examines output diversity, reputation, and component contributions.

Generalization

  • Trained on Medium and Hard MATH subsets, SPARTA ALIGNMENT outperforms all baselines on held-out difficulty subsets.Figure 2 evaluates training on Easy, Medium, or Hard and measures OOD performance on the other two subsets.Source: §5 · p. 7, Figure 2 · p. 7

Scaling the Model Pool

  • Compared with a 3-model setting, a 10-model pool raises performance by 19.1% on Alpaca, 33.1% on COM2, and 6.1% on MedQA.Increasing the number of models consistently improves results across all four benchmarks.Source: §5 · p. 7, Figure 3 · p. 7

Generation Diversity

  • On instruction-following diversity metrics, SPARTA outperforms SELF-REWARD and META-REWARD across all dimensions using 10 models versus one.Diversity decreases over iterations while the best performance increases; preference pairs also show greater diversity and selected responses are more complete, clinically informative, direct, and logically structured.Source: §5 · pp. 7–8, Table 2 · p. 7

Model Diversity Matters

  • The fully diverse 10 × 1 pool outperforms the fully repetitive 1 × 10 pool by 18.5% averaged across Alpaca, COM2, and MedQA.Performance consistently improves as pool diversity increases across the tested configurations.Source: §5 · p. 8, Figure 4 · p. 8

Correlation between Reputation and Performance

  • Model reputation scores positively correlate with task performance, with an average correlation coefficient of 0.21 across tasks.The analysis also reports hierarchical stratification among the 10 models.Source: §5 · p. 8, Figure 5 · p. 8

Ablation Study

  • Removing any single component lowers performance across four datasets; removing matchmaking randomness hurts Alpaca and COM2 most, while removing top-k constraints can compromise stability.The reported findings associate stochastic opponent selection with behavioral diversity and generalization in instruction-following, and constrained pairing with balancing exploration and reliable supervision.Source: §5 · pp. 8–9, Table 3 · p. 8

6 Related Work

Prior alignment methods use human or AI feedback, while self-alignment can be limited by self-bias and decreasing response diversity. Related work also motivates multi-LLM collaboration as a way to draw on model specialization and cross-evaluation.

Reinforcement Learning from AI Feedback (RLAIF)

  • RLAIF uses AI-generated feedback to train reward models, while self-alignment methods include self-judging, self-rewarding, and iterative response refinement.Constitutional AI uses principles to critique and revise responses, while Self-Rewarding and Meta-Rewarding use model-generated rankings or rewards.Source: §6 · p. 9
  • Self-feedback can suffer from self-bias, and declining response diversity can blur positive-negative distinctions and create preference inconsistency.The passage describes these as bottlenecks to further self-improvement and alignment effectiveness.Source: §6 · p. 9

Multi-LLM Collaboration

  • Multi-LLM collaboration is motivated by single-model limits in representing linguistic diversity, task specialization, and users’ diverse needs and values.Prior work explores models evaluating and reflecting on one another and multi-model systems such as modular systems and diverse reward models.Source: §6 · p. 9
  • SPARTA ALIGNMENT combines multi-LLM collaboration with competition and cross-judging, and is reported to outperform single-model methods across several task families.The cited task families include instruction-following, reasoning, and cultural adaptation.Source: §6 · p. 9

7 Conclusion

SPARTA ALIGNMENT is presented as a game-theoretic approach for collectively improving multiple LLMs through competition. The conclusion reports gains over the best baseline and findings on generalization, reputation, and pool composition.

7 Conclusion

  • SPARTA ALIGNMENT iteratively pairs diverse models to generate responses, has other pool members judge them using reputation-weighted aggregation, and rewards winners while penalizing losers.Experiments report up to 28.1% improvement over the best baseline and 7% average improvement across 10 tasks spanning several domains.Source: §7 · p. 9
  • The conclusion reports better generalization to unseen data settings, positive correlation between reputation and task performance, and benefits from larger and more diverse participant pools.The authors state that the abstract and introduction claims reflect the paper’s contributions and scope.Source: §7 · p. 9, §1 · p. 18

2. Limitations

The checklist responses state that the paper discusses limitations in Section A and provides assumptions, proofs, and experimental details in designated sections. The authors also report plans to open-source data and code.

2. Limitations

  • The authors state that limitations are discussed in Section A, but the supplied checklist passage does not specify their substance.This is a checklist response rather than a description of a particular limitation.Source: §2 · p. 18

3. Theory assumptions and proofs

  • The authors state that assumptions and proofs are provided in Section 2.The checklist marks the theoretical-result assumptions and proof question Yes.Source: §3 · pp. 18–19

4. Experimental result reproducibility

  • The authors report that Section 3 provides information needed to reproduce the main experimental results.The checklist marks the reproducibility question Yes.Source: §4 · p. 19

5. Open access to data and code

  • The authors state that they will make data and code openly accessible through an open-source platform and submit them via OpenReview.Source: §5 · p. 20

6. Experimental setting/details

  • The authors state that Section 3 provides training and test details needed to understand the results.The checklist identifies examples such as data splits, hyperparameters, and optimizer type.Source: §6 · p. 20

7. Experiment statistical significance

The checklist answers yes on statistical-significance reporting and compute-resource information, citing Sections 4 and 3, respectively.

7. Experiment statistical significance

  • The authors answer yes that experiments report statistical-significance information, with the justification pointing to Section 4.The supplied passages do not specify the particular statistical tests or reported quantities.Source: §7 · p. 20

8. Experiments compute resources

  • The authors answer yes that computer-resource information sufficient for reproduction is provided, with the justification pointing to Section 3.The supplied passages do not enumerate the resources or per-run compute requirements.Source: §8 · p. 21

9. Code of ethics

The checklist reports ethics compliance, describes the paper’s societal-impact discussion, and records how the authors address assets, new code, and human-subjects considerations.

9. Code of ethics

  • The authors answer yes that the research conforms to the NeurIPS Code of Ethics.The checklist also says potential positive and negative societal impacts are discussed in the Introduction.Source: §9 · p. 21

14. Crowdsourcing and research with human subjects

  • The authors state that the project involves neither human participants nor crowdsourcing, marking the participant-instruction and IRB items not applicable.The checklist separately says the research followed ethical guidelines, including privacy protection and responsible AI development.Source: §9 · pp. 21, 23

Responsible release

  • The authors mark safeguards for high-risk releases not applicable, stating that they use no risky data or models.Source: §9 · p. 22

12. Licenses for existing assets

  • The authors answer yes that relevant reports and papers are cited for existing assets, but the supplied justification does not state asset licenses or terms.The checklist asks for licenses and terms to be specified for assets.Source: §9 · p. 22

New assets

  • The authors say a new code repository will be submitted with supplementary materials.The checklist answer is yes that new assets are documented alongside the assets.Source: §9 · p. 23

A Limitations

The section identifies uncertainty about selecting diverse model pools and declining diversity during training, alongside compute costs, a bias-amplification risk, and appendix details on judge reweighting and dataset splits.

A Limitations

  • The optimal model combination for diversity probing remains unclear, and the experiments select models using their initial validation-performance ranking.Source: A Limitations · p. 25
  • Model collaboration uses more compute than a single model, while the authors report weak-to-strong generalization across tasks and initially weaker models becoming top performers.The latter result selects the best-performing checkpoint from a pool of 80 using validation performance.Source: A Limitations · p. 25
  • Model diversity diminishes over iterations, which the authors say indicates that performance improvements may not be infinite.They identify preserving diversity as important for sustained gains.Source: A Limitations · p. 25

B Ethics Statement

  • The authors warn that a biased model may propagate or amplify bias through collective alignment, particularly if reputation rewards patterns or datasets it overfits.Source: B Ethics Statement · p. 25

C.1 Reweighting Mechanism

  • From iteration 2, one additional low-ranked judge is down-weighted per iteration; with γ = 0.1, assigned reductions remain fixed for the rest of training.The mechanism is intended to maintain diversity while preventing unreliable judges from affecting preference-label consistency.Source: §C.1 · p. 25

C.2 Experiment Dataset Statistics

  • The appendix describes dataset splits into train, validation, and test sets, with validation formed by splitting the original test set evenly.Table 4 is described as covering 11 tasks; the Normad split is balanced across Country, Value, and Rule-of-Thumb subdomains.Source: §C.2 · pp. 25–26, Table 4 · p. 26

C.3 SFT Details

The experiments describe fine-tuning and controlled baseline settings, specify task-specific evaluations, and report MedQA strengths in informativeness and logical consistency alongside a near-leading comprehensiveness score.

C.3 SFT Details

  • Fine-tuning uses LoRA with a learning rate of 2e-4, cosine scheduling, effective batch size 32, warm-up ratio 0.1, and five default epochs; ShareGPT uses one epoch.The training mixture uses Tulu-v2 subsets and replaces GPT4 Alpaca with Gemini Alpaca.Source: §C.3 · p. 26

C.4 Baseline Experiment Settings

  • Baseline methods use identical inference and training configurations, including optimization hyperparameters, iterations, and sampling prompts, to isolate alignment-algorithm effects.Source: §C.4 · p. 26

C.5 Experiment Details

  • MedQA responses are assessed for informativeness, logical coherence, and comprehensiveness using LLM-as-a-judge scoring.The supplied prompt passages describe criteria including medical-information depth, reasoning structure, and query coverage.Source: §C.5 · p. 27
  • SPARTA ALIGNMENT scores highest on MedQA Informativeness and Logical Consistency, while SPIN leads Comprehensiveness at 6.75 versus SPARTA ALIGNMENT at 6.73.The three metrics assess medical-response content, logic, and completeness.Source: §C.5 · p. 28

C.6 Details in Generalization

Across the MATH generalization evaluation and further analyses, SPARTA ALIGNMENT is reported to improve performance while responses become more aligned, and preference pairs favor richer answers. The appendix also describes how model pools and response diversity are evaluated.

C.6 Details in Generalization

  • Models trained on one MATH difficulty subset are tested on the other two, and reported performance improves across basic, intermediate, and advanced problem-solving contexts.The benchmark subsets are Easy, Medium, and Hard.Source: §C.6 · p. 28

C.7 Details in Scaling Law of Model Pool

  • The pool-size analysis selects the top 3, 5, or 7 models with maximally divergent test performance, using Alpaca, COM2, and MedQA.ALPACA and COM2 scores are divided by 10 for unified comparison across task scoring scales.Source: §C.7 · p. 29

C.8 Further Analysis of Generation Diversity

  • The diversity comparison table reports higher character-level entropy and lower semantic similarity for SPARTA than SELF-REWARD.Source: Table 7 · p. 29
  • Compared with SELF-REWARD, SPARTA preference pairs show greater differences in semantic similarity and length, and their chosen responses are more complete, clinically informative, direct, and logically structured.The authors note that character entropy is especially sensitive to response length.Source: §C.8 · p. 29
  • Cross-response diversity is tracked through pairwise edit, BLEU, and embedding distances, plus LLM-as-a-Judge scores across ten models at each iteration.These measures cover structural, lexical, and semantic variation, as well as an LLM's assessment of perceived quality.Source: §C.8 · pp. 29, 31
  • Edit, BLEU, and embedding scores decline as test performance improves across iterations, while the LLM-as-a-Judge score remains relatively stable.The authors interpret the declines as convergence toward more structured, semantically aligned responses, potentially at the expense of surface-level diversity.Source: §C.8 · p. 31
  • MedQA case studies report that chosen responses are comprehensive, accurate, direct, and rich in clinical information compared with rejected responses.The case-study figures also describe chosen responses as more logically structured and clinically informative.Source: §C.8 · p. 31, Figure 9 · p. 34, Figure 11 · p. 36

C.9 Details in Correlation

The appendix relates model reputation to task performance and describes reputation tiers that emerge over iterations. It also flags the risk of inaccurate reputation assignments and identifies finer evaluation as a direction for future work.

C.9 Details in Correlation

  • KCross (r = 0.52), Math (Medium) (r = 0.4), and Math (Hard) (r = 0.5) show positive correlations between task performance and average reputation.Source: §C.9 · p. 37
  • Across iterations, the ten models form distinct reputation tiers, with some consistently in upper strata and others converging toward lower tiers.The authors suggest high-reputation models may guide lower-reputation models, while leaving a possible bottom-up effect as an open question.Source: §C.9 · p. 37, Figure 13 · p. 37
  • If lower-performing models consistently occupy upper tiers, the reputation assessment may be misaligned and could let less capable models disproportionately influence collective decisions.The authors identify more granular metrics for edge cases and task-specific nuances as a way to refine reputation assignments.Source: §C.9 · pp. 37–38

C.10 Case Study

Related papers

Loading 2506.04721v3…