LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamics

1 Introduction

• Static averages can treat a foundational failure and advanced-task excellence as independent addends, despite sequential workflows where the failure blocks downstream capability.The paper frames this as path-dependent model utility.Source: §1 · p. 2, Figure 2 · p. 3
• CSD converts multi-benchmark results into a dynamic competition, with benchmark importance emerging from tournament structure rather than subjective weights.Early-round failures alter pairing trajectories and maximum potential scores.Source: §1 · pp. 2–3
• Monte Carlo simulation over 100,000 iterations estimates E[Sm] to mitigate random-pairing luck in the competition-aware ranking.The paper describes E[Sm] as a statistically robust score representing predicted cumulative victories.Source: §1 · pp. 2–3
• CSD prioritizes consistent performance across benchmarks and penalizes models with localized deficiencies, distinguishing robust generalists from aggressive specialists.The introduction names this distinction as a central ranking behavior.Source: §1 · pp. 2–3
• Varying Tk provides a quantifiable Failure Sensitivity Analysis of model performance under different per-round elimination quantities.The analysis profiles models by risk appetite.Source: §1 · pp. 2–3

2.1 Framework Components and Rationale

• CSD models sequential deployment risk by testing models across benchmarks and rewarding full-spectrum capability under cumulative performance pressure.The framework is motivated by settings where early failures can make later strengths insufficient, including multi-step coding and supply-chain tasks.Source: §2.1 · pp. 3–4

2.1.1 The Pairwise Win-rate Tensor (W)

• W converts raw benchmark performance into binary pairwise win/loss outcomes, avoiding subjective benchmark weights and defining importance through sequence and competitive pressure.W is pre-calculated once, so it is not recomputed during Monte Carlo sampling.Source: §2.1.1 · p. 4

2.1.2 The Swiss-System Pairing Engine

• Swiss-System pairing groups models by cumulative score and matches similar scorers, concentrating comparisons among competitors of comparable proven strength.This tests high-performing models against other high performers and supports differentiation among low performers.Source: §2.1.2 · p. 4
• A bye earns zero points, so the score reflects competitive victories rather than chance-based score gains.Repeated Monte Carlo contests are described as mitigating single-instance stochasticity from pairing or elimination.Source: §2.1.2 · p. 5

2.1.3 The Structured Elimination Mechanism

• After each round, CSD permanently removes Tk models from the minimum-score group, making elimination pressure a tunable penalty for failure.The mechanism targets models with the poorest relative performance so far.Source: §2.1.3 · p. 5

2.2 Formal Stochastic Process

• At each round, active models are partitioned into score groups, and a model’s expected score depends on its group and its benchmark win rates against group members.For odd-sized groups, the expectation accounts for the possibility of receiving a zero-point bye.Source: §2.2 · p. 5
• Models add their round scores to cumulative totals, after which the minimum-score group is identified and elimination forms the next surviving set.The process ends if fewer than two models remain.Source: §2.2 · pp. 5–6
• The single-instance algorithm takes models, sequenced benchmarks, the win-rate tensor, rounds, and elimination parameters as inputs, and returns final scores and a ranking.Each round uses its current benchmark for contests before score updates and elimination.Source: §2.2 · p. 6

2.3 Intractability of the Analytical Solution

• The target expected final score is computed across K rounds, with round-level expectations combined by linearity of expectation.The derivation invokes the law of total expectation to compute each round’s expected score.Source: §2.3 · p. 7
• Each round’s state depends on earlier random pairings and eliminations, making the process path-dependent.The state includes active models and their cumulative scores.Source: §2.3 · p. 7
• Contest histories grow exponentially, making direct computation of every state probability infeasible for non-trivial M, K, and W.Source: §2.3 · p. 7

2.4 Monte Carlo Approximation of E[Sm(K)]

• Monte Carlo simulation approximates each model’s expected final score because the analytical solution is computationally intractable.Source: §2.4 · p. 7
• Each simulation runs the full K-round contest, and the estimated expected score is the sample mean of the final scores.The passage gives N = 10,000 as an example iteration count.Source: §2.4 · p. 7
• E[Sm] represents expected cumulative victories under sequential elimination, combining win rates with robustness against failure and resilience to random chance.The estimate converges toward E[Sm] by the law of large numbers.Source: §2.4 · p. 7

2.5 Failure Sensitivity Analysis (FSA)

• FSA traces E[Sm] across elimination settings Tk, turning a fixed ranking score into a model risk profile.The framework runs Monte Carlo simulations across Tk values to trace the score curve.Source: §2.5 · p. 7
• A more negative sensitivity coefficient Λm marks greater score decline and susceptibility to elimination, while a near-zero slope indicates stability.Λm is estimated as the slope of the E[Sm](Tk) function.Source: §2.5 · p. 8
• Aggressive Specialists score highly at low penalty but lose score as the penalty rises, whereas Robust Generalists remain relatively insensitive to elimination pressure.These categories distinguish models by the trade-off between high-scoring potential and defensive robustness.Source: §2.5 · pp. 7–8

3 Related Work

• Pointwise benchmarks measure isolated task proficiency, but their aggregate rankings depend on subjective weights across benchmarks.The passage names HELM, GLUE, and MMLU as examples of pointwise evaluation.Source: §3 · p. 8
• Elo and Bradley-Terry convert pairwise outcomes into skill ratings; CSD instead measures Expected Win Score within a prescribed competition structure [5].Elo and Bradley-Terry are described as modeling win-loss records to estimate latent skill.Source: §3 · p. 8
• Swiss-System pairings depend on accumulated wins, so CSD models path-dependent competition in which high performers face progressively harder opponents.This differs from Elo and Bradley-Terry systems designed for independent pairwise outcomes.Source: §3 · p. 8
• CSD adds failure sensitivity through Tk, profiling vulnerability to elimination from a short board—an evaluation dimension standard Elo systems lack.The paper positions contest structure as a diagnostic tool, rather than primarily a way to optimize pairing efficiency.Source: §3 · pp. 8–9
• The authors frame CSD as a risk-informed, context-aware method for ranking LLMs for specific deployment pipelines.They characterize it as a step toward next-generation evaluation.Source: §3 · p. 9

4.1 Overall Ranking of Advanced LLMs Across 38 Benchmarks

• The evaluation uses 38 recent, widely used, open-source benchmarks across six capability categories, organized into 12 sequential sub-categories.The categories include knowledge, reasoning, instruction following, coding, agent capabilities, and factuality.Source: §4.1 · p. 9
• The benchmark sequence moves from general to specific and foundational to challenging tasks, giving foundational performance higher implied weight in the elimination-based contest.The authors place general knowledge and instruction following before more specialized capabilities.Source: §4.1 · pp. 9–10
• Claude-Sonnet-4.5-thinking and DeepSeek-V3.2-thinking form a relatively robust second tier, while Qwen3-Max leads a competitive third tier.The third-tier cluster includes GLM-4.6, Gemini-2.5-pro, DeepSeek-V3.2-Exp-Thinking, Gemini-2.5-Flash-Preview, and DeepSeek-V3.1-Terminus-Thinking.Source: §4.1 · p. 10
• Across 38 benchmarks, Figure 3 groups 29 LLMs into four performance tiers, led by Gemini-3-pro and GPT-5 variants.The first tier models show minimal score degradation as elimination pressure increases.Source: §4.1 · p. 10, Figure 3 · p. 11
• The fourth tier consists mainly of previous iterations, lightweight models, and base models without external thinking or planning mechanisms.The paper associates this cohort with suboptimal operational or architectural trade-offs and performance degradation.Source: §4.1 · p. 11
• Figure 4 plots base performance against sensitivity; Gemini-3-pro and several GPT-5 and other models exemplify robust generalists, while three Qwen models are classified as aggressive specialists.The performance drop compares Tk = 0 with Tk = 2; the sensitivity coefficient is the corresponding change per elimination increment.Source: §4.1 · p. 11, Figure 4 · p. 12

4.2 Sensitivity Analysis of CSD Framework

• Under an extreme perturbation, CSD minimally affects Qwen3-Max’s overall ranking, while simple averaging drops it to 12th; under a severe perturbation, CSD ranks it 10th (third tier) versus 19th for simple averaging.Source: §4.2 · p. 12
• CSD shows controlled rank declines under the tested isolated severe perturbations, but widespread unreliable benchmark scores require retesting.The paper also reports low sensitivity to extremely high scores on several specific benchmarks.Source: §4.2 · p. 13

5.1.1 Agentic Performance Prediction

• CSD’s sequential benchmark structure aligns with agentic workflows, but optimization for open-source agentic benchmarks can make prediction on novel tasks harder through contamination or overfitting.The discussion illustrates the mapping with a multi-step web navigation and data extraction task.Source: §5.1.1 · pp. 13–14

5.1.2 Ranking Models within One Single Benchmark

• For a single benchmark, CSD orders questions into difficulty tiers using average model success rates, then evaluates them sequentially, favoring consistent performance over isolated strengths.The discussion says models that fail easier questions but succeed on harder ones incur a performance drop.Source: §5.1.2 · p. 14
• Across MMLU-pro and SuperGPQA, Kimi models decline as T_k varies from 0 to 2, while Gemini-3-pro, GPT-5.1-high, and Claude-Sonnet-4.5-thinking are robust.The figure shows CSD sensitivity for models ranked within a single benchmark.Source: Figure 6 · p. 14, §5.1.2 · p. 14

5.1.3 Applying CSD to Datasets with Weights

• When benchmark weights are known but order is not, CSD samples benchmark permutations so higher-weight benchmarks are more likely to appear earlier, then estimates expected scores with Monte Carlo.Source: §5.1.3 · p. 15

5.2 Limitations

• No universally accepted ground-truth ranking exists, and CSD’s resilience-focused ordering may differ from utilities that prioritize peak performance on a specialized task.The discussion says external validation against an objective best list is consequently difficult.Source: §5.2 · p. 15
• Direct comparison with aggregate baselines is difficult because CSD’s shortfall-under-pressure metric is incommensurate with averages, while its resilience focus can produce different rankings from peak-performance methods.Source: §5.2 · p. 15

• The analysis relies on internal evaluation results, and rankings may fluctuate across evaluation setups because LLM scores, especially on agent benchmarks, are sensitive to environmental factors and API stability.Source: §6 · p. 16
• The code is available at the paper’s GitHub repository.Source: §6 · p. 16

• CSD simulates a dynamic, multi-round competition to address limitations of current LLM evaluation paradigms.Source: §7 · p. 16
• CSD makes benchmark importance depend on sequence and frequency in the contest rather than subjective weighting.Source: §7 · p. 16
• The ranking prioritizes dynamic competitive fitness and penalizes shortfalls to support LLM selection for complex, multi-stage deployment.Source: §7 · p. 16
• Future work includes integrating sequential dependencies for agentic prediction and exploring correlations between CSD rankings and real-world task failure rates.Source: §7 · p. 16