PaperlayerView exact revision on arXiv

Source-linked AI summary

Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation

Jhen-Ke Lin

arXiv:2608.30044v1cs.AIcs.CLcs.LG

TL;DR

Benchmark lists often grow without an explicit measurement design, allowing equal weighting to make publication density an implicit capability weight. BoB uses inverse-density semantic weighting and a query-conditioned residual field after score equating. Across held-out prediction and duplication tests, the components respectively support task-conditioned prediction and robustness to benchmark multiplicity.

  • Problem

    Equal-weight aggregation can repeatedly count densely benchmarked capability regions, while existing approaches do not model how a retained benchmark list samples task space.

  • Method

    BoB embeds benchmark descriptions for inverse-density weighting, equates heterogeneous scores onto a common latent scale, and uses nearby residuals to condition global ability on a task query.

  • Results

    BoB reaches 0.462 profile correlation in held-out prediction on 586 models and 14 benchmarks, while inverse-density weighting stabilizes aggregates under benchmark multiplicity.

  • Takeaways & Limitations

    The residual field provides task-conditioned prediction, inverse-density weighting provides multiplicity robustness, and benchmark-list composition becomes an explicit measurement decision.

  • Takeaways & Limitations

    Semantic proximity imperfectly represents redundant coverage, so richer capability-overlap representations are the main stated route for improvement.

Abstract

from arXiv · show

Language models are commonly compared by averaging scores across a benchmark list with equal weight. Such lists grow through publication outside an explicit measurement design, so equal weighting turns the density of published benchmarks into an implicit capability weight: densely benchmarked regions count repeatedly. We introduce Balance of Benchmarks (BoB), which embeds benchmark descriptions and assigns each benchmark an inverse-density semantic weight. Nearby entries share aggregate influence at a disclosed density scale. After equating heterogeneous scores onto a common latent scale, a residual field uses the same geometry to condition model rankings on a task query. The two components serve distinct empirical roles. On a snapshot of 586 models and 14 benchmarks, BoB predicts which models are unusually strong on a held-out task beyond their general ability, reaching a profile correlation of 0.462 compared with 0.049 under equal weighting. It also limits the influence of densely repeated benchmarks on the aggregate. After adding four copies of each benchmark in turn, the resulting rankings retain a Kendall tau of 0.995, compared with 0.936 under equal weighting. The residual field therefore provides task-conditioned prediction, and inverse-density weighting provides robustness to benchmark multiplicity. Together, they turn benchmark-list composition from an incidental property of evaluation suites into an explicit, controllable part of measurement design, providing a principled foundation for task-aware and multiplicity-robust model evaluation.

1 INTRODUCTION

Equal-weight benchmark aggregation can turn publication density into an implicit capability weight. BoB addresses this with inverse-density semantic weighting and a task-conditioned residual field, whose roles are separately evaluated.

  • Motivation: Equal weighting makes densely benchmarked capability regions count repeatedly in leaderboard aggregates, even without an explicit decision to weight them more.AIME and AIME’25 have angular distance 0.29, while Terminal-Bench variants have distance 0.36, against a median pairwise distance of 0.89.
  • Motivation: Existing methods either impose discrete category boundaries, depend on the current model population, or select subsets rather than continuously weighting a retained benchmark list.The stated gap is a model of how the benchmark list samples task space.
  • BoB: BoB embeds benchmark descriptions and uses local semantic density to assign continuous weights, sharing aggregate influence among nearby entries while preserving more influence for isolated entries.The construction avoids both capability taxonomies and duplicate thresholds.
  • Task-conditioned evaluation: Characteristic curves first equate heterogeneous benchmark scores onto a common latent scale before nearby residuals adjust a model’s global ability for a user-specified task.The adjustment returns toward global ability as nearby observed support vanishes.
  • Evaluation: Held-out prediction tests whether semantic locality identifies models unusually strong on a task beyond general ability, while densification and re-listing tests assess multiplicity robustness.The paper assigns task-conditioned prediction to the residual field and multiplicity robustness to inverse-density weighting.
  • Evaluation: On 586 models and 14 benchmarks, BoB reaches 0.462 profile correlation in held-out prediction, while shuffled descriptions average 0.336 and reach 0.430 at most.The comparison isolates an increment associated with correct semantic placement.

2 RELATED WORK

Related work addresses benchmark aggregation through curation, redundancy or selection methods, and semantic task representations. BoB combines continuous task-space composition modeling with task-conditioned ranking.

  • Leaderboard aggregation: Holistic suites and arenas emphasize breadth while leaving final aggregation to curation, so equal weighting can make benchmark counts determine capability contributions.Hand-maintained taxonomies make the choice explicit but introduce category boundaries and maintenance costs.
  • Benchmark redundancy and selection: Benchmark redundancy and selection methods estimate agreement, dimensionality, predictability, or marginal coverage, but score-based estimates depend on the evaluated model population.Related approaches may also compress evaluations while preserving model rankings.
  • Semantic task representations: Semantic task representations support score prediction and model routing for unseen tasks, while BoB uses locality to adjust global ability with nearby benchmark residuals.This produces a task-conditioned ranking rather than only a task or model representation.

3 METHOD

BoB separates semantic benchmark weighting from score-based ability and residual modeling. It equates heterogeneous scores, discounts dense semantic regions, and uses nearby residuals to produce task-conditioned rankings.

  • Overview: BoB assumes benchmark descriptions locate measured capabilities and that model performance varies smoothly through the resulting semantic space.The method uses one description per benchmark alongside a score matrix with missing entries.
  • 3.1 SCORE EQUATING: Score equating places heterogeneous benchmark scores on a common latent ability scale before residual modeling.Characteristic curves address differing difficulty and saturation ranges; θm is the resulting model-level ability.
  • 3.1 SCORE EQUATING: Robust cell weights limit large residual outliers, while alternating estimation fits benchmark discrimination and difficulty parameters and updates model abilities.The fitted residuals support later task-conditioned adjustments.
  • 3.2 SEMANTIC INVERSE-DENSITY WEIGHTING: Inverse-density weighting gives more influence to benchmarks in sparse semantic neighbourhoods and normalizes these contributions into semantic weights.The effective benchmark mass W depends on the disclosed density bandwidth hdens.
  • 3.2 SEMANTIC INVERSE-DENSITY WEIGHTING: A benchmark’s marginal effective mass decreases continuously with similarity to existing points, so dense regions receive progressively less aggregate influence without category boundaries or duplicate thresholds.For one isolated benchmark at distance d, ΔW = (1 − k(d))/(1 + k(d)).
  • 3.3 TASK-CONDITIONED RESIDUAL FIELD: A task query selects semantic neighbours whose residual performance adjusts global ability into a task-conditioned score.Kernel proximity, inverse-density weights, information, and robust cell weights jointly shape the local adjustment; sparse support regularizes it toward θm.
  • 3 METHOD: BoB keeps description-derived weighting independent of the model population while using score-derived quantities for fitted abilities, residuals, robustness, information, and bandwidth selection.The construction explicitly separates semantic and score channels, with semantic weights entering the ability fit.

4 EXPERIMENTS

The experiments evaluate BoB through held-out benchmark prediction, semantic densification, duplication robustness, component controls, and population-shift stability. BoB improves task-specific prediction while limiting ranking changes from repeated benchmarks and added model families.

  • Held-out benchmark prediction: 586 models with at least 5 of 14 benchmarks scored are evaluated using leave-one-benchmark-out prediction across 14 folds.BoB receives the held-out benchmark description as a query point absent from fitting, and profile correlation measures task-specific prediction beyond general ability.
  • Held-out benchmark prediction: BoB attains the highest mean level and profile correlations among the evaluated methods, with its profile advantage over equal weighting surviving Holm correction.The comparison includes fixed rules and two score-population-dependent endogenous weighting rules.
  • Semantic densification and duplication robustness: Benchmark marginal value falls smoothly with local semantic density: inserted points contribute 0.49, 0.94, and 1.00 units at distances 0.33, 0.60, and 0.80.The values are within 0.02 of the isolated-pair formula, and no duplicate threshold is used.
  • Semantic densification and duplication robustness: 0.995 versus 0.936 is the mean Kendall τ after adding four copies in BoB and equal-weighting re-listing scenarios, respectively.Across 14 scenarios, BoB’s within-scenario median and 95th-percentile displacements are 0.5 and 4.3 ranks of 586.
  • Ablations and controls: 0.462 is the complete method’s profile correlation, compared with 0.163 for characteristic curves alone and 0.089 for a residual field on the linear scale.The construction study compares complete, construction-specific alternatives rather than treating the rows as an additive decomposition.
  • Population-shift stability: Adding 38–75 models from one family leaves the pre-existing BoB ranking at Spearman 0.9999 or higher.The description-derived semantic weights remain fixed while score-side quantities are refitted.

5 LIMITATIONS

BoB’s semantic weighting is limited by how benchmark capability overlap is represented and by the scope of its evaluation. Its re-listing robustness is substantial but not exact, and held-out benchmark queries are easier than arbitrary deployment tasks.

  • Representation boundary: Semantic proximity can conflate re-listings, fresh samples, and independent benchmarks measuring the same capability.BoB gives these cases the same geometric treatment because score covariance would make weights population-dependent.
  • Representation boundary: The τ 2-Bench and τ-Bench Banking pair is not discounted relative to equal weighting under the current representation.Their distance rises from 0.509 to 0.647 without the lineage sentence, while global normalization assigns 2.08 equal-weight benchmark units.
  • Representation boundary: The semantic distribution depends on one embedded point per benchmark, a chosen bandwidth, and one embedding family across 14 benchmarks.Internal task extent and suite packaging remain outside the current estimator.
  • Re-listing robustness: Four-copy re-listing changes remain small relative to equal weighting but do not preserve rankings exactly.Across 14 scenarios, median and 95th-percentile displacements average 0.5 and 4.3 ranks, while the mean maximum is 16.6.
  • Deployment scope: Held-out benchmark descriptions represent an easier task-query regime than unconstrained user requests.External task distributions and prospective model selection are needed to test deployment behavior.

6 CONCLUSION

BoB makes benchmark-list composition an explicit measurement choice by combining score equating, inverse-density weighting, and task-conditioned residuals. Held-out prediction supports task conditioning, while densification and duplication experiments support multiplicity robustness.

  • 6 CONCLUSION: BoB makes benchmark density an explicit measurement choice rather than an incidental consequence of publication.Its components equate heterogeneous scores, adjust dense semantic neighbourhoods, and condition aggregates on user queries.
  • 6 CONCLUSION: Held-out prediction and shuffled geometry support task conditioning, while densification and duplication experiments identify inverse-density weighting’s distinct role.The residual field supports prediction beyond calibrated general ability, and inverse-density weighting stabilizes aggregates under multiplicity.
  • 6 CONCLUSION: The evaluation uses a fixed Artificial Analysis snapshot of 586 models and 14 benchmarks with frozen regeneration artifacts.The artifact includes anonymized code, frozen results, descriptions, queries, embeddings, fitted parameters, and regeneration scripts.
  • 6 CONCLUSION: Language models serve as evaluation subjects and also generate benchmark descriptions, paraphrases, and canonicalized demonstration queries.The text inputs and embedding configuration are frozen and specified in the implementation materials.

A DATA AND BASELINES

The data analysis uses a fixed Artificial Analysis API snapshot, explicit benchmark coverage handling, and several aggregation baselines. Two evaluation-side oracles estimate structure from the current score population but are excluded from BoB.

  • Dataset: The dataset contains 624 API rows, of which 586 retained models are evaluated across 14 named benchmarks.The snapshot was retrieved directly from the Artificial Analysis Data API at a specified timestamp.
  • Coverage: Missing JSON values are represented as missing cells, while observed numeric zeros are retained in curve fitting and residual-field computation.Robust cell weights attenuate large standardized residuals, and Table 4 reports coverage.
  • Coverage: Table 4 reports cell coverage among the 586 retained API rows, distinguishing observed zeros from JSON null counts.Observed zero is a subset of observed values.
  • Baselines: Equal weighting, hand-assigned categories, and nearest-benchmark aggregation provide comparison baselines.The category labels group benchmarks into mathematics, knowledge, coding, agentic, instruction following, and long context.
  • Baselines: The published Artificial Analysis Intelligence Index correlates 0.938 by Spearman with the equal-weight raw mean among 248 complete models.It is treated as an opaque baseline because its normalization is undocumented.
  • Baselines: Score-covariance GLS and the first principal component estimate structure from the current score population, but neither oracle enters BoB.GLS uses a ridge-regularized empirical correlation matrix, while PCA fills missing cells with column means.

B INVERSE-DENSITY WEIGHTING AND DUPLICATION DECOMPOSITION

Inverse-density weighting reduces the marginal influence of benchmarks added near existing entries, with exact cancellation for isolated co-located groups. Remaining ranking movement comes from shared-neighbour effects and refitting, and stability is measured with mean Kendall τ under re-listing.

  • Inverse-density weighting: A new benchmark increases existing densities by their similarities to the new point, while its own density equals one plus total similarity.The marginal weight change decreases componentwise as similarities increase.
  • Inverse-density weighting: For an isolated co-located group of size nG, adding an identical benchmark produces zero marginal aggregate weight change.This follows from each group member having density nG and similarity one to the new copy.
  • Inverse-density weighting: Against one isolated benchmark at distance d, the marginal weight change is ΔW = (1 − k)/(1 + k).Here k is the similarity kernel value k(d).
  • Neighbour effects: The zero marginal change is exact only for isolated groups because co-located copies can also increase neighbouring densities.For close pairs outside the duplicated group, cG ranges from 0.003 to 0.16, while the AIME pair reaches 0.46–0.50.
  • Ranking movement: 16.6 ranks is the scenario-averaged maximum displacement after four-copy re-listing with inverse-density weighting and refitting.Holding item parameters fixed yields 10.6 ranks from shared-neighbour effects; refitting raises this to 16.6, with a maximum θ change of 0.098 standard deviations.
  • Ranking movement: Figure 4 measures re-listing stability as mean Kendall τ between original and perturbed rankings across fourteen scenarios.Each benchmark is re-listed with 1, 2, 4, or 8 additional identical score columns.

C HELD-OUT EVALUATION DETAILS

The held-out evaluation compares BoB with ability-only and alternative controls for predicting benchmark-specific model strength. Under isotonic residualization, BoB improves profile correlation over calibrated ability alone, while paired tests assess baseline differences across folds.

  • Held-out design: Table 5 separates held-out performance for benchmarks inside and outside the AA Intelligence Index across 10 and 4 folds.This split is part of the per-benchmark held-out evaluation reported for the primary comparison.
  • Ability controls: 0.785 mean R2 explains held-out benchmark ordering with a quadratic in pool ability, while ability alone reaches 0.883 level.The corresponding quadratic profile metric is 0.094, motivating stricter isotonic residualization.
  • Statistical comparisons: Table 6 reports paired profile differences between BoB and each baseline using one-sided Wilcoxon signed-rank tests with Holm adjustment.The table summarizes comparisons across the held-out folds rather than a single aggregate split.
  • Statistical comparisons: Table 7 reports profile correlations after three controls for general ability and tests each baseline against BoB under isotonic residualization.The binned control uses sample-size-weighted mean Spearman correlation within ability deciles, omitting undersized or degenerate bins.
  • Profile evaluation: 0.446 profile correlation is achieved by BoB under isotonic residualization, versus 0.349 for calibrated θ alone.The paired difference is significant at p = 0.0012 across the 14 held-out folds.

D ABLATIONS AND VALIDITY CHECKS

Ablations separate BoB’s task-conditioned prediction from its inverse-density weighting and test alternative residual conventions. Flat weighting preserves profile prediction comparably but substantially worsens duplication robustness.

  • Construction controls: Rows A–D of Table 2 are construction-specific comparisons rather than an additive decomposition, and an additional linear z-score construction reaches 0.014 profile.The full-method controls instead use the pool-only evaluation path of the headline experiment.
  • Weighting ablation: 0.451 profile correlation under flat weighting is statistically comparable to BoB, but duplication shift rises to 0.973 standard deviations versus 0.098.Flat weighting also reaches 0.896 level, whereas inverse-density weighting contributes multiplicity robustness.
  • Description ablation: 0.336 ± 0.042 profile correlation results from 200 shuffled-description runs, below the observed 0.462.The maximum shuffled-run profile is 0.430, separating correct semantic placement from generic calibrated-residual pooling.
  • Residual convention: 0.435 profile correlation is obtained by a first-order latent-offset residual scale, providing a sensitivity check against the direct standardized-residual convention.The alternative reaches 0.904 level and performs slightly below the reported direct convention.
  • Support specification: 0.460 profile correlation results when normalized item information is included in the support factor, indistinguishable from the primary profile result at p = 0.67.The conclusion is unchanged when information enters both the residual average and support factor.

E TEXT AND REPRESENTATION ROBUSTNESS

Representation and selection checks show that held-out prediction is stable across paraphrased text, embedding sizes, and parameter choices. The reported semantic scales also determine how density and task-field locality are measured.

  • Text robustness: 0.474 profile correlation under independently paraphrased descriptions preserves the held-out result.The corresponding level is 0.903.
  • Text robustness: 0.454 profile correlation remains after removing cross-benchmark references from descriptions, compared with 0.462 using the original descriptions.AIME and AIME’25 remain the closest pair, while Terminal-Bench variants remain second.
  • Embedding-size robustness: 24.4, 16.1, and 16.6 ranks are the four-copy displacements for 0.6B, 4B, and 8B models, below equal weighting’s 96.9.Level and profile vary by less than 0.01 across embedding sizes.
  • Bandwidth selection: 0.45 times the median pairwise distance is selected for field bandwidth in every nested fold, yielding 0.902 level and 0.462 profile.The non-nested criterion selects h = 0.402 from the candidate bandwidths.
  • Density scale: 0.25 times the median distance sets density bandwidth hdens = 0.224 and effective benchmark mass W = 12.5.Across Figure 5a, W falls from 14.0 at 0.05 times the median distance to 1.6 at the median.
  • Parameter robustness: 0.991 and 0.987 are the Spearman correlations of benchmark weights when binomial sampling floors are doubled or halved.Robust-cap and floor sweeps move level by at most 0.004 and profile by at most 0.008.

G.7 PER-BENCHMARK CURVE PARAMETERS

The curve-parameter table records benchmark discrimination, difficulty, residual scale, fit quality, and semantic weight. The fitting procedure alternates between score calibration and model ability estimation, with robust residual weighting shared across the construction.

  • Parameter table: Table 12 reports discrimination ai, difficulty bi, residual scale σi, curve R2, and normalized semantic weight vi for all fourteen benchmarks.These are the fitted characteristic-curve parameters and description-derived weights used in the analysis.
  • Semantic weights: AIME and AIME’25 receive semantic weights 0.054 and 0.052, jointly 0.106 rather than 0.143 under two equal-weight benchmarks.The reduced aggregate weight corresponds to their reduced marginal mass.
  • Curve fitting: Six alternating calibration rounds fit each benchmark’s curve by weighted least squares and each model ability θm over a standardized grid.Abilities are standardized to mean zero and unit variance after fitting.
  • Robust fitting: wmi = min(1, (3/|rmi|)2) retains unit weight within three standard deviations and attenuates larger residuals quadratically.The weights are floored at 10−3 and enter the ability fit, residual field, and support mass.
  • Ranking comparison: 0.967 Spearman correlation and 0.855 Kendall τ characterize agreement between the two full rankings across 584 model configurations with both scores.The observed column counts non-missing scores among the fourteen analyzed benchmarks.
  • Ranking comparison: Table 13 compares the union of the top 20 model configurations under BoB’s global level score and the AA Intelligence Index.Bold entries indicate top-20 membership.
Loading 2608.30044v1…