PaperlayerView exact revision on arXiv

Source-linked AI summary

DISCO: Diversifying Sample Condensation for Efficient Model Evaluation

Alexander Rubinstein, Benjamin Raible, Martin Gubri, Seong Joon Oh

arXiv:2510.07959v2cs.LGcs.AI

TL;DR

Modern model evaluation can require hundreds to thousands of GPU hours, and existing anchor selection often relies on clustering. DISCO selects samples that elicit model disagreement and predicts performance from model signatures; it reports lower costs and stronger performance than prior methods, with a stated limitation under shifts in the model population.

  • Problem

    Evaluating modern models is increasingly expensive because models, datasets, and benchmarks have grown, particularly for general-purpose LLMs requiring broad evaluation.

    Source: §6 · p. 10

  • Method

    DISCO selects a small evaluation subset using model disagreement and predicts performance from the models’ outputs on that subset.

    Source: §1 · p. 2, §4.1.2 · p. 5, §4.2.2 · p. 6

  • Results

    Across language and vision, DISCO consistently outperforms prior methods; on MMLU, it reduces evaluation cost by 99.3% with 1.07 percentage points of error.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results support selecting samples on which models disagree rather than relying on broad coverage of sample difficulty.

    Source: §5.5 · p. 10

    DISCO’s robustness is limited under shifts in the model population, such as new architectures, training methods, or objectives that introduce unseen patterns.

    Source: §6 · p. 10

Abstract

from arXiv · show

Evaluating modern machine learning models has become prohibitively expensive. Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model. Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact. The typical approach follows two steps. First, select an anchor subset of data. Second, train a mapping from the accuracy on this subset to the final test result. The drawback is that anchor selection depends on clustering, which can be complex and sensitive to design choices. We argue that promoting diversity among samples is not essential; what matters is to select samples that $\textit{maximise diversity in model responses}$. Our method, $\textbf{Diversifying Sample Condensation (DISCO)}$, selects the top-k samples with the greatest model disagreements. This uses greedy, sample-wise statistics rather than global clustering. The approach is conceptually simpler. From a theoretical view, inter-model disagreement provides an information-theoretically optimal rule for such greedy selection. $\textbf{DISCO}$ shows empirical gains over prior methods, achieving state-of-the-art results in performance prediction across MMLU, Hellaswag, Winogrande, and ARC. Code is available here: https://github.com/arubique/disco-public.

1 INTRODUCTION

Modern model evaluation can require hundreds to thousands of GPU hours, while existing efficient approaches select anchor subsets and predict full-benchmark performance from them. DISCO instead targets samples that elicit diverse model responses and reports a stronger efficiency–precision trade-off than prior methods.

1 INTRODUCTION

  • Modern benchmark evaluation can take hundreds to thousands of GPU hours as models grow, test-time computation scales, and evaluation requirements broaden.LMMs-Eval is reported to require 30–1400 hours on 8 A100 GPUs, while HELM requires more than 4000 GPU hours.Source: §1 · p. 1
  • Prior efficient-evaluation frameworks select static anchor points, often through clustering response similarities across reference models, then extrapolate full-benchmark performance from subset accuracy.Source: §1 · p. 1
  • DISCO argues that informative selection should maximize diversity in model responses, and uses direct model signatures with simple predictors rather than estimating hidden model parameters.The paper states that inter-model disagreement is most informative for differentiating and ranking models.Source: §1 · p. 2
  • 99.3% lower evaluation cost with 1.07 percentage points of error on MMLU illustrates DISCO’s stronger efficiency–precision trade-off against prior methods.The approach is evaluated in language and vision domains.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses costly evaluation through label-efficient testing, benchmark construction, and anchor-point selection. DISCO distinguishes itself by selecting for model disagreement and predicting performance from model signatures.

2 RELATED WORK

  • LMMs-Eval can require 30–1400 GPU hours per model, and HELM can exceed 4000 GPU hours, as model scale, test-time computation, and evaluation breadth increase costs.Source: §2 · p. 2
  • Active-testing methods allocate labeling budgets to informative examples, but are not directly applicable when inference costs matter because they require full-test-set inference.Source: §2 · p. 2
  • Anchor-point approaches choose small subsets using source-model correctness patterns and estimate target performance from weighted accuracy; other work also develops efficient benchmarks and benchmark-compression methods.Source: §2 · p. 2
  • DISCO uses model disagreement instead of sample representativeness and maps model signatures to performance instead of relying on direct accuracy or latent-parameter estimation.Source: §2 · p. 3

3 PROBLEM

The problem is to estimate a model’s performance on a benchmark cost-effectively by evaluating a smaller subset of the dataset. The setup distinguishes source models with known performance from target models whose performance is to be estimated.

3 PROBLEM

  • The task is to estimate a predictive model’s performance on a dataset sampled independently from a distribution.Source: §3 · p. 3
  • The method seeks a subset of size K smaller than the original dataset so that performance can be estimated cost-effectively.Source: §3 · p. 3
  • A held-out source-model set has known ground-truth performance, while target-model performance is the quantity to estimate.Source: §3 · p. 3

4 SOLUTION

DISCO selects a small evaluation subset from source-model outputs, then estimates target-model performance from outputs on that subset. Its selection favors disagreement, while its prediction stage uses model signatures rather than scalar anchor accuracy or latent-parameter modeling.

4 SOLUTION

  • DISCO has two stages: select a smaller subset using an original dataset and held-out source models, then predict performance from target-model outputs on the selected set.Source: §4 · p. 3

4.1 DATASET SELECTION

  • Samples are ranked by an informativeness score, and the top-k are selected to preserve information about the full dataset.Source: §4.1 · p. 4

4.1.1 PRIOR SELECTION METHODS

  • DISCO replaces clustering-based anchor selection with sample-wise statistics that target model diversity rather than confidence or correctness.Source: §4.1.1 · p. 4

4.1.2 DISCO SELECTION

  • The selection method takes top-k samples by disagreement, and the paper concludes that the sample with greatest generalized Jensen–Shannon divergence carries the most information for predicting performance.Predictive diversity score (PDS) is also considered as a more interpretable diversity measure, continuous with the number of unique argmax predictions.Source: §4.1.2 · p. 5

4.2 PERFORMANCE PREDICTION

  • After selecting a subset, DISCO uses a target model’s responses on it to estimate the model’s true performance.Source: §4.2 · p. 5

4.2.1 PRIOR PREDICTION METHODS

  • Unlike prior methods based on scalar anchor-set summaries, DISCO maps concatenated raw outputs directly to performance, avoiding psychometric modeling complexity.Source: §4.2.1 · p. 6

4.2.2 DISCO PREDICTION

  • A model signature concatenates outputs on the selected subset; PCA may reduce its dimensionality to lower storage burden and improve generalizability.The signature can be high-dimensional because it combines output dimensions across selected samples.Source: §4.2.2 · p. 6
  • DISCO considers kNN prediction from nearby model signatures and parametric regression trained on source-model signatures to estimate performance.The parametric mapping may use a neural network, linear regression, or Random Forest.Source: §4.2.2 · p. 6

5 EXPERIMENTS

The experiments evaluate a shared anchor-selection and performance-prediction protocol across language and vision benchmarks. DISCO combines model signatures with predictive diversity scoring (PDS), and its results are examined across compression rates and design choices.

5.1 EVALUATION PROTOCOL

  • The shared protocol trains a predictor from source-model outputs on selected anchors, then tests its estimated performances against ground truth.All methods follow the same training and testing sequence.Source: §5.1 · p. 7

5.2 SETUP

  • The language experiments cover four benchmarks and 424 models, with a chronological split training on models published before January 13, 2024 and testing on newer models.The split uses a 9:1 train-test ratio.Source: §5.2 · pp. 7–8

5.3 MAIN RESULTS

  • 1.07%p MAE and .987 rank correlation: on MMLU, DISCO with PDS improves on model signatures with uniform random sampling (1.81%p MAE, .933 rank correlation).Table 1 covers MMLU, HellaSwag, Winogrande, and ARC; Figure 4 compares true and estimated MMLU performance for 40 models.Source: Table 1 · p. 8, §5.3 · p. 8, Figure 4 · p. 8
  • DISCO maintains the best efficiency–precision trade-off across compression rates; at 10 samples, kNN outperforms Random Forest.Figure 5 reports MAE and Spearman rank correlation across compression rates.Source: §5.3 · pp. 8–9, Figure 5 · p. 9

5.4 FACTOR ANALYSIS

  • Chronological splitting gives .987 rank correlation versus .986 for uniform splitting, while stratification gives .978 versus .987 with PDS.The comparisons use MMLU; stratification allocates equal anchor counts across its 57 tasks.Source: §5.4 · p. 9, Table 2 · p. 9
  • With 100 source models DISCO reaches .969 rank correlation, and performance rises to .987 with 382; PCA at 256 dimensions also reaches .987.Without dimensionality reduction, correlation is .918; Random Forest achieves the highest correlation among prediction models at .987.Source: §5.4 · p. 9, Table 2 · p. 9

5.5 RESULTS FOR VISION DOMAIN

  • On ImageNet-1k, 100 anchors reduce inference cost by 99.8%; PDS with Random Forest yields 0.63%p MAE and .969 rank correlation.DISCO with uniform random sampling and Random Forest yields 0.86%p MAE and .944 rank correlation.Source: Table 3 · p. 10, §5.5 · p. 10

6 CONCLUSION

DISCO targets the high cost of broad ML-model evaluation by estimating performance from a small informative subset, with reported evaluation-cost reductions and practical uses. Its accuracy is constrained by shifts in the model population and by tasks without predefined answer choices.

6 CONCLUSION

  • DISCO selects a small informative evaluation subset and estimates model performance from predictions on it, cutting evaluation costs by over 99% with minimal error.The conclusion also reports that DISCO consistently outperforms prior methods.Source: §6 · p. 10
  • Efficient evaluation can support limited-compute use, frequent performance tracking during training, and inexpensive checks of deployed models.These are the practical uses identified by the authors.Source: §6 · p. 10
  • Robustness is limited under shifts in the model population, where new architectures, training methods, or objectives can introduce unseen patterns and reduce estimator accuracy.The authors suggest adaptive sample selection or periodic retraining as possible future work.Source: §6 · p. 10
  • DISCO requires predictive probabilities over predefined answer choices, so open-ended translation or summarization first requires defining correct and incorrect output sets.The authors leave experiments on such tasks for future work.Source: §6 · p. 10

DISCLAIMER FOR USE OF LLMS

The authors used LLMs to assist with coding and writing, but not with ideation or experimental design.

DISCLAIMER FOR USE OF LLMS

  • LLMs were used in coding co-pilot applications to facilitate experimentation and help with plotting code.The paper describes these as assistance uses.Source: DISCLAIMER FOR USE OF LLMS · p. 16
  • LLMs also assisted in refining the paper's writing, and the authors reviewed and finalized the final version.The authors state that they carefully reviewed the final version.Source: DISCLAIMER FOR USE OF LLMS · p. 16
  • LLMs were not used in ideation or experimental design.This distinguishes the disclosed assistance from the research conception and design.Source: DISCLAIMER FOR USE OF LLMS · p. 16

A EXTENDED EXPERIMENTAL SETUP

The experimental setup covers four language benchmarks and describes DISCO's computational context and reusable offline/online pipeline.

A EXTENDED EXPERIMENTAL SETUP

  • DISCO is evaluated on MMLU, HellaSwag, Winogrande, and ARC, including MMLU's 57 tasks and Winogrande's 273 pronoun-resolution problems.ARC contains natural grade-school science questions, while HellaSwag focuses on commonsense inference.Source: A EXTENDED EXPERIMENTAL SETUP · p. 16

B COMPUTATIONAL COSTS

  • Cost estimates cover DISCO's main stages and direct target-model evaluation on one H100, extrapolated from five diverse 32B LLMs evaluated on MMLU.The estimates report standard deviation across five runs.Source: B COMPUTATIONAL COSTS · p. 16

B.1 DISCO PIPELINE OVERVIEW

  • Online, each target model is evaluated on the anchors, its outputs form a signature, and the offline-trained predictor estimates full-test performance.The offline anchor points and predictor are reused for each target model.Source: §B.1 · p. 16
  • Offline, DISCO evaluates source models, selects 100 PDS/JSD anchors, forms signatures, and trains a predictor to estimate full-test performance.The experiment uses 385 source models.Source: §B.1 · p. 16

B.2 COST METRICS

DISCO's reported cost savings are weighed against its offline setup cost; the section also compares model rankings and run-to-run stability.

B.2 COST METRICS

  • 3284 GPU-hours are required by the offline stage, while DISCO saves 8.46 GPU-hours per evaluated model on average.The reported mean saving is computed from 8.53 GPU-hours for direct evaluation minus 0.07 GPU-hours of online DISCO cost.Source: §B.2 · p. 16, Table 4 · p. 17

B.3 BREAK-EVEN ANALYSIS: HOW MANY EVALUATIONS JUSTIFY DISCO SETUP?

  • 389 evaluations are DISCO's break-even point; an OLMo-2-32B training run's 753 checkpoints already exceed it.The authors note that downloadable offline predictions can sometimes remove the need to evaluate source models.Source: §B.3 · p. 17

B.4 COMPARISON TO ALTERNATIVE APPROACHES

  • 99.3% of evaluation cost is saved versus full evaluation; the comparison table reports DISCO as dominating the other methods.Online costs differ negligibly across methods, and offline costs are equal up to rounding.Source: Table 6 · p. 17, §B.4 · p. 18

C QUALITATIVE MEANING OF RANK CORRELATION IMPROVEMENTS

  • 91.6 to 98.7 is the reported rank-correlation change from direct evaluation to DISCO, whose predicted ranks align almost perfectly with true ranks.Figure 6 plots true versus predicted ranks; ρs denotes Spearman rank correlation.Source: C QUALITATIVE MEANING OF RANK… · p. 18, Figure 6 · p. 18

D.1 REPORT CONFIDENCE INTERVALS

  • DISCO results are more stable than IRT and random sampling; the table summarizes MMLU MAE and Spearman rank results over five runs.The authors attribute the difference to Random Forest initialization being DISCO's only random component, unlike IRT's additional stochastic-gradient randomness.Source: §D.1 · p. 18, Table 7 · p. 19

D.2 MULTIPLE TRAIN/TEST SPLIT FOR CHRONOLOGICAL EVALUATION

Across bootstrapped chronological splits, DISCO remains superior despite modest shifts in mean results. In the calibration analysis, higher model confidence is associated with lower DISCO prediction error and appears to be the dominant ECE component influencing that error.

D.2 MULTIPLE TRAIN/TEST SPLIT FOR CHRONOLOGICAL EVALUATION

  • Each chronological run split models into 385 old and 40 new, then bootstrapped 346 source and 36 test models from those groups.The protocol was used to create five train/test chronological splits.Source: §D.2 · p. 19
  • For DISCO, rank correlation shifted from 98.6 to 98.0 and MAE from 1.06 to 1.25 across bootstrapped chronological splits, without changing its superiority over baselines.Table 8 covers five runs and reports MAE (%p) and Spearman rank correlation (%).Source: §D.2 · p. 19, Table 8 · p. 19

E SENSITIVITY OF DISCO TO MODEL CALIBRATION

  • A Pearson correlation of 0.49 between MAE and ECE indicates that better-calibrated models have lower DISCO prediction error.Figure 7 presents the correlation between DISCO prediction error (MAE) and Expected Calibration Error (ECE).Source: E SENSITIVITY OF DISCO TO MODEL… · p. 19, Figure 7 · p. 20
  • The analysis attributes confidence's informativeness to its relationship with correctness: calibration makes confidence patterns more predictive of performance.For perfectly calibrated models, confidence and correctness are deterministically and monotonically related, yielding high mutual information; miscalibration can reduce it.Source: E SENSITIVITY OF DISCO TO MODEL… · p. 19
  • A Pearson correlation of ´0.47 between MAE and mean confidence suggests overall confidence is the dominant ECE component influencing DISCO performance.Figure 8 presents the corresponding scatter plot.Source: E SENSITIVITY OF DISCO TO MODEL… · pp. 19–20, Figure 8 · p. 20

F PERFORMANCE GAP EXPERIMENTS

The section examines DISCO’s prediction behavior across source–target performance gaps and its anchor-selection trade-offs in vision, alongside theoretical bounds relating model-response heterogeneity to predictive information.

F PERFORMANCE GAP EXPERIMENTS

  • The gap experiments compare top-10% or top-30% targets with bottom-90% or bottom-50% sources, yielding gaps of 0.07%p or 8.18%p.The authors characterize a substantial source/target performance gap as an unrealistic failure mode: model accuracies are often mixed as new models develop, and extra evaluations can keep the gap low.Source: F PERFORMANCE GAP EXPERIMENTS · pp. 20–21
  • For a source/target split with a performance gap, DISCO differs from direct evaluation by 1.8%p, lower than IID (6.5%p), chronological (7.1%p), and performance splits without a gap (8.7%p).Source: F PERFORMANCE GAP EXPERIMENTS · p. 20

G.0.3 PROPOSITION. ´

  • Under a uniform model-index prior and deterministic predictions, Proposition 3 equates mutual information between predictions and model index with JSD when the statistic S is injective.A nonuniform model prior can be represented by replicating models in proportion to their sampling probabilities; DISCO’s JSD and predictor benefit from heterogeneous conditional prediction distributions, while duplicate copies yield zero JSD and mutual information.Source: §G.0.3 · pp. 21–22, §G.0.2 · p. 21

H.1.2 PROPOSITION.

  • JSD is bounded below quadratically and above linearly by total variation between each model’s predictive distribution and the ensemble mixture.The bounds apply for M ≥ 2 model-conditional distributions; the authors describe JSD as quadratic near equality and linear under worst-case separation.Source: §H.1.2 · p. 23

H.3 FINAL SANDWICH INEQUALITY

  • Total variation is bounded by the predictive diversity score through spread–envelope bounds, yielding a quadratic lower and linear upper bound on JSD in PDS − 1.The per-class spread–envelope bounds depend on z, the maximum number of models whose class probability exceeds the ensemble mean.Source: §H.2.2 · p. 24, §H.3 · p. 25

I IMPLEMENTATION DETAILS

  • For MMLU, using all source models to compute disagreement performed worse than using a subset, so the authors randomly sampled M = 100 models as a tuned hyperparameter.They attribute the effect to highly similar or redundant models diluting ensemble heterogeneity; parametric-model training took less than one minute in both domains.Source: I IMPLEMENTATION DETAILS · p. 25
  • Gradient Boosting uses scikit-learn defaults with 200 estimators, while PCA and UMAP component counts are varied as shown in Table 2.The implementations use scikit-learn (Pedregosa et al., 2011) and umap-learn, respectively.Source: I IMPLEMENTATION DETAILS · p. 26

J.1 SETUP

  • The setup evaluates 400 pretrained ImageNet models on 1.28 million images, using a chronological model split and MAE and Spearman rank correlation.Models range from 0.3M to 300M parameters; the split cutoff is 5 April 2023, with an 88:12 train-test ratio.Source: §J.1 · p. 26

J.2 ABOUT BASELINES FOR VISION DOMAIN

J VISION RESULTS

  • Across ImageNet compression rates, DISCO achieves a better efficiency–precision trade-off than prior methods.Figure 10 reports MAE and Spearman rank correlation; at 100 samples, its results are identical to Table 3.Source: Figure 10 · p. 27

J.2 ABOUT BASELINES FOR VISION DOMAIN

  • The paper argues that diversity-inducing data points are preferable to difficulty coverage, and tests this claim against Prabhu et al. (2024) and Fogliato et al. (2024) on ImageNet.The baseline results were computed by the authors because those papers did not report ImageNet results; the comparison uses the same setup.Source: §J.2 · p. 27

J.3 MAIN RESULTS

In the vision results, adding predictive diversity scoring (PDS) improves model-signature predictions over uniform random sampling. DISCO also maintains strong agreement with true performance and outperforms baselines with as few as 10 samples.

J.3 MAIN RESULTS

  • 0.63%p MAE and .969 rank correlation with PDS improve on 0.86%p MAE and .944 rank correlation with uniform random sampling.The passage reports that this two-stage trend is similar to the language domain and that the uniform-sampling model-signature approach significantly outperforms the naive baseline.Source: §J.3 · p. 27
  • A .970 Pearson correlation indicates strong agreement between estimated and true performance for 50 ImageNet models.Figure 9 presents the scatter plot of estimated versus true accuracy.Source: §J.3 · p. 27, Figure 9 · p. 27
  • With as few as 10 samples, DISCO consistently outperforms all baselines; kNN is the exception to the otherwise stable method rankings as anchor counts vary.Figure 10 compares performance across levels of test-set reduction.Source: §J.3 · p. 27
Loading 2510.07959v2…