PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

Guoming Ling, Yupei Lin, Zhongzhan Huang, Junxin Li, Junjie Liu, Shanshan Zhong, Hefeng Wu, Liang Lin

arXiv:2610.18248v1cs.CL

TL;DR

Redundancy in widely used LLM benchmarks makes evaluation unnecessarily expensive, and differences among compression methods leave their requirements and reliability difficult to compare. This survey synthesizes 119 papers into a taxonomy and compares their information needs, costs, reliability checks, and limitations. It finds that methods reduce costs through representative subsets, cheaper or reusable information, and adaptive testing, with risks including poor coverage, inaccurate estimates, and sequential delays.

  • Problem

    Redundancy in widely used LLM benchmarks makes evaluation unnecessarily expensive, while varied information needs, cost assumptions, and validation approaches leave researchers without clear guidance for comparing methods.

    Source: §1 · p. 1

  • Method

    The survey reviews 119 papers, organizes benchmark compression into a taxonomy, and compares method families by information use, practical costs, testing, and limitations.

    Source: §1 · pp. 1–2

  • Results

    Methods reduce evaluation costs through representative subsets, cheaper or reusable information, and adaptive testing, while risking poor coverage, inaccurate estimates, and sequential delays.

    Source: §1 · p. 1, §8 · p. 8

  • Takeaways & Limitations

    The taxonomy and cross-family comparison provide guidance for method selection and validation in efficient LLM evaluation.

    Source: §1 · p. 2

    The survey covers work only through August 2026, gives a high-level account rather than describing every method in detail, and cannot quantitatively combine studies with widely different evaluation settings.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.

1 Introduction

Widely used LLM benchmarks contain redundancy that makes evaluation costly, while differences among compression methods leave researchers without clear guidance on method fit, costs, and reliability. This survey organizes 119 papers into a taxonomy and compares their approaches, trade-offs, and limitations.

1 Introduction

  • Fewer than 20% of test samples can preserve results close to full-set results on LongBench, GSM8K, and MMLU, while redundancy makes evaluation unnecessarily expensive.The introduction cites prior studies for these benchmarks and links the redundancy to slower model development.Source: §1 · p. 1
  • Differences in information needs, cost assumptions, and validation make compression methods difficult to compare and leave researchers without guidance on fit, cost savings, and reliability.Source: §1 · p. 1
  • The survey reviews 119 papers and organizes the literature into Single-Benchmark Compression, Cross-Benchmark Compression, Adaptive Evaluation, and Benchmark Analysis and Foundations.It compares information used, practical costs, testing, and failure modes across these categories.Source: §1 · p. 1
  • Compression methods reduce costs through representative subsets, cheaper or reusable information, and adaptive testing, but risk poor coverage, inaccurate estimates, and sequential delays.Source: §1 · pp. 1–2
  • The survey presents a principled taxonomy and decision path organized by evaluation setting and available information.The authors describe it as the first systematic survey of LLM benchmark compression, to their knowledge.Source: §1 · p. 2
  • The cross-family synthesis clarifies method requirements, reliability, and limitations, and offers guidance for selection and validation plus a research roadmap.Source: §1 · p. 2

2 Problem Formulation and Taxonomy

The survey defines compression as estimating a full-evaluation result from a lower-cost partial evaluation, then classifies methods by contribution, selection timing, benchmark scope, and information used. Its hierarchy gives each paper one primary label while recording other properties separately.

2.1 Problem Formulation

  • A compressed evaluation measures a lower-cost subset, fixed or adaptive, and may use auxiliary information to estimate the full-evaluation result.A full evaluation aggregates outcomes across units, which can include benchmark items, human judgments, or agent runs.Source: §2.1 · p. 2
  • Benchmark compression seeks a score or ranking estimate within an acceptable difference from the full result while using less evaluation cost.Source: §2.1 · p. 2

2.2 Taxonomy Overview

  • The taxonomy first separates operational cost-reduction methods from foundational analyses of benchmark evaluation and compression.Source: §2.2 · pp. 2–3
  • Fixed subsets are static compression, while dynamic selection during evaluation is adaptive; adaptive methods choose the next item or govern stopping.Source: §2.2 · p. 3
  • Among static methods, jointly compressing several benchmark evaluations is Cross-Benchmark Compression; otherwise the category is Single-Benchmark Compression.Source: §2.2 · p. 3
  • Single-Benchmark Compression is divided by information source into Content-Based, Historical-Response, and Target-Conditioned methods.These use benchmark-side information, earlier-model responses, or a cheap signal from the evaluated model, respectively.Source: §2.2 · p. 3
  • The hierarchy assigns each paper one primary label based on its main contribution and evaluation design, while recording other properties as secondary attributes.Figure 3 groups studies by primary contribution; the appendices provide classification rules, boundary cases, and detailed paper tables.Source: §2.2 · p. 3, Figure 3 · p. 3

3 Static Benchmark Compression

Static compression fixes subsets before the main evaluation, reducing one benchmark or several related benchmarks by using item data, prior responses, current-model signals, or shared suite information. The methods trade evaluation cost against the information needed to recover full results and the reliability of that recovery.

3.1 Single-Benchmark Compression

  • Single-benchmark compression estimates a full-benchmark score or ranking from a fixed subset, selected using item data, historical responses, or a cheap current-model signal.Static compression fixes the subset before the main evaluation begins.Source: §3.1 · p. 3, §3 · p. 3

3.1.1 Content-Based Compression

  • Content-based methods range from random or stratified sampling to coresets, content-guided sampling, and calibrated-judge correction, without a historical response matrix.Small random subsets can have high variance, and textual or embedding diversity may not ensure distinct abilities are tested; reduced information can make recovery less stable or accurate.Source: §3.1.1 · p. 4

3.1.2 Historical-Response Compression

  • Historical-response methods use past model-by-item results for item coverage, psychometric or learned recovery, and statistical correction; their patterns must transfer to new models.Building the response matrix can require many full evaluations unless public results are available, and its advantage may shrink under large model shifts.Source: §3.1.2 · p. 4

3.1.3 Target-Conditioned Compression

  • Target-conditioned methods use a cheap signal from the current model to tailor a fixed subset, including probes, uncertainty or disagreement, and proxy correction.They retain parallel batch execution, but probes or proxies must cost much less than the tests or reviews saved; weak signals can miss items or introduce proxy errors.Source: §3.1.3 · pp. 4–5

3.2 Cross-Benchmark Compression

  • Cross-benchmark compression can select informative items across benchmarks, run only a few complete tasks, or substitute cheaper proxy tasks to estimate target results.Methods use shared ability models, score-matrix completion, regression, importance weighting, or multilingual tensor models.Source: §3.2 · p. 5
  • Cross-benchmark methods share information across related benchmarks to recover scores or rankings, but their estimates may not transfer to unseen models.They can skip complete benchmarks or costly tasks; validate on unseen models and run unreliable components directly.Source: §3.2 · p. 5

4 Adaptive Evaluation

Adaptive evaluation uses results collected during testing to choose what to test next or whether to stop. Its potential savings depend on reliable score correction and stopping rules, while sequential execution can add delay and complicate comparisons.

4 Adaptive Evaluation

  • Adaptive evaluation uses results collected so far to select the next test or decide when to stop.Item selection updates estimates after each response or batch; stopping methods check after stages whether more testing is needed.Source: §4 · p. 6, §4.1 · p. 6

4.1 Adaptive Item Selection

  • Adaptive item selection directs testing toward informative items using ability estimates, predicted responses, uncertainty, rare failures, or noisy judge cells.Group weights or sampling probabilities can correct uneven selection, and repeated-cell means can estimate noisy comparisons.Source: §4.1 · p. 6
  • Adaptive item selection can lower cost, but serial rounds increase wall-clock time and give models different test paths, complicating direct comparison and reproduction.Adaptive subsets may stop representing the full benchmark, so reliable scores depend on suitable correction and sometimes calibrated historical data; complete test-path logs matter.Source: §4.1 · p. 6

4.2 Sequential Decision and Stopping

  • Sequential decision methods target a score precision, pairwise comparison, or best model under a fixed budget, and can accept, reject, escalate, or abstain.They stop when a score interval, rank, or pairwise difference meets a target, focus tests on plausible winners, or route uncertain cases to more costly evaluation.Source: §4.2 · p. 6
  • Sequential savings depend on the task: close models may require nearly the full benchmark, fixed-budget methods redistribute rather than reduce cost, and some return only a decision.Reliable early stopping requires a valid sequential rule; staged execution can increase wall-clock time and requires complete logs for reproduction.Source: §4.2 · p. 7

5 Benchmark Analysis and Foundations

The foundational analysis examines why benchmarks can be compressed, how many items can be removed, which evaluation settings must remain fixed, and how compression differs from benchmark repair.

Why Can Benchmarks Be Compressed?

  • Redundancy arises when items measure overlapping capabilities and add limited discrimination, appearing as correlated responses, low item discrimination, or low effective dimensionality.A subset that preserves overall ranking may still distort score gaps or reverse close models.Source: §5 · p. 7

How Many Items Can Be Removed?

  • There is no universally safe compression ratio: item count depends on calibration, score differences, paired variance, task-group coverage, and allowed error.When retained items cannot reliably separate two models, their comparison should remain unresolved.Source: §5 · p. 7

Which Settings Must Stay Fixed?

  • Full and compressed evaluations should keep judges, prompts, decoding settings, and agent harnesses the same, because protocol changes can alter comparisons.Aggregate judge accuracy can hide local errors, and removing weaker judges can discard useful signals.Source: §5 · p. 7

Compression or Benchmark Repair?

  • Compression seeks to reproduce the original result, whereas benchmark repair removes flawed questions to create a cleaner benchmark whose scores or rankings may change.A repaired benchmark must be validated as a new benchmark.Source: §5 · p. 7

6 Comparison Across Method Families

The comparison organizes method choice around when information becomes available, how full results are recovered, lifecycle costs, and the limitations that affect deployment.

6.1 Subset Selection: Information and Timing

  • Subset selection ranges from fixed choices based on benchmark data or historical responses to target-conditioned batches and adaptive decisions using new results.More target-specific information can improve selection but reduces reuse and parallelism.Source: §6.1 · p. 7

6.2 Recovering the Full Result

  • Recovery methods differ in their target: sampling and weighting can estimate the original pool, predictors depend on transfer, and sequential methods may return only a decision.Residual or gold-label correction can reduce dependence on prediction accuracy under suitable sampling conditions.Source: §6.2 · pp. 7–8

6.3 Lifecycle Cost

  • Lifecycle cost includes construction, target-side signals, expensive tests, recovery, updates, and latency, so item counts, calls, reviews, and wall-clock time should be reported separately.Historical-response and cross-benchmark approaches can have higher construction and update costs but lower cost per new target when reused; close models can make sequential stopping approach full evaluation.Source: §6.3 · p. 8

6.4 Limitations and Method Choice

  • Choose methods according to available information and required results; when assumptions fail, use broader random sampling, test unsupported tasks directly, continue evaluation, or abstain.Adaptive item selection requires acceptable serial rounds and correction for selection, while sequential stopping requires a valid rule and may save little for close models.Source: §6.4 · p. 8

7 Challenges and Future Directions

The section proposes integrating compression into benchmark construction and maintenance while addressing transfer, adoption, and evaluation cost.

Generalization Across Models and Over Time

  • Compressed subsets built from existing-model responses may lose coverage and discrimination on other model families or newer, more capable models.The authors call for testing on newly released models and refreshing subsets when their coverage or discrimination declines.Source: §7 · p. 8

Updating Established Benchmarks

  • Full evaluations remain necessary for leaderboard and prior-work comparisons, so compression mainly saves costs during iteration, screening, and repeated validation.The authors propose combining compression with benchmark repair to remove flawed, redundant, overly easy, or weakly discriminating items.Source: §7 · p. 8

Compression Before Release

  • New benchmarks should be released with compact, informative item sets that cover intended capabilities instead of being compressed after release.The authors state that compression by design can reduce unnecessary computational and environmental costs.Source: §7 · p. 8

8 Conclusion

The survey organizes benchmark compression research and compares its methods across information needs, costs, and reliability. It concludes that useful information at lower cost must be paired with reliable results.

8 Conclusion

  • The survey compares method families by their information needs, costs, and reliability.It groups cost-reduction approaches around representative items, cheaper or reusable information, and evaluation focused on uncertain cases.Source: §8 · p. 8
  • Reliable compression requires sufficient coverage, accurate recovery, and valid adaptive decisions.The survey frames efficient evaluation as collecting useful information at lower cost while preserving reliable results.Source: §8 · p. 8

Limitations

The survey acknowledges limits in coverage, detail, and quantitative comparability, and explains how its catalogue and comparisons organize the literature.

Limitations

  • The survey covers work only through August 2026, summarizes methods at a high level, and cannot quantitatively combine studies with widely different evaluation settings.Appendices E and D provide representative-method descriptions and in-depth family comparisons.Source: Limitations · p. 9

A Paper Catalogue

  • The catalogue maps papers to the taxonomy, supporting paper lookup, corpus counts, and examples of how classification rules handle methods with multiple properties.Appendices E and D define categories and compare their information requirements, costs, and recovery rules.Source: A Paper Catalogue · p. 16

A.1 Reading the Catalogue

  • Each paper receives one primary label for stable counts, while secondary attributes record timing, scope, information, saved unit, and returned result.Labels use method names when available; venue and year follow verified publication records or verified submissions and preprints.Source: §A.1 · p. 16

A.2 Boundary Examples

  • Boundary examples apply ordered rules to papers with several properties, preserving one primary label while keeping other properties visible.The deciding reason follows the paper’s primary contribution.Source: §A.2 · p. 19

B Comparison with Prior Surveys

  • The authors report finding no earlier survey unifying compression within benchmarks, across benchmark suites, and during adaptive evaluation with redundancy and reliability.Their search included survey-focused queries and backward and forward citation tracing through August 9, 2026.Source: B Comparison with Prior Surveys · p. 19

B.1 Adjacent Methodological Surveys

  • Adjacent methodological surveys cover psychometrics, computerized adaptive testing, or model-based judging, whereas this survey connects fixed reduction, adaptive testing, joint benchmark use, cost, and reliability.The related surveys are Ye et al. (2025), Zhuang et al. (2026), and Gu et al. (2026).Source: §B.1 · p. 19

B.2 Broader Evaluation Surveys

The section positions the survey against broader evaluation reviews, defines the taxonomy’s scope and decision rules, and compares compression methods across selection, recovery, cost, and validation.

B.2 Broader Evaluation Surveys

  • Broader evaluation surveys map tasks, benchmarks, protocols, or benchmark quality, while this survey focuses on methods designed to reduce evaluation resources and preserve reliable results.The comparison distinguishes systematic treatment of efficiency and reliability from broad landscape coverage.Source: §B.2 · pp. 19–20, Table 9 · p. 21

C Taxonomy Guide

  • The taxonomy assigns one primary label per paper using a decision path, with secondary attributes recording overlapping methodological properties.The scope requires a predefined full evaluation, fewer costly units, a recovered score or ranking, uncertainty estimate, or reproduced decision, and cost reduction as a central goal.Source: C Taxonomy Guide · p. 20, §C.1 · p. 21, §C.4 · p. 23

C.1 Scope

  • A short benchmark counts as compression only when tested against a predefined full evaluation; changing the measured target is benchmark repair or related analysis.The category rules distinguish compression from work that changes the benchmark’s measurement target.Source: §C.1 · p. 22, §C.2 · p. 22, Table 10 · p. 22

C.3 Decision Path

  • The decision path first separates operational compression from analysis, then distinguishes adaptive methods, joint multi-benchmark methods, and single-benchmark methods by essential information source.Adaptive methods are classified by whether current results alter later tests, budget, or stopping; single-benchmark sources include content, past responses, and cheap current-model signals.Source: §C.3 · pp. 22–23, Table 11 · p. 23

C.5 Boundary Cases

  • Boundary cases are resolved by two authors applying the scope criteria and decision path, assigning one primary label and recording other mechanisms as secondary attributes.Ambiguous assignments are settled through discussion based on each paper’s main contribution and evaluation design.Source: §C.5 · p. 23

D.1 Detailed Problem Formulation

  • Compression starts from a fixed reference evaluation, and its output may be an estimate, ranking, decision, interval, or result allowing abstention.For comparable point results, the stated goal is to keep the difference within ε while using lower cost; these comparison goals are not guarantees shared by every method.Source: D Detailed Comparison · p. 23, §D.1 · pp. 23–24

D.2 Selection and Recovery

  • Selection may be fixed or adaptive, while recovery relies on designs or predictors whose assumptions and transfer determine the validity and precision of estimates.Adaptive methods may reallocate a fixed budget or stop at requested precision; probability weighting needs valid probabilities and support, and residual correction uses observed reference errors.Source: §D.2 · p. 24, §D.3 · pp. 24–25

D.4 Lifecycle Cost and Operational Risk

  • Total savings depend on build, signals, execution, recovery, audits, fallbacks, and reuse, while validation should match the transfer required at deployment.Near ties or weak transfer may trigger further evaluation and erase expected savings; model, family, time, and task holdouts test different transfer scopes.Source: §D.4 · p. 25, §D.5 · pp. 25–26

D.6 Method Choice and Result Checks

Method choice and validation should match the result sought, available information, costs, and deployment setting. The examples distinguish fixed subsets, historical reuse, and target-conditioned signals by their savings, assumptions, validation, and fallback conditions.

D.6 Method Choice and Result Checks

  • Check the claimed result directly: scores need error and bias, rankings need correlation and near-tie reversals, and decisions, intervals, selective estimates, distributions, and quantiles need matching checks.Fair comparisons align reference protocol, output, resources, and budget; compare error at equal cost or cost at equal quality, and report run variation.Source: §D.6 · p. 26

E Detailed Analysis

  • The appendix uses representative, methodologically diverse studies to compare saved costs, assumptions, validation targets, failure conditions, and fallbacks across operational categories.The first six categories compress evaluation; Benchmark Analysis and Foundations assesses compression methods and underlying benchmarks.Source: E Detailed Analysis · pp. 26–27

E.1 Content-Based Compression

  • Fixed content-based subsets recover full results through averaging, design weights, or fitted models, saving model-item runs or human labels but relying on coverage and correction assumptions.Small subsets may be unstable, and item features may miss latent skills; broaden stratification or revert to the full benchmark when repeated samples are unstable or underperform a random baseline.Source: §E.1 · p. 27

E.2 Historical-Response Compression

  • Historical-response methods reuse model-by-item results to select anchors or recover scores, saving target runs but depending on past model patterns transferring to new targets.Hold out complete target models for validation; stale or narrow histories may misrepresent new model families, so add random tests, refresh the matrix, or run the full benchmark when errors are unstable or exceed a random subset.Source: §E.2 · p. 27

E.3 Target-Conditioned Compression

  • Target-conditioned methods use a cheap signal from the current model to choose a fixed expensive batch, saving model runs or human reviews while risking signal mismatch and additional scan costs.Count signal scanning, candidate generation, cheap scoring, and high-quality checks in total cost; the category becomes adaptive only when each new high-quality result can change subsequent tests or stopping.Source: §E.3 · pp. 27–28

E.4 Cross-Benchmark Compression

The survey distinguishes shared cross-benchmark transfer, adaptive item selection, final-decision stopping, and benchmark analysis by their targets and failure conditions. It also documents its literature collection and broader impact.

E.4 Cross-Benchmark Compression

  • Cross-benchmark compression shares selection or transfer across benchmarks, saving item runs, whole tasks, or expensive target runs, but depends on transferable relations and suitable source data.Validate on new models or benchmarks; use the same saved unit and final result for cost comparisons, and run unsupported targets directly when held-out error rises.Source: §E.4 · p. 28

E.5 Adaptive Item Selection

  • Adaptive item selection uses each new target result to choose the next test, but reliable estimates require selection records, coverage checks, and an explicit final-score calculation.Repeated-run instability can arise when a few items receive very large weights; recalibrate, increase uniform exploration, or run the full benchmark.Source: §E.5 · p. 28

E.6 Sequential Decision and Stopping

  • Sequential decision methods allocate tests toward a final choice, interval, escalation, abstention, or stopping rule rather than necessarily recovering every full score.Validate the stated decision rule; near-tied candidates may use almost the full budget, and unmet stopping or agreement criteria call for more testing, abstention, or human review.Source: §E.6 · pp. 28–29

E.7 Benchmark Analysis and Foundations

  • Benchmark analysis diagnoses redundancy, scoring reliability, unresolved comparisons, and benchmark repair, guiding operational compression rather than forming another compression process.A revised benchmark changes what is measured and needs separate validation; narrow model populations or ability ranges can destabilize short forms.Source: §E.7 · p. 29

F Literature Collection

  • The literature search covered publications and public preprints through August 9, 2026, then iteratively expanded seed papers and screened for operational compression or benchmark-analysis evidence.The inclusion criteria required direct LLM evidence and excluded work such as training-data selection, standard inference acceleration, and benchmark construction that did not preserve a predefined full-evaluation result.Source: F Literature Collection · pp. 29–30

G Additional Statement

  • The authors report using AI assistants only for language polishing and literature collection and verification, while presenting the survey as supporting transparent, efficient, and reliable evaluation.The survey’s broader-impact discussion says this guidance can help reduce computational, financial, and environmental costs.Source: G Additional Statement · p. 30

H Survey Contributions

The survey organizes a rapidly growing field into a shared taxonomy and compares its methods, costs, and limitations. It also proposes a research agenda connecting benchmark efficiency with ongoing relevance and reliable comparison.

Survey Contributions

  • The survey synthesizes 119 papers into a taxonomy and decision path spanning benchmark settings and available information, making method selection and validation more comparable.Its lifecycle cost analysis includes constructing, calibrating, and refreshing compression schemes alongside target evaluation costs.Source: H Survey Contributions · pp. 30–31
  • A decision tree assigns primary labels from each paper’s main contribution and evaluation design, while secondary attributes capture finer methodological differences.The survey connects its category roadmap and representative studies to concrete methods, strengths, limitations, and suitable settings.Source: H Survey Contributions · p. 31
  • The family comparisons support realistic efficiency reporting and deployment by accounting for whether savings across repeated evaluations justify construction and maintenance costs.The analysis compares information requirements, score recovery, validation, failure conditions, and fallbacks.Source: H Survey Contributions · p. 31
  • The research agenda links generalization, benchmark updating, and pre-release compression to benchmark design, adoption, and maintenance as model capabilities evolve.It identifies shared priorities for reducing repeated costs, widening participation, and sustaining informative evaluation.Source: H Survey Contributions · p. 31

Related papers

Loading 2610.18248v1…