PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

Kaifeng Tan, Yudong Li, Linlin Shen

arXiv:2609.27510v1cs.CLcs.AI

TL;DR

Benchmark contamination can make evaluation scores less reliable, and task-based assessment can obscure base-model capabilities. Uncheatable Eval uses lossless compression of newly published text to compare base models; compression scales with model size, varies with available context across model families, and is strongly associated with zero-shot MMLU accuracy.

  • Problem

    Benchmark data can enter training corpora and inflate scores through memorization, while task-based evaluation can obscure base-model capabilities because of limited instruction-following ability.

    Source: §1 · p. 2

  • Method

    Uncheatable Eval evaluates base models by converting next-token probabilities on newly published text into code lengths, without task instructions or reference answers.

    Source: §3.2 · p. 4, §5 · p. 8

  • Results

    Across 80 models and 14 text categories, compression rate declines with model size, changes with context differently across model families, and lower technical-text compression is strongly associated with higher zero-shot MMLU accuracy.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Compression of newly published text supports direct comparison of base models across domains.

    Source: §5 · p. 8

    Newly published text cannot be guaranteed absent from training, the protocol evaluates base models rather than post-trained instruction-following abilities, and evaluation data must be refreshed as it ages.

    Source: §6 · p. 9

Abstract

from arXiv · show

Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results. Reliable evaluation is particularly challenging for base models, whose limited instruction-following ability complicates task-based assessment. We introduce Uncheatable Eval, a dynamic benchmark that regularly collects newly published text to evaluate base language models and reduce the risk of data contamination. Drawing on the relationship between a model's predictive ability and its ability to compress data losslessly, we use compression rate to evaluate how well models predict new text. We evaluate 80 models across 14 text categories, study how compression changes with context length, and examine the correlation between compression rate and zero-shot MMLU accuracy. Our results yield three main findings: (1) compression performance follows a consistent scaling trend with model size; (2) attention-based, hybrid, and recurrent models differ in how their compression performance changes as more context becomes available; and (3) lower compression rates are strongly associated with higher zero-shot MMLU accuracy. Code is available at https://github.com/Jellyfish042/uncheatable_eval.

1 Introduction

Benchmark contamination can inflate scores through memorization, while task-based evaluation can obscure base-model capabilities. Uncheatable Eval instead measures prediction of newly published text through compression and evaluates broad model, text-category, context, and MMLU comparisons.

1 Introduction

  • Benchmark contamination can inflate scores through memorization, making genuine generalization difficult to distinguish from recall.Studies find contamination at several semantic levels and show it is difficult to detect reliably.Source: §1 · p. 2
  • Task-based evaluation can obscure base-model capabilities because these models have limited instruction-following ability.Dynamic benchmarks reduce contamination risk by collecting newly published data or generating new examples.Source: §1 · p. 2
  • Uncheatable Eval measures base models’ predictions of newly published text through lossless compression, without task instructions or reference answers.Next-token probabilities guide arithmetic coding, so better prediction requires fewer bits to encode the observed text.Source: §1 · p. 2
  • 80 models across 14 text categories are evaluated, with analyses of model size, document position, and association with zero-shot MMLU accuracy.The benchmark covers diverse newly published text and compares attention-based, hybrid, and recurrent model families.Source: §1 · p. 2

2 Related Work

Related work addresses contamination through detection, screening, exposure controls, and dynamic data, while prediction-based evaluation includes probability and compression measures. Uncheatable Eval extends this line to regularly updated, diverse text and comparisons of base models.

2 Related Work

  • Prior contamination work examines behavioral detection, contamination levels, screening errors, and defenses including varied instances, limited exposure, and resistant datasets.The literature reports that overlap does not affect every model equally and inflated scores need not reorder leaderboards.Source: §2 · p. 2
  • Dynamic benchmarks add recently released questions, programming problems, or knowledge-based examples to reduce contamination risk.Related benchmarks also test use of updated evidence, changing facts, and new knowledge.Source: §2 · p. 2
  • FreshBench studies performance by data age relative to model release, whereas Uncheatable Eval compares base models through compression rate and examines size and document position.FreshBench measures probabilities assigned to texts from different times and accuracy in predicting future events.Source: §2 · pp. 2–3
  • Prediction-based evaluation includes perplexity across domains and compression of text published after model training cutoffs.Prior work also reports that better text compression tends to accompany higher knowledge, code, and mathematics benchmark scores; Uncheatable Eval extends this approach to regularly updated diverse text and model families.Source: §2 · p. 3

3 Uncheatable Eval

Uncheatable Eval collects and curates newly published text, then evaluates models through compression rate. It also measures compression by document byte position to compare how performance changes with more context.

3.1 Data Collection and Curation

  • Newly published fiction, news, encyclopedias, papers, code, and documentation are filtered, deduplicated, normalized, and language-balanced for non-English encyclopedias.Samples are also truncated to a common token budget checked across diverse tokenizers, producing shared text for evaluation.Source: §3.1 · p. 3

3.2 Compression-Based Evaluation

  • Compression rate expresses model-implied ideal code length as a share of original UTF-8 size, with lower rates indicating more efficient prediction.Within a category, scores are byte-weighted; the Overall score gives each category equal weight.Source: §3.2 · p. 4

3.3 Long-Context Evaluation

  • Long-context scoring assigns token loss across its UTF-8 bytes, enabling comparisons at the same byte positions across models with different tokenizers.Source: §3.3 · p. 4

4 Experiments and Results

Across 80 models and 14 text categories, the experiments compare compression performance across domains, context lengths, model sizes, and zero-shot MMLU accuracy. Results show model- and text-type differences in compression and a strong association between lower technical-text compression rates and higher MMLU accuracy.

4.1 Experimental setup

  • The main evaluation covers 80 models across 14 text categories with 500 samples each; long-context evaluation tests 54 models on 500 documents in each of four scientific-paper categories, retaining up to 32,768 UTF-8 bytes.Source: §4.1 · p. 4

4.2 Main results

  • Qwen models lead several scientific-paper categories and remain strong code compressors, while RWKV7-G1J performs particularly well on general text; several MoE models are competitive with dense models of similar total size.Qwen3.5-35B-A3B-Base approaches the table’s best Overall score despite having far fewer active than total parameters.Source: §4.2 · p. 5, Table 2 · p. 5

4.3 Long-context evaluation

  • At 7–8B, Falcon-H1, Qwen3, and RWKV7-G1J have long-context CR slopes of −0.454, −0.426, and −0.314, respectively, with the ordering holding across four categories.Each improves fastest on mathematics and slowest on computer science; longer-context-trained RWKV checkpoints show smaller relative CR-gap increases than older checkpoints.Source: §4.3 · p. 6, Figure 2 · p. 6, Table 3 · p. 6

4.4 Scaling Laws for Compression Rate

  • For aggregate CR, the power-law-plus-constant fit explains 91.5% of cross-model variation, and its fitted curve flattens as model size grows.Across text types, exponents range from −0.317 for code to −0.146 for English fiction; category-level R2 ranges from 0.594 to 0.935, and frontier fits may shift as models are added.Source: §4.4 · p. 7, Figure 3 · p. 7, Table 4 · p. 7

4.5 Correlation with MMLU

  • Across 80 models, lower technical-text CR is associated with higher zero-shot MMLU accuracy (Spearman ρ = −0.884, 95% CI [−0.942, −0.783]).The pattern also appears for CR pooled across all 14 categories and in each of the nine technical categories.Source: §4.5 · p. 8, Figure 4 · p. 8

4.6 Correlation analyses across text types

  • All five text types have positive raw CR correlations; residual CR correlations are strongest for Scientific Paper–Code (ρ = 0.838) and News–English fiction (ρ = 0.812), while Code–News is near zero.Residuals are computed after fitting each type’s CR against Overall CR.Source: §4.6 · p. 8

5 Conclusion

Uncheatable Eval compares base models by compressing newly published text, with results spanning model size, context length, text type, and zero-shot MMLU association.

  • Lower compression rates on technical text are strongly associated with higher zero-shot MMLU accuracy.Source: §5 · p. 8
  • Across 80 models and 14 text categories, compression rate declines with model size and varies across text types.Source: §5 · p. 8
  • Attention-based, hybrid, and recurrent models differ in how their compression rates change as more context becomes available.Source: §5 · p. 8

6 Limitations

Newly published evaluation text reduces contamination risk but does not guarantee that evaluated models have never encountered the content.

  • Incomplete publication dates and training cutoffs, along with copies of the same content, prevent guaranteeing that evaluation text was absent from training.Source: §6 · p. 9
  • Raw-text prediction limits compression-rate evaluation to base models because it does not capture instruction-following abilities acquired through post-training.Source: §6 · p. 9
  • As public evaluation texts age, new models may train on them, requiring regular dataset refreshes.Source: §6 · p. 9

A Complete Model-by-Category Results

The tables report category-level compression rates for the 80-model analysis cohort, organized by parameter count and summarized with an Overall score.

B Long-Context Results across Model Sizes

Figures 5 report long-context compression rates across model-size groups, showing absolute rates and differences from the Qwen3 model in each group, pooled over four scientific-paper categories.

  • For the first three size groups, Figure 5 covers 33 models and shows absolute compression rates alongside differences from the Qwen3 model in each group.Source: Figure 5 · p. 13
  • Both views pool results across four scientific-paper categories.Source: Figure 5 · pp. 13–14
  • For the last two size groups, Figure 5 covers 21 models with the same absolute-rate and within-group Qwen3-difference views.Source: Figure 5 · p. 14

C Complete Category-Level Scaling Results

This section presents category-level scaling curves for all 80 models and power-law-plus-constant fits.

C Complete Category-Level Scaling Results

D Pareto-Frontier Scaling Results

Pareto-frontier fits show an all-category scaling relationship and category-specific exponent ranges, with Figure 7 distinguishing frontier models from others.

D Pareto-Frontier Scaling Results

E Tokenization Statistics

The section reports native-token segmentation diagnostics for the 80-model analysis cohort, including a residual correlation measure.

E Tokenization Statistics

  • Table 8 presents native-token segmentation diagnostics for the 80-model analysis cohort.Source: Table 8 · p. 17
  • Residual ρ is the Spearman correlation between CR and bytes-per-token residuals after each is regressed on log10 parameter count with an intercept.Source: Table 8 · p. 17
  • The diagnostics concern native-token segmentation in the 80-model analysis cohort.Source: Table 8 · p. 17
Loading 2609.27510v1…