Source-linked AI summary
LeakScale: Estimating the Causal Effect of Benchmark Exposure
Divyansh Singh
TL;DR
Benchmark contact does not show how much it affected a score, leaving the performance attributable to exposure difficult to quantify. LeakScale controls access to private information in fresh executable tasks and estimates its control-adjusted effect; accuracy rises in every model-by-domain cell, with gains from +7.17 to +27.31 percentage points.
Problem
Provenance evidence can establish benchmark contact, but does not quantify how different a score would have been without access to benchmark-specific information.
Source: §1 · p. 1Method
Results
Exposure increased accuracy in all four model-by-domain cells, with effects ranging from +7.17 to +27.31 percentage points.
Source: §1 · p. 2Takeaways & Limitations
Benchmark interpretation requires characterizing both whether contact occurred and how strongly a score depends on it, which can be monitored across a benchmark’s lifecycle.
Source: §7 · p. 6, §8 · p. 7The study estimates a controlled post-training exposure regimen, not naturally occurring web-scale contamination or unrestricted program generation.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Evidence that evaluation material entered training does not reveal how much it affected evaluation. This distinction leaves a contaminated benchmark score difficult to interpret: provenance can establish contact, but only a counterfactual can quantify the performance attributable to that contact. We present LeakScale, an interventional framework for estimating this missing quantity. LeakScale creates fresh executable tasks that require private, family-specific information absent from and non-derivable from the public task, controls access to that information, and estimates the resulting control-adjusted change in executable accuracy. Across 2,048 unique families, two model families, two executable domains, and 262,144 generations, exposure improves accuracy in every model-by-domain combination, with gains ranging from +7.17 to +27.31 percentage points. These findings separate two empirical questions that are often conflated: whether benchmark contact occurred and how strongly a reported score depends on it. LeakScale makes the latter directly measurable.
1 INTRODUCTION
Benchmark contact can make scores difficult to interpret, while evidence of contact alone does not quantify its effect. LeakScale controls access to private task information and finds exposure-attributable accuracy gains in all four model-by-domain cells.
1 INTRODUCTION
- Benchmark scores shape rankings, research priorities, and deployment decisions, so contamination makes their intended interpretation uncertain (Brown et al., 2020; Touvron et al., 2023; Zhou et al., 2023).The premise is that success should reflect the capability a benchmark measures, not privileged access to benchmark-specific information.Source: §1 · p. 1
- Provenance audits ask whether contact occurred, but the consequential question is how different a score would be without that access (Shi et al., 2023; Zhang et al., 2024).Retrospective detection can indicate risk but generally cannot isolate the performance change produced by exposure.Source: §1 · p. 1
- LeakScale intervenes on access to private information required for correct behavior but not recoverable from the public task, then estimates the change against a balanced no-key group before and after adaptation.This separates exposure-attributable change from contemporaneous adaptation drift.Source: §1 · pp. 1–2
- Exposure increased accuracy in all four model-by-domain cells, with effects ranging from +7.17 to +27.31 percentage points.The study covers 2,048 fresh families, two model families, and two executable domains.Source: §1 · p. 2
- The paper contributes an exposure-consequence estimand, an executable identification strategy, and a crossed study showing positive effects across its four model-by-domain evaluations.The strategy combines necessary, non-derivable family-specific information, executable counterfactuals, and a balanced no-key comparison.Source: §1 · p. 2
2 MEASUREMENT TARGET
The measurement target is the change in benchmark accuracy attributable to controlled access to benchmark-specific information, rather than the probability that contact occurred. A two-group, two-state contrast subtracts adaptation-wide movement in the no-key control from the change under targeted exposure.
2 MEASUREMENT TARGET
- The target is the change in benchmark accuracy attributable to controlled access, measured against a contemporaneous group undergoing the same adaptation without family-specific information.Exposure is manipulated rather than inferred from evidence of contact.Source: §2 · p. 2
- Each family pairs a public task and opaque key with four candidates, while a private association designates the correct candidate for executable evaluation.The model selects a candidate, a deterministic renderer realizes it as SQL or Python, and an executable oracle grades behavior on hidden fixtures.Source: §2 · p. 2
- The two-group, two-state difference-in-differences contrast separates targeted acquisition from general adaptation drift by comparing before-to-after changes across C2 and C0.C0 is the no-key control and C2 is targeted exposure.Source: §2 · p. 2, Figure 2 · p. 3
3 EXECUTABLE EXPOSURE INSTRUMENT
LeakScale identifies exposure effects by making private family-specific information necessary for correct executable behavior and varying access to that information. It certifies candidate behavior, then balances targeted-exposure and no-key families for comparison.
3 EXECUTABLE EXPOSURE INSTRUMENT
- LeakScale pairs fresh executable tasks with necessary, non-derivable private associations, certifies every candidate's behavior, and varies access to the association.The design compares targeted exposure with a balanced no-key group.Source: §3 · p. 3
3.1 PRIVATE INFORMATION WITH AN EXECUTABLE CONSEQUENCE
- The key-to-value association is absent from public task materials and disjoint background data, so only the private registry specifies which candidate behavior is required.Public information determines what each candidate would do, but not which behavior is required.Source: §3.1 · p. 3
3.2 CERTIFICATION OVER ALL COUNTERFACTUAL CHOICES
- Every candidate is executed on hidden fixtures before evaluation; exactly one passes the full set, while each alternative fails at least one quarter.This family-level check establishes necessity and behavioral grading rather than relying on surface-form agreement.Source: §3.2 · p. 3
3.3 CONTROLLED ACCESS AND BALANCED NO-KEY COMPARISON
- Within each domain-by-shape cell, construction assigns 16 C0 and 16 C2 families and balances truth rank and displayed position; no training record comes from a C0 family.The crossed design shares domain families across models and measures each family in base and adapted states using 32 generations per state; audits found zero C0 keys and reference artifacts.Source: §3.3 · pp. 3–4, Table 1 · p. 4
4 CROSSED EVALUATION STUDY
The study crosses two model families with SQLite and deterministic Python tasks built from 2,048 families, using a fixed adaptation and evaluation protocol. Its inferences are limited to a controlled post-training regimen, the two model families and domains, and uncertainty that does not include training-seed variation.
4.1 MODELS, DOMAINS, AND TASK FAMILIES
- The study crosses Qwen3-4B-Instruct-2507 and Gemma 4 31B with SQLite and deterministic Python transformations across 2,048 families and 64 shapes.Each domain has 1,024 families across 32 structural shapes.Source: §4.1 · p. 4
4.2 ADAPTATION AND EVALUATION PROTOCOL
- Each condition-by-state cell receives 32 independently seeded draws per family under a fixed one-epoch BF16 LoRA adaptation protocol.Draws are scored using each selected artifact’s frozen certification outcome.Source: §4.2 · p. 4
4.3 SCOPE OF INFERENCE
- The study estimates a controlled post-training exposure effect, but its two model families and two domains do not establish universality.Each cell has one adapter training realization, and the confidence intervals do not capture training-seed variation.Source: §4.3 · p. 4
5 EMPIRICAL FINDINGS
Controlled access raises accuracy in all four model-by-domain cells, with effects spanning +7.17 to +27.31 percentage points. The control trajectory matters: adaptation changes the no-key group differently from the exposed group, and design-integrity checks passed.
5.1 EXPOSURE RAISES ACCURACY IN ALL FOUR CROSSED CELLS
- Exposure effects are positive in all four model-by-domain cells, spanning +7.17 to +27.31 percentage points; all four 95% confidence intervals exclude zero.Figure 3 displays the difference-in-differences estimates and their 95% confidence intervals.Source: §5.1 · p. 4, Figure 3 · p. 5, Table 2 · p. 5
5.2 THE CONTROL TRAJECTORY CHANGES THE CONCLUSION
- C0 and C2 begin at 25.52% and 25.12% aggregate accuracy, then change by −2.34 and +12.05 points on average, respectively.A treatment-only before-to-after comparison would confound targeted acquisition with adaptation-wide drift.Source: §5.2 · p. 5
5.3 COMPLETION AND DESIGN-INTEGRITY CHECKS
- All 262,144 planned generations completed without protocol failure, and the prespecified balance, leakage, and artifact-integrity audits passed.The complete ledgers and immutable identities are provided in Appendices E and F.Source: §5.3 · p. 5
6 RELATION TO EXISTING EVALUATION APPROACHES
Prior work audits whether benchmark contact occurred and studies controlled score effects or methods that reduce exposure risk. LeakScale targets the complementary causal quantity by controlling access and measuring its effect on executable accuracy.
6.1 AUDITING PROVENANCE AND DETECTING PRIOR CONTACT
- Provenance audits use corpus matching, reconstruction, likelihood, and ordering probes, but their signals vary with fine-tuning, scale, duplication, context, and post-training.These methods estimate evidence of contact; LeakScale instead controls exposure and measures its effect on executable accuracy.Source: §6.1 · pp. 5–6
6.2 CONTROLLED EXPOSURE, FRESH TASKS, AND MITIGATION
- Controlled studies vary exposure conditions or calibrate observational measures, while LeakScale combines non-derivable family information, hidden-execution certification, and balanced pre/post no-key comparisons.Fresh and dynamic evaluations reduce exposure opportunities rather than estimate exposure effects; mitigation methods also address contamination risk from different directions.Source: §6.2 · p. 6
7 IMPLICATIONS FOR BENCHMARK EVALUATION
Benchmark scores depend on training history and the exposure path, so exposure effects are specific to a model and benchmark system. The findings support treating contact and its performance consequence as distinct questions and monitoring benchmark freshness over time.
7 IMPLICATIONS FOR BENCHMARK EVALUATION
- A benchmark score also depends on the exposure path and scoring interface, so the measured exposure response belongs to a particular model and benchmark system.The four cells move in the same direction but by different amounts, making a universal penalty for anything labeled contaminated unsuitable.Source: §7 · p. 6
- Combining provenance evidence with sensitivity experiments could prioritize detectors by associated effects and judge mitigation by removed exposure dependence.Contact and consequence support different decisions: one assesses plausible contact, the other whether contact can move performance enough to change score interpretation.Source: §7 · p. 6
- Benchmark developers can preserve private counterfactual variants and repeat exposure-sensitivity estimates after important model or benchmark updates.Freshness is temporary as tasks, annotations, solutions, and derivatives circulate.Source: §7 · p. 6
8 CONCLUSION
LeakScale experimentally controls benchmark exposure to measure the score change attributable to it. Its conclusion separates evidence of contact from how strongly scores depend on that contact.
8 CONCLUSION
- LeakScale identifies positive exposure effects in every cell of a two-model, two-domain study.The study uses necessary private associations, certified executable counterfactuals, and a balanced no-key trajectory.Source: §8 · p. 6
- LeakScale treats benchmark exposure as an experimentally controlled input and measures its attributable score change.Source: §8 · p. 6
- The result distinguishes whether benchmark contact occurred from how strongly reported scores depend on it.The authors argue credible benchmark interpretation requires characterizing both.Source: §8 · pp. 6–7
AI USE STATEMENT
Generative AI tools were used across ideation, research, development, analysis, and manuscript preparation. The authors retained scientific responsibility and checked empirical claims against artifacts and tests.
AI USE STATEMENT
- Generative AI tools assisted with ideation, methodological design, benchmark and software development, experiments, analysis, literature review, writing, and typesetting.Source: AI USE STATEMENT · p. 7
- Scientific choices remained with the authors, who inspected result-producing code and logs and checked quantitative claims against immutable artifacts.Source: AI USE STATEMENT · p. 7
- The authors reran relevant tests and build checks and take responsibility for the manuscript.Model-generated text was not accepted as empirical evidence without deterministic or artifact-backed verification.Source: AI USE STATEMENT · p. 7
A EXECUTABLE TASK CATALOG AND FAMILY CONSTRUCTION
This section describes LeakScale’s construction of executable task families around private policy-key associations and its controls for studying training access while keeping those associations private. It documents training and artifact controls, family-level uncertainty estimation, and the distribution of accuracy outcomes across 2,048 latent families.
A EXECUTABLE TASK CATALOG AND FAMILY CONSTRUCTION
- Each family pairs a program skeleton with a private key-to-value association and appears in three forms: exact-public R0, semantics-preserving R1, and isomorphic R2.Each domain has 32 program structures; the experiment uses 2,048 families and evaluates R0 for every family.Source: A EXECUTABLE TASK CATALOG AND FAMILY… · p. 9
A.1 SQL PROGRAM FAMILIES
- The SQLite catalog spans 32 query structures, with family-specific opaque handles masking the literals and R2 renaming tables and columns while retaining program structure.The inventory covers filtering, aggregation, joins, subqueries, ranking, windows, set operations, and other relational patterns; queries specify aliases and deterministic ordering.Source: §A.1 · pp. 9, 11
A.2 PYTHON TRANSFORMATION FAMILIES
- The Python catalog spans 32 transformation structures, exposing four candidate literals and supporting filters, aggregates, ranking, grouping, nested data, and arithmetic operations.R2 applies a fixed input-field renaming and updates the certified function accordingly, changing the public API but not behavior.Source: §A.2 · p. 11, §C.3 · p. 14
B.1 EVALUATION-FACING CHOICE PROMPT
- Evaluation prompts request one registered candidate without exposing the private association, hidden fixtures, expected outputs, reference artifacts, or condition labels.SQL presents opaque handles while Python presents literals; a strict family-specific JSON schema accepts only one of the four candidates with its original type and no extra fields.Source: §B.1 · p. 12, §B.2 · p. 13
B.3 DETERMINISTIC COMPILATION AND SCORING
- The trusted compiler deterministically maps a model’s selected candidate to its pre-certified artifact and fixture utility, assigning malformed or out-of-enum responses utility zero.Artifacts are materialized and certified before inference, so evaluation uses a frozen lookup rather than model-conditioned program synthesis or repeated fixture execution.Source: §B.3 · p. 13
C.1 DOMAIN-INDEPENDENT ACCEPTANCE CRITERION
- A family is released only when its registered candidate is fully correct and every alternative fails on at least one quarter of hidden fixtures.SQL candidates are checked against canonical reference relations across eight fixtures; Python utility is the proportion of 16 fixture sets with exact canonical agreement.Source: §C.1 · p. 13, §C.2 · p. 14, §C.3 · p. 14
C.4 PRE-RELEASE SOURCE INVARIANTS
- Pre-release tests verify distinct shapes and candidates, one fully correct choice, bounded incorrect-candidate utility, deterministic regeneration, balanced assignments, and no private truth in prompts.They also check compiler recovery, output-schema enforcement, executable references, and zero normalized content overlap with earlier benchmark sources.Source: §C.4 · p. 15
D.1 TARGET ASSOCIATIONS AND THE NO-KEY CONTROL
- Adaptation trains on paired exact-public and direct-lookup records for C2 families, while excluding all C0 keys, assignments, labels, and artifacts from training.Each C2 family contributes two physical target records repeated 32 times; disjoint public-format preservation tasks and unrelated background supervision supplement them.Source: §D.1 · p. 15, §D.2 · p. 16
D.2 BACKGROUND SUPERVISION AND BEHAVIOR PRESERVATION
- Token counts are computed after applying each model’s chat template and tokenizer, with target-answer length distinguished from the full supervised assistant suffix.Prompt and padding tokens are excluded from loss, and records are accepted only after assistant-boundary token alignment succeeds.Source: §D.2 · p. 16
D.3 OPTIMIZATION SETTINGS AND ADAPTER CONSTRUCTION
- Training uses one epoch of unquantized BF16 base weights, fused AdamW, learning rate 10^-4, and rank-32 LoRA adapters, with reported strengths selected using disjoint development families.LoRA-B tensors are scaled while LoRA-A, alpha, and base weights remain fixed; runs bind completion records to model, corpus, configuration, and diagnostic artifacts, and reuse requires exact hash agreement.Source: §D.3 · p. 17
D.4 INDEPENDENT CONFIRMATION OF ADAPTER ACTIVITY
- Every selected adapter passes an independent activity probe requiring an absolute C2 target-token log-probability shift greater than 10^-7.The probe compares adapter-enabled and disabled forward passes and recomputes base-model and adapter hashes.Source: §D.4 · p. 17
D.5 ARTIFACT IDENTITIES FOR MODELS, TOKENIZERS, CORPORA, AND ADAPTERS
- The reported experiments bind model, tokenizer, corpus, and selected-adapter artifacts to recorded identities for Qwen3-4B-Instruct-2507 and Gemma 4 31B.Table 9 reports SHA-256 bindings for base-model trees, serialized training corpora, and selected inference adapters.Source: §D.5 · p. 18, Table 9 · p. 18
E STATISTICAL DETAILS, ROBUSTNESS SUMMARIES, AND EXECUTION AUDITS
- Each condition-by-domain cell contains 512 of the 2,048 latent families, with 32 independently seeded generations per family and model state.Displayed percentage-point values are rounded, but calculations use individual binary generation outcomes.Source: E STATISTICAL DETAILS, ROBUSTNESS… · p. 18
E.1 LATENT-FAMILY ESTIMATION AND UNCERTAINTY
- A 10,000-replicate percentile bootstrap estimates 95% intervals by resampling C0 and C2 families independently and carrying each sampled family’s outcomes as a block.Each cell has 512 families; a sampled family contributes all 32 base and 32 adapted outcomes to the replicate.Source: §E.1 · p. 19, Table 10 · p. 19
E.2 INTEGER OUTCOMES AND FAMILY ACCURACY PROFILES
- Integer correctness totals use a denominator of 16,384 per entry and recover the component percentages without intermediate rounding.The totals are shown in Table 11.Source: §E.2 · p. 19, Table 11 · p. 19
- Exposed-family adaptation shifts the accuracy distribution’s location and upper tail, while several base cells have a median of zero despite pooled means near chance.The quartiles treat 512 families—not individual generations—as observations.Source: §E.2 · p. 19
E.3 ROBUSTNESS WITHIN THE BALANCED ANSWER STRATA
The robustness analyses find positive exposure effects across balanced answer strata and describe how effects vary by structural shape and across the crossed design. They also report protocol validity and cryptographic identities for the analyzed streams.
E.3 ROBUSTNESS WITHIN THE BALANCED ANSWER STRATA
- All 16 truth-rank and 16 display-position marginal DiD estimates are positive, so no single rank or position accounts for the four-cell finding.These are descriptive robustness summaries without additional confidence intervals or hypothesis tests.Source: §E.3 · p. 19, Table 13 · p. 20
E.4 EXPOSURE EFFECTS BY STRUCTURAL SHAPE
- Shape-specific marginal effects vary by model and task, and are diagnostic descriptions rather than separately tested effects.The observed spread does not support a single correction applicable everywhere.Source: §E.4 · pp. 20–21, Table 14 · p. 20, Table 15 · p. 21
- All 128 leave-one-shape-out effects remain positive, with each cell’s original estimate inside a narrow recomputed range.This descriptive diagnostic does not replace the family-clustered intervals.Source: §E.4 · p. 22, Table 16 · p. 22
E.5 SUMMARIES ACROSS THE CROSSED DESIGN
- The crossed-design summaries average each model over its two domains, each domain over its two models, and weight all four cells equally in the grand mean.The interaction is the difference between the models’ Python-versus-SQL marginal-effect contrasts.Source: §E.5 · p. 22, Table 17 · p. 22
E.6 BANK COMPLETION AND RESPONSE VALIDITY
- All 262,144 terminal results were schema-valid, with zero protocol failures in every bank and condition.The zero failure rates rule out protocol failure as the source of state changes or the C2-minus-C0 contrast.Source: §E.6 · p. 22, Table 18 · p. 22
E.7 CRYPTOGRAPHIC IDENTITIES FOR RESULTS AND ATTEMPTS
- SHA-256 digests bind each analyzed result stream to its append-only attempt stream, with each recorded digest split for typesetting and recovered by concatenation.The table summarizes the result and attempt stream identities for each bank.Source: §E.7 · p. 23, Table 19 · p. 23
F ARTIFACT IDENTITIES AND REPRODUCIBILITY LEDGER
The reproducibility ledger binds tasks, assignments, attempts, and results to immutable identities and records deterministic seeds. It also specifies regeneration checks, artifact organization, and an ordered verification procedure.
F.1 IMMUTABLE IDENTIFIERS AND APPEND-ONLY PROVENANCE
- Immutable identifiers link families, tasks, requests, attempts, compiled candidates, and terminal results, whose records preserve the data and provenance needed to reconstruct each result chain.Task, assignment, attempt, and terminal-result records each store distinct identifying and execution details.Source: §F.1 · p. 23
- Append-only attempt storage prevents silent overwrites; completion requires exactly one terminal result per frozen case and a valid attempt chain.Retries are limited to transport failures or a specified length-tier transition; malformed completed responses remain terminal failures.Source: §F.1 · p. 24
F.2 DETERMINISTIC SEEDS
- Inference seeds are deterministic hashes of bank seed, family, domain, ring, and sample index, making them invariant to execution concurrency and resume ordering.The root seeds for source construction and analysis are listed in the seed ledger.Source: §F.2 · p. 24, Table 20 · p. 24
F.3 SOURCE REGENERATION AND MANIFEST DIGESTS
- Independent source regeneration matched both source trees byte-for-byte under SHA-256, alongside recorded identities for task, assignment, and four-cell path-manifest data.The table reports SHA-256 identities for source construction and analysis paths.Source: §F.3 · p. 24, Table 21 · p. 24
F.4 ARTIFACT LAYERS AND VERIFICATION SEQUENCE
- The artifact tree separates source, training, banks, analysis, and audits into five immutable layers, with a recursive SHA-256 manifest proposed for binding every byte.Verification proceeds from manifest and regeneration checks through balance, leakage, inference, completion, and reproduction of the reported analysis.Source: §F.4 · p. 25