PaperlayerView exact revision on arXiv

Source-linked AI summary

Recipe-Matching, Not Equivalence

Ali Habibullah, Mohammad Alshiekh, Yazan Alshoibi, Salman Khan, Naeemullah Khan

arXiv:2609.31927v1cs.IRcs.CLcs.LG

TL;DR

MathNet-Retrieve tests whether retrievers find equivalent math problems across different wording, but its LLM-written golds may reward learning the benchmark’s generation procedure. The paper compares matched training data built with that procedure or verified without an LLM, then tests benchmark tiers and real duplicates. LLM-written data explains much of the easy-tier gain, while the benchmark-specific remainder does not transfer to real duplicates.

  • Problem

    When a benchmark’s gold documents are produced by one fixed LLM procedure, it is unclear how much retrieval scores reflect the claimed skill versus matching that procedure.

    Source: §1 · p. 2

  • Method

    The authors compare matched training arms using benchmark-prompt LLM pairs or computer-algebra-verified pairs, then evaluate benchmark tiers and duplicates no LLM wrote.

    Source: §1 · p. 2

  • Results

    Half to two thirds of the easy-tier lead comes from LLM-written pairs generally, while the remaining recipe-specific gain vanishes on real duplicates no generator wrote.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    On MathNet-Retrieve, benchmark score and real retrieval can move in opposite directions after a single training-file edit.

    Source: §1 · p. 2, §8 · p. 9

    The three attacked LLM-written surfaces were each tested on only one benchmark, limiting how generally their results can be interpreted.

    Source: §7 · p. 8

Abstract

from arXiv · show

MathNet-Retrieve asks a retriever to find, for a math problem, a document stating the same problem. An LLM under one fixed prompt writes each gold document and its near-miss distractors; LLM judges filter them. We call this procedure the "recipe", training on pairs built the same way "recipe-matching", and ask how much score it buys beyond the ability the benchmark claims to test. Two models from one base, matched in rows and settings, differ only in the training file: pairs written under the benchmark's published prompt by another vendor's LLM and judge, or computer-algebra-verified pairs with no LLM anywhere. The first leads by 45 R@1 points on the easy tier. By a non-LLM paraphrase control, half to two thirds of that gap comes from the pairs being LLM-written at all: LLM rewrites under two unrelated prompts, with the verified model's negatives, recover 30 and 22 of the 45 points; back-translations with the same negatives recover almost none. The remaining 15 to 25 points appear only under the benchmark's own prompt and vanish on real duplicates no generator wrote, the same problem in two languages. The hard tier rewards the recipe's pair structure, a deep rewrite against a minimal-edit near-miss: LLM rewrites alone score zero on it, attaching negatives unlocks it, and every negative that does so costs cross-language points; the sets scoring highest on it separate near-misses no LLM wrote worse than LLM rewrites with verified negatives. MELD also moves when a model trains on pairs built its way, without losing retention; on SABER-Math the registered attack fails, and the one gain, from its LLM-written summaries, is small but holds at a matched budget. Only on MathNet-Retrieve could we pin an inversion, benchmark score up and real retention down, to one edit of a training file. We release the generator-free duplicate evaluations, the near-miss test and three trained models.

1 Introduction

The paper studies whether math-retrieval benchmarks reward matching the procedure that generated their gold documents, rather than only the retrieval skill they claim to measure. On MathNet-Retrieve, training data built with the benchmark’s recipe boosts benchmark scores, but much of the gain does not transfer to real duplicates.

1 Introduction

  • MathNet-Retrieve’s three tiers share queries and corpus but vary how heavily the gold document is disguised, with each gold paired against three minimal-edit near-misses.The hard tier is framed as deep mathematical equivalence, but existing models score almost zero at rank 1 because surface wording favors the near-misses.Source: §1 · p. 2
  • The controlled comparison uses two 6,145-row Qwen3-Embedding-0.6B training sets matched in source problems and settings, differing only in their training files.One uses computer-algebra-verified pairs; the other follows the benchmark prompt and LLM judging, with contamination matched across arms.Source: §1 · p. 2
  • 45.33 easy-tier R@1 points separate the recipe arm from the verified arm, and the recipe arm reaches 9.42 hard-tier R@1 while the verified arm scores zero.LLM rewrites under unrelated prompts recover 29.85 and 22.15 easy-tier points; back-translated paraphrases recover 1.76 with the same verified negatives.Source: §1 · p. 2
  • The recipe arm’s lead over the verified arm shrinks to under ten points on cross-language duplicates and disappears on same-language reprints.The benchmark-to-real-data drop is 35.59 points, of which 27.24 is recipe-specific.Source: §1 · p. 2
  • Across MELD and SABER-Math, the effects differ: MELD improves without worsening cross-language duplicates, while SABER-Math’s registered attack fails and its summary-based gain is small.The summary-based SABER-Math gain is robust to a matched budget.Source: §1 · p. 2
  • The paper releases generator-free duplicate evaluations, near-miss tests, and three trained models.It also reports that one training-file edit can raise MathNet-Retrieve scores while lowering real retrieval.Source: §1 · p. 2

2 Related Work

The related-work discussion places recipe-matching alongside benchmark contamination and shortcut learning while distinguishing it from item leakage and generator self-recognition. It argues that a benchmark-wide writing procedure can create a validity problem that item-level fixes do not address.

Math and reasoning retrieval

  • The paper situates MathNet-Retrieve among reasoning-retrieval benchmarks and models, noting that tested RaDeR, ReasonIR, and MathLeap baselines remain below 0.01 hard-tier R@1.Related benchmarks include SABER-Math [3], BRIGHT [1], and MELD [4].Source: §2 · p. 3

The blind spot of item-level contamination detection

  • Item-level contamination checks miss recipe-matching because training rewords public-corpus problems in a benchmark-wide style that can be regenerated without leaking exact test items.The paper reports that leakage explains 1.24 of the 45 points.Source: §2 · p. 3

Shortcut learning at the procedure level

  • Recipe-matching resembles shortcut learning, but its signal is the benchmark’s writing procedure rather than an accidental pattern in a particular item.The paper distinguishes it from LLM judges favoring their own outputs because the training generator differs in vendor and model family from the benchmark generator.Source: §2 · p. 3

3 Experimental Setup

The experiments compare matched training arms, benchmark performance, and duplicates that no LLM wrote. The setup also checks contamination and accounts for differences in training signal and evaluation uncertainty.

Terms and design

  • The evaluation tests benchmark tiers, real cross-language and same-language duplicates, and an n-gram probe of whether similarity scores track generator style.The recipe, verified, D4, and reference training sets differ in their positive and negative construction, with D4 using unrelated-prompt LLM restatements and no negatives.Source: §3 · p. 3

Task and metric

  • The training objective is an in-batch contrastive loss in which each hard negative becomes its own training example.Retrievers rank documents by cosine similarity, and R@k is the share of queries whose gold document appears in the top k.Source: §3 · p. 3

Benchmark and harness

  • The benchmark has 117,088 corpus documents and 15,000 shared queries across three tiers, which differ in the gold rewrite; reported metrics are R@1, R@5, and R@10.The reimplementation reproduces the published easy-tier result for all-mpnet-base-v2, though the best Qwen3-Embedding-4B setting remains 1.57 points below the authors’ reported result after a sweep.Source: §3 · p. 3

Two supervision arms at a matched budget

  • The recipe and verified arms each use 6,145 rows from the same 7,089 source problems, with shared hyperparameters and eight seeds.The recipe uses LLM paraphrases and judging; the verified arm uses symbolic equivalence edits and counterexample-refuted minimal edits.Source: §3 · p. 4

Matched training signal

  • Matching row counts did not match training signal because the recipe arm had 1.7× as many negatives per row, but a verified arm matched on negatives scored within noise of the original verified arm.An uncapped verified arm with more rows, sources, and examples did no better.Source: §3 · p. 4

Contamination

  • 1.24 ± 0.08 of the 45.33-point gap is attributed to leakage, and corrected-filter retrainings still score 43.11 and 44.45.Both arms had about 33% of rows anchored on benchmark queries, and the gap persisted on clean query slices.Source: §3 · p. 4

4 Recipe-Matching Games the Benchmark

On MathNet-Retrieve, the benchmark’s recipe yields gains that split between LLM-written text generally and the benchmark’s own prompt, while real-duplicate performance does not show the same lead. The hard tier instead rewards a particular positive-negative pair structure.

4 Recipe-Matching Games the Benchmark

  • The recipe arm leads on every tier, but the medium-tier lead is no larger on the benchmark than on real duplicates, and the hard-tier comparison is read against the reference because the verified arm is near zero.Table 1 reports benchmark tiers, real duplicates, and the difference between benchmark and real-duplicate leads.Source: §4 · p. 4, Table 1 · p. 5
  • The verified arm wins on computer-algebra-built pairs, LLM-rewrite arms win on LLM-built pairs, and the untrained base wins on duplicates no generator wrote.The result varies with how the test pairs were built, rather than showing one training file winning across test types.Source: §4 · p. 4
  • 29.85 of the 45.33 easy-tier points are recovered by unrelated-prompt LLM restatements with verified negatives, while back-translation recovers 1.76.A second unrelated prompt recovers 22.15; the reported recipe-specific share is 15.47 to 23.18 points in this comparison.Source: §4 · p. 5
  • 15.47 to 24.59 easy-tier points are attributed to the benchmark’s own prompt across comparisons, but the recipe arm trails the reference by 11.77 on cross-language duplicates.The recipe arm’s lead shrinks by 27.24 points against the reference between benchmark and real-data performance; on same-language reprints, it and the reference are within noise.Source: §4 · p. 5, Table 1 · p. 5
  • The recipe arm’s lead transfers across generators and model scales, but the ordering reverses on cross-language duplicates at 4B: 70.99 versus 83.72, below the untrained base’s 91.60.At 4B, easy-tier scores are 77.77 for the recipe arm and 39.51 for the verified arm.Source: §4 · p. 5

5 What the Recipe Adds, and the Fingerprint It Leaves

The easy tier rewards LLM-rewrite style across prompts, while the hard tier rewards deep rewrites paired with near-misses. Recipe-prompt models also track an LLM-style fingerprint, but that style-tracking does not explain the hard-tier gains.

What the Recipe Adds

  • Hard-tier R@1 falls from 9.90 to 0.04 as prompts move from D1 to D4, but deep D4 rewrites with negatives reach 10.30 with verified negatives and up to 26.97 with recipe-prompt negatives.Neither positives nor negatives alone suffice; changing only D4’s negatives raises hard-tier score, with recipe-prompt near-misses adding 3.75–6.03 over unrelated-prompt near-misses.Source: §5 · p. 6
  • The easy tier rewards LLM-style rewrites rather than a particular prompt: D2 and D3 score five to six points above the recipe arm, while D4 reaches the reference with verified negatives.D2 and D3 are indistinguishable across seeds, contrary to the predicted stepwise decline as prompts diverged from the benchmark.Source: §5 · p. 6

Real-data trade-off

  • D4 scores 93.81 cross-language R@1 against the base’s 84.73, but adding negatives raises hard-tier performance while costing 8.31–15.79 cross-language points.The recipe arm trails the reference by 11.77 on cross-language duplicates despite leading it on the easy tier.Source: §5 · pp. 6–7, Figure 1 · p. 6

Fingerprint

  • The n-gram probe detects benchmark golds at 0.804 AUC, while recipe-prompt models track style more than the base or verified arm.The recipe arm’s style-score correlation is ρ = +0.171 and exceeds the verified arm on 94.2% of queries; D4 does not show the same tracking.Source: §5 · p. 7
  • The 1,401 hard-tier queries won only by the recipe arm skew toward golds sharing query wording, not toward LLM-styled golds.The cells with the highest hard-tier scores carry little style-tracking, so chasing style is not what buys the tier.Source: §5 · p. 7

6 What Transfers to Real Data, and Two External Benchmarks

On real duplicates, most of the easy-tier gap between recipe and verified training does not transfer; other benchmarks show different patterns. MELD’s score rises without loss of duplicate retention, while SABER-Math’s document attack fails and its summary-based gain is small.

Real duplicates

  • 45.33 points of easy-tier separation shrink by 35.59 [29.51, 41.47] on cross-language duplicates, with 27.24 [21.40, 32.93] of the shrinkage recipe-specific.The recipe arm retains a 9.74 [3.96, 15.71] lead over the verified arm on cross-language duplicates, but both score below the untrained base; same-language reprints show no recipe-arm lead.Source: §6 · p. 7

Other retrieval benchmarks

  • On BRIGHT, the recipe arm leads the verified arm by 8.36 [3.42, 13.99] and 0.75 [0.16, 1.52] nDCG points on two splits, while both arms remain below the base.Both arms also fall below the base on BRIGHT’s math splits and ImpliRet; the base leads both arms on ten of eleven human-curated MIRB tasks.Source: §6 · p. 7

SABER-Math

  • On SABER-Math, the registered document attack fails, while summary-only training gains +0.0255 ± 0.0041 nDCG@10 over the base at a matched budget.The gain is not proof of matching SABER’s rule: correlation rises only 1.14×, and the attack targets discovery rather than the tournament that scores pairs.Source: §6 · p. 7

MELD

  • On MELD, reconstructed-prompt training gains +26.97 R@1 on average and reaches 90.67 cross-language retrieval, above the base in every seed.A rewritten-prompt arm retains 80% of the gain; most is attributed to cross-dialect pairing rather than exact wording, and same-language retention stays at the base.Source: §6 · pp. 7–8

Cross-benchmark comparison

  • Only on MathNet-Retrieve does one training-file edit raise benchmark score while lowering real retrieval at once.MELD’s recipe raises score while preserving real retrieval, and SABER’s matched-budget summary gain is small.Source: §6 · p. 8

7 Limitations

The evidence is bounded by benchmark-specific attacks, limited replication, and automated probes. The reported MathNet-Retrieve inversion also comes from headline arms trained with a leaky exact-text filter.

Scope and replication

  • Each LLM-written surface is attacked on only one benchmark, limiting claims about how general the results are.On SABER-Math, the summary gain does not establish that matching the construction rule buys the score, and the attack omits the scoring tournament.Source: §7 · p. 8

MELD

  • MELD’s evaluation prompt is unpublished, so the attack reconstructs it; whether the MathNet recipe arm’s +9.45 MELD lead extends to human-written pairs is untested.MELD’s pairs are LLM-written before human review.Source: §7 · p. 8

Residual contamination

  • About 33% of each headline arm’s anchors were a benchmark query’s own statement under the leaky exact-text filter.Corrected-filter retrains confirm rather than replace the headline arms, and one used different source lists.Source: §7 · p. 8

Automated evidence and probes

  • No human has read the 1,401 hard-tier queries uniquely won by the recipe arm, and the shared-wording interpretation relies on an automated classifier.The duplicate probes omit near-misses; the cross-language probe also mixes equivalence learning with alignment, and beyond the main arms each cell has three seeds.Source: §7 · p. 8

8 Conclusion

MathNet-Retrieve’s easy and hard tiers reward different properties, and the benchmark’s own recipe can improve its score while reducing real retrieval. The paper recommends generator-free evaluation material and reporting recipe sensitivity alongside benchmark results.

Conclusion

  • The easy tier rewards LLM-rewrite genre, while the hard tier rewards deep rewrites paired against minimal-edit near-misses.Half to two thirds of the easy-tier lead comes from LLM-written pairs; the remainder requires the benchmark prompt and vanishes on real duplicates no generator wrote.Source: §8 · p. 8
  • Only on MathNet-Retrieve does one training-file edit demonstrably raise benchmark score while lowering real retrieval at once.MELD training preserves real retrieval, and SABER-Math’s summary-based gain is small but robust to a matched budget.Source: §8 · p. 9

Recommendations

  • The authors recommend human-written test material, a generator-untouched subset, and a published recipe-sensitivity number.Regenerating golds is weaker: the benchmark’s own recipe removes under a quarter of the gap, and multiple prompts remove under two fifths.Source: §8 · p. 9

Ethics and Artefact Release

The paper treats its benchmark attacks as a construct-validity finding, not a vulnerability requiring embargo. It releases the materials needed to reproduce the analysis and generator-free evaluations of the intended ability.

Ethics and Artefact Release

  • The authors characterize the attacks as reproducible from benchmark construction descriptions, not as non-public exploits requiring embargo.Two prompts are public; the third was reconstructed from a published procedure paragraph.Source: Ethics and Artefact Release · p. 9
  • The published rewriter prompt is the recipe arm’s attack vector, and practitioners can reproduce it unintentionally.Source: Ethics and Artefact Release · p. 9
  • The release includes analysis artefacts, commands to regenerate large intermediates, two mined duplicate evaluations, and three trained models.The duplicate evaluations are presented as generator-free targets for the benchmark’s intended ability.Source: Ethics and Artefact Release · p. 9

Reproducibility Statement

The paper makes its results regenerable through a repository that records commands, jobs, inputs, and outputs. It reports calibration checks, including the checks that failed.

Reproducibility Statement

  • Each result can be regenerated from the repository, which records the exact command, SLURM job, input files, and output artefacts.Source: Reproducibility Statement · p. 9
  • Evaluation harnesses are calibrated against published rows where available, and failed checks are reported in Appendix A.The failures include MathNet-Retrieve 4B, MELD, and ImpliRet.Source: Reproducibility Statement · p. 9
  • Pre-registered predictions are stored in the producing job scripts, while absent training data and rank dumps are listed with rebuild commands.Source: Reproducibility Statement · p. 9

AI Use Statement

The paper identifies LLM generation as an experimental manipulation and discloses other AI assistance in coding, review, and writing. The authors retain responsibility for the final work.

AI Use Statement

  • LLMs generated and filtered every LLM-written training set, while the verified arm, duplicate evaluations, and near-miss probe contain no LLM-written text.Back-translation controls use NLLB-200.Source: AI Use Statement · p. 9
  • An AI coding assistant helped write and debug scripts, evaluation harnesses, and analysis code; results were re-derived from repository files.Source: AI Use Statement · p. 9
  • Six adversarial review passes by the assistant drove many reported corrections and controls, while authors conducted the research design and interpretation.Source: AI Use Statement · p. 9
  • The assistant helped draft and revise the manuscript, but the authors reviewed the work and take responsibility for its claims and artefacts.Source: AI Use Statement · p. 9

A Data construction, matched-budget asymmetries, and harness calibration

This section examines matched-budget data construction and evaluation choices through real-duplicate and generator-free probes, alongside harness calibration. It reports evaluation-slice-dependent training effects and that hard-tier performance does not reliably track near-miss discrimination, while probe depth, seed counts, and unresolved or within-paper comparisons limit the findings’ scope.

Data construction and matched-budget asymmetries

  • The verified arm uses computer-algebra-checked pairs, while the recipe arm uses LLM-generated pairs under the benchmark prompt; equal row counts still leave unequal training volume and source coverage.The recipe arm has 2.63 negatives per row versus 1.52 in the verified arm’s training draw, and trains on 15,431 versus 9,921 examples; different verified-arm seed draws also add data variation.Source: A Data construction, matched-budget… · pp. 13–14

Matched-budget asymmetries and signal control

  • Signal matching leaves the verified arm at 16.57 ± 1.07 easy-tier R@1, not significantly above its usual band, while the easy-tier gap remains 45.33 points.The preregistered confirmation is that matching the recipe arm’s negative volume and example count does not raise the verified arm’s score, so this imbalance is not a first-order driver of the gap.Source: A Data construction, matched-budget… · p. 14, Table 3 · p. 15

Harness calibration

  • The Qwen3-Embedding-4B calibration remains unresolved: the closest tested setting reaches 13.19 easy R@1, still 1.57 below the published 14.76.No tested setting met the preregistered 0.5-point replacement threshold, so the paper retains its existing baseline settings.Source: A Data construction, matched-budget… · p. 15
  • SABER-Math calibration reproduces its metric code and three reference scores, but BRIGHT, MELD, and ImpliRet results are limited to comparisons among the paper’s own models.BRIGHT uses a 1,024-token limit rather than the official model-specific maximum; MELD’s reimplementation does not reproduce its published scores, and the ImpliRet ReasonIR check misses its registered tolerance.Source: A Data construction, matched-budget… · p. 16

Real-duplicate evaluation

  • Both controlled arms fall below the untrained base on every cross-language query slice, and the recipe arm loses more where shared formulas are scarce.The losses include answer-corroborated queries and the bottom quartile of shared rare-formula mass; the table reports slice-level gaps and 95% nested-bootstrap intervals.Source: A Data construction, matched-budget… · p. 17, Table 4 · p. 17

Same-language duplicate evaluation

  • On the primary same-language slice, recipe and verified arms score 46.30 and 51.30 R@1, respectively, with a −5.00 [−13.42, 3.35] gap.The interval includes zero, while the registered difference-in-differences is 50.33 [41.85, 58.75]; the table reports each model’s scores across the primary, English, leaked, and exact-reprint slices.Source: Table 5 · p. 18, A Data construction, matched-budget… · p. 19

Generator-free near-miss probe

  • The generator-free near-miss probe contains 1,107 computer-algebra-checked items, and the recipe arm scores 60.49 ± 2.50 R@1 on the full corpus and 68.38 ± 2.40 on probe-only ranking.Each item has a verified positive and up to three minimal-edit negatives; the supplied results do not include the comparison needed to classify the recipe arm’s advantage.Source: A Data construction, matched-budget… · p. 19

Families of tests and correction

  • Uncorrected multiple testing means exploratory p-values describe the data rather than establish significance; the registered prompt ladder, SABER and MELD attacks, same-language readout, and factorial reading map are confirmatory.The easy-tier D1−D3 comparison is post hoc, and Table 9’s real-duplicate column is reported for completeness rather than as a rung ranking.Source: A Data construction, matched-budget… · p. 20

Statistical reporting and estimator variation

  • Reported uncertainty depends on the data design: exact-budget files share rows across seeds, whereas larger files add row-draw variation; cross-language duplicate intervals resample clusters and then queries.Gap estimates generally use nested query-and-seed bootstraps with 5,000 replicates; the eight-seed arm gap and difference-in-differences nest cluster resampling inside seed resampling.Source: A Data construction, matched-budget… · p. 20

Generator-free near-miss probe

  • Hard-tier score did not track generator-free discrimination: Spearman’s ρ was +0.06 on the full corpus and +0.19 on near-miss ranking across 22 out-of-family cells.The two highest hard-tier cells scored 30.50 and 38.06 probe R@1, below D4 alone’s 42.31 and the reference’s 53.51; back-translated positives scored 74.79 probe R@1.Source: A Data construction, matched-budget… · p. 20, Table 6 · p. 21

Evaluation-set, scale and supervision caveats

  • Evaluation and supervision caveats include shared source problems, compressed differences among strong prompted models, recipe-specific checkpoint selection, and formal negatives that may misclassify constrained-variable problems.The 4B replication changes three settings for both arms and omits the V2 mixing fix and prompt ladder; replacing verified counterexamples with recipe near-misses further reduces cross-language performance.Source: A Data construction, matched-budget… · p. 22

Scope and limits

  • Probe and ladder findings are depth-limited: the probe tests shallower rename and reformulation edits, while unrelated near-misses and paraphrase controls separate authorship from depth only to a limited extent.The ladder rungs have three seeds each except D1, and the easy-tier D2−D3 difference flips sign across seeds; the back-translated control is shallower than a free restatement.Source: A Data construction, matched-budget… · pp. 20, 22

Difference-in-differences scale robustness

  • The difference-in-differences was positive on all three scales, with intervals excluding zero: 35.59 [32.57, 38.32] points, 1.676 [1.562, 1.785] log-odds, and 0.328 [0.272, 0.384] error reduction.The share of the benchmark gap not reaching real data varied by scale: 78.5% on points, 80.3% in log-odds, and 60.0% as error reduction.Source: A Data construction, matched-budget… · p. 22

B The contamination audit in full

The audit finds substantial query-text overlap in both training arms, but estimates that it accounts for only a small part of the recipe arm’s lead. The lead persists on clean queries and in corrected-gate retrains, though none of those retrains reproduces the headline experiment exactly.

B The contamination audit in full

  • About 33% of both arms’ rows anchor on benchmark queries, with contamination slightly heavier in the verified arm, which loses.The reported ranges are 31.98–33.26% for recipe and 33.20–35.43% for verified.Source: B The contamination audit in full · p. 23
  • Gold-document overlap is mostly in negatives, which training teaches the model to rank lower, so this overlap works against the recipe arm at test time.Source: B The contamination audit in full · p. 23
  • 46.25 ± 0.86 points on gate-caught queries and 44.73 ± 0.86 on the strictest clean slice preserve the recipe arm’s easy-tier lead.The gap is larger than the 45.33 ± 0.93 overall gap on the gate-caught slice.Source: Table 7 · p. 23, B The contamination audit in full · p. 23
  • 1.24 ± 0.08 of the 45.33-point gap is attributed to contamination after netting each arm’s gains on leaked queries against its untrained-copy baseline.Using the gate-caught slice as the counterfactual instead gives a net attribution of 0.59 ± 0.06 points.Source: B The contamination audit in full · pp. 23–24
  • The controls are not interchangeable: filtering reduces the recipe arm’s data, fresh-source regeneration changes source lists, and same-source regeneration gives problems a second rewrite.The fresh-source real-duplicate comparison is not like-for-like with the other controls.Source: B The contamination audit in full · pp. 24–25
  • 43.11 ± 0.90, 44.45 ± 0.87, and 44.38 ± 1.46 are the easy-tier gaps in the three corrected-gate retrains, against 45.33 ± 0.93 headline.The filtered retrain uses 4,101 matched rows; the two regenerated controls restore 6,145 rows.Source: B The contamination audit in full · pp. 24–25, Table 8 · p. 25
  • Zero-leakage readers should use Table 8 and the clean slices of Table 7, because every model outside the retrains uses the original gate.No retrain repeats the headline experiment: filtering reduces data, while regeneration changes source assignment or adds second rewrites.Source: Table 7 · p. 23, B The contamination audit in full · p. 25, Table 8 · p. 25

C The rewriter-prompt ladder and the style probe, in detail

This section details the rewriter-prompt ladder and style probe, examining tier scores, interactions between rewrites and negative sources (including retention), and how recipe-family versus verified training tracks a cross-generator LLM-rewrite signature.

The rewriter-prompt ladder and the style probe, in detail

  • Hard-tier R@1 falls at every step from D1 to D4 across seeds, while D1 trails D2 and D3 on the easy tier and D4 leads on real duplicates.D1’s hard-tier lead over D2 and easy-tier deficit versus D3 retain their direction using D1’s eight-seed values; D4’s near-zero hard score is attributed to missing negatives, not its prompt.Source: Table 9 · p. 26, C The rewriter-prompt ladder and the… · pp. 26–27
  • Three-seed replication confirms the hard-tier D1−D2 step, but finds D2 and D3 indistinguishable on the easy tier.Hard-tier D1−D2 averages +3.14 across seeds; easy-tier D2−D3 averages −0.34, with its sign flipping. Single-run query bootstraps can reverse the apparent easy-tier ordering.Source: C The rewriter-prompt ladder and the… · p. 26
  • D3’s 9.0% row shortfall cannot explain its lower hard-tier score: cutting D2 to the same size raises hard R@1, and matching D3’s sources does not erase the gap.At three seeds the row-count cut gains +0.96 hard-tier points; the matched-row prompt effect is −3.05, and retraining on D3’s own sources scores 7.81, on D2’s side of the midpoint.Source: C The rewriter-prompt ladder and the… · p. 27
  • D4 restatements paired with CAS negatives reach 10.30 ± 1.15 hard R@1, matching the recipe arm’s 9.42, while either component alone stays near zero.The hard tier rewards deep LLM rewrites paired with hard negatives, not benchmark-prompt wording; Table 12 reports the factorial comparisons.Source: C The rewriter-prompt ladder and the… · p. 28, Table 12 · p. 29
  • At matched count, recipe-prompt near-misses score 20.71 ± 0.82 hard R@1 versus 10.30 with CAS negatives; unrelated-prompt near-misses score 16.96 ± 2.08.Thus LLM-written near-misses account for most of the climb above CAS negatives, with the recipe prompt adding 3.75 points at matched count. At full volume, recipe near-misses raise hard-tier points but cost 6.02 more cross-language retention points than unrelated-prompt near-misses.Source: C The rewriter-prompt ladder and the… · pp. 29–31
  • Near-copy positives paired with recipe near-misses score 0.11 ± 0.03 hard R@1 and 12.47 ± 0.68 cross-language R@1, so neither the tier nor retention is rescued.The deep-rewrite positive is also required for hard-tier performance; this near-copy pairing has the lowest cross-language retention among trained cells described here.Source: C The rewriter-prompt ladder and the… · p. 31
  • At verified negatives, the recipe-specific share of the easy-tier gap ranges from 15.47 to 23.18 points across D4 and D5 references; D5 alone scores 18.71 ± 1.31 easy R@1.Source: C The rewriter-prompt ladder and the… · p. 32
  • Separating the recipe prompt’s positive and near-miss calls leaves hard R@1 at 9.79 ± 0.66, within the registered band around the recipe arm’s 9.42.The shared-phrasing account is unsupported; the split-call arm remains 17.18 points below D4 restatements with recipe near-misses and loses easy-tier and cross-language performance.Source: C The rewriter-prompt ladder and the… · p. 32

Recipe-prompt ladder and factorial comparisons

  • Three back-translation hops recover none of the reference’s easy-tier gain: the cell scores 12.85, 34.06 points below the reference, while noisier positives raise hard R@1 from 1.09 to 4.47.About one sixth of rows return unchanged and the rest lose fluency, so the added depth is translation noise rather than deeper restatement; the hard tier rewards distance from a minimal-edit near-miss, not equivalence.Source: C The rewriter-prompt ladder and the… · pp. 32–33
  • The ladder still shows hard-tier decline across D1–D3, but the D3-to-D4 drop reflects missing negatives, and the ladder cannot isolate wording from near-miss quality.D4 positives score 10.30 hard R@1 with verified negatives and 26.97 with recipe negatives; the factorial, not the ladder, shows that recipe near-misses add an increment over unrelated-prompt near-misses.Source: C The rewriter-prompt ladder and the… · p. 33
  • CAS negatives lift D4 restatements from 30.03 to 46.91 easy R@1, while CAS positives without negatives score 7.42, below the base’s 8.32.The verified arm’s easy-tier gain over the base, from 8.32 to 17.05, therefore comes from its counterexample negatives, not its positives; D4 with CAS negatives remains 15.47 points below the recipe arm.Source: C The rewriter-prompt ladder and the… · p. 33
  • Attaching negatives to D4 restatements raises hard R@1 but lowers cross-language retention from 93.81, showing that one negative-file edit can invert the ranking.Verified negatives yield 10.30 hard R@1 and 78.02 cross-language R@1; recipe negatives yield 20.71 or 26.97 hard R@1 and 83.89 or 78.88 cross-language R@1, depending on volume.Source: C The rewriter-prompt ladder and the… · p. 34
  • The recipe arm leads the reference by 15.47 easy-tier points but trails it by 11.77 on cross-language duplicates; with shared recipe negatives, it trails D4 by 12.63–17.63 cross-language points.At matched negatives, the recipe arm leads D4 by 17.79 easy-tier points with count-matched negatives and 24.59 with the full set; its hard-tier deficit is 11.29 or 17.55 points, respectively.Source: C The rewriter-prompt ladder and the… · p. 34

Style probe

  • The style detector transfers from Qwen rewrites to benchmark Gemini rewrites at 0.804 AUC, while minimal-edit near-misses score 0.562, near chance.[tbl13#b4f990] D4 rewrites are detected at 0.676 and recipe-family prompts add 0.204 [0.186, 0.222], supporting an LLM-rewrite genre signature with a recipe increment. The detector reaches 0.880 AUC on held-out problems after formatting artefacts are stripped; otherwise those artefacts inflate it to 0.969.Source: C The rewriter-prompt ladder and the… · pp. 34–35, Table 13 · p. 35
  • The recipe arm prefers LLM-styled documents more than the verified arm (+0.171 versus −0.019), and recipe-family ladder rungs track the fingerprint more strongly than D4.[tbl14#a27b9b] D2 tracks at +0.164, D3 at +0.105, and D4 at −0.000; within golds and near-misses separately, the recipe arm still tracks style, ruling out ranking golds above near-misses as the explanation. The highest hard-tier factorial cells show little style tracking (+0.046 and +0.050).Source: C The rewriter-prompt ladder and the… · pp. 35–36, Table 14 · p. 36
  • All six public models fall between −0.07 and +0.004 style correlation, unlike recipe-family models at +0.10 to +0.17; this is no evidence of recipe inflation among measured leaderboard models.[tbl15#68a1d0] The detector transfers to benchmark golds at 0.804 AUC, not perfectly, and the result covers only six public models. On hard-tier queries won only by the recipe arm, word overlap separates wins from other queries at AUC 0.639, while recipe phrasing and LLM-style scores do not.Source: C The rewriter-prompt ladder and the… · pp. 36–37, Table 15 · p. 37

D Full result tables, the mixing grid, and the two external attacks in detail

This section details the full results and mixing grid, examining how the recipe-arm lead varies across backbones, evaluation-gold authors and prompts, and how the gains differ by rank and tier. It also reports the two external attacks and related checks, including MELD’s genre effect and SABER’s document- and summary-channel results.

Backbone replication

  • The easy-tier lead recurs on multilingual-e5-large (+19.12 R@1), BGE-large-en-v1.5 (+15.91), and at 4B, where real-duplicate retention reverses the benchmark ordering.At 4B, easy R@1 is 77.77 for the recipe arm against 39.51 for the verified arm, while real-duplicate R@1 is 70.99 against 83.72. BGE’s English same-language probe shows no recipe-arm retention lead.Source: D Full result tables, the mixing grid,… · p. 37

Late-interaction retriever

  • ColBERTv2’s recipe arm leads by +32.48 easy R@1, while losing 8–11 points on same-language retention; the verified arm preserves the base score.The cross-language probe is unusable for this English-only backbone; the retention comparison uses the same-language reprint probe. The benchmark lead and retention loss come from one seed and one run per cell.Source: D Full result tables, the mixing grid,… · pp. 37, 40, Table 21 · p. 40

Mixing grid

  • Across three seeds, V2 reaches 22.19 ± 0.79 easy R@1 and 74.22 ± 2.42 cross-language retention; its instruction alone raises the untrained base’s retention to 90.59.The data mixture restores medium-tier recall depth: V2 scores 67.25 medium R@5, above the untrained base’s 60.91. Without the instruction, V2 falls from 21.70 to 16.51 easy R@1 and from 74.30 to 68.70 retention.Source: Table 19 · p. 39, D Full result tables, the mixing grid,… · pp. 40–41

SABER-Math attack

  • The SABER attack is an imperfect reconstruction: it omits two relevance stages, reuses a threshold tuned elsewhere, and uses a different summary generator.Its one-sided containment gate also missed reverse-direction overlap: 11 sources exceeded 0.35 and three restated an evaluation query, so “disjoint” overstates what was verified.Source: D Full result tables, the mixing grid,… · p. 41
  • On SABER-Math, the document attack scores below the base, but the pure summary arm exceeds it in every seed; a matched-budget summary arm retains a smaller gain.The document attack changes nDCG@10 by −0.0129 [−0.0212, −0.0042] promptless, far from the registered +0.02 threshold. At matched steps, the summary arm gains +0.0255 ± 0.0041 nDCG@10 over the base; its cross-language retention matches the unmatched summary arm.Source: D Full result tables, the mixing grid,… · pp. 41–42

External retrieval benchmarks

  • On BRIGHT and ImpliRet, both controlled arms trail the base; on human-curated MIRB [32], the base leads both arms on 10 of 11 tasks, and recipe trails verified on five.On ImpliRet, recipe scores 9.23 against the base’s 16.03, while V1 and V2 score 16.04 and 16.20. The BRIGHT recipe arm beats verified on both math splits, but both arms remain below the base.Source: Table 20 · p. 40, D Full result tables, the mixing grid,… · pp. 42–43, Table 22 · p. 43

MELD attack

  • The MELD attack is an approximate, cross-vendor reconstruction, and its proximity gate removed 39 pairs—not the 62% rejected by quality filtering.After gating, no exact match remained; at most 7.04 of the 26.48 R@1 points gained could be explained by the bounded residual exposure estimate.Source: D Full result tables, the mixing grid,… · p. 43
  • On MELD, the own-domain arm beats the heldout arm by +7.04 [2.96, 10.93] at R@1, but the arms match at R@5 and on same-framing ranking.The advantage keeps its sign across three seeds (mean +6.36), and the attacked model scores 90.67 ± 0.53 cross-language R@1 against the base’s 84.73. A matched prompt rerun attributes 80% of the attack to the equivalence format and 20% to wording.Source: D Full result tables, the mixing grid,… · p. 44

Two asymmetries, and a registration that landed the other way

  • The registered MELD attack prediction failed: the MELD-attacked model reaches 90.84 and 91.86 strict R@1 on cross-language duplicates, while the SABER-attacked model falls to 34.86.The passage concludes that MELD’s recipe carries over to genuine cross-language duplicates, so gaming its score and keeping real ability are not in opposition.Source: D Full result tables, the mixing grid,… · p. 45

The medium tier

  • On the medium tier, the recipe arm leads the verified arm by 5.70 points, but the difference-in-differences is −4.03, with 95% CI [−10.12, 1.82].Against the reference, the recipe arm leads by 4.62 on the tier and trails by 11.77 on cross-language duplicates; the passage says the inflation claim rests on the easy tier.Source: D Full result tables, the mixing grid,… · p. 45

The contrasts at R@5 and R@10

  • The recipe-versus-verified lead is 7.33 at R@5 on the easy tier and 7.48 on cross-language duplicates, while at R@10 the leads are 2.88 and 4.01.The passage characterizes the inflation as rank-1: by R@5, the arms and reference retrieve the gold about equally often on benchmark and real duplicates.Source: D Full result tables, the mixing grid,… · p. 45

Verified-supervision numbers, and why the 4B one stays out of the ranking

  • The 4B verified arm reaches 39.51 easy R@1, while the recipe arm’s 77.77 against 39.51 comes with cross-language retention of 70.99 against 83.72.The 4B result is excluded from Table 18’s ranking because its rename and reformulation positives are paraphrases, the genre tested by the easy tier.Source: D Full result tables, the mixing grid,… · p. 45

The regeneration defence experiment: registered outcomes

The regeneration defence rerun with the benchmark’s own vendor

  • In the vendor rerun, the Gap is 19.61 for exact-template golds, 21.83 for paraphrase, 16.59 for different style, and 17.74 for mixed prompts.Every regenerated row collapses for every model; the base scores 0.00 R@1 on all four. The rerun uses deeper rewrites than the shipped golds and cannot isolate a vendor fingerprint.Source: D Full result tables, the mixing grid,… · pp. 46–47, Table 25 · p. 47

The regeneration defence experiment: registered outcomes

E Forgetting, the mixing frontier, and the regeneration defence

Training on verified pairs or near-misses often reduces real cross-language retrieval, while the proposed remedies trade benchmark score against retention. The hard tier remains buyable but is not yet shown to measure real discrimination, and regeneration only partly defends against recipe sensitivity.

E Forgetting, the mixing frontier, and the regeneration defence

  • Most fine-tuning on verified pairs or LLM near-misses lowers cross-language duplicate retrieval below the 84.73 strict R@1 base, and more such rows mean greater losses.A few files escape: back-translated positives with verified negatives and D4 restatements with unrelated-prompt near-misses stay at the base; same-language reprints show no beyond-noise gains.Source: E Forgetting, the mixing frontier, and… · pp. 47–48
  • 22.19 easy R@1 at 74.22 cross-language retention is the V2 mixing result; replay and weight averaging trade benchmark score for retention rather than dominating outright.Weight soups of V1 with the base retain more than soups of verified-only training at similar easy-tier scores, but near-miss ranking survives neither remedy except at α = 1.Source: E Forgetting, the mixing frontier, and… · p. 48
  • 31.48 points of the recipe arm’s 45.64-point lead remain after golds are drawn from three prompts, while a different-style test lets same-style-trained D3 overtake it by 11.28.The registered expectation of a step-by-step decline from template to paraphrase failed; drawing golds from several prompts removes under two fifths of the gap.Source: E Forgetting, the mixing frontier, and… · p. 48
  • 20.71 hard R@1 coexists with cross-language retention within noise of the base (83.89 against 84.73), but same-language retention falls (47.73 against 62.40).The authors call the hard tier buyable, not yet shown to be measured; the highest-scoring training sets do worse than the reference on a near-miss test no LLM wrote.Source: E Forgetting, the mixing frontier, and… · p. 48
  • Five public models remain in the untrained base’s narrow style-probe band, far from models trained on recipe prompts, so no current leaderboard model appears to show the recipe signature.The authors argue that leaderboard scores merit scrutiny unless evaluations use material no generator wrote or report how much matching the recipe can gain.Source: E Forgetting, the mixing frontier, and… · p. 48

F The prompts, verbatim

The appendix documents the prompts and procedures used for the attacks, including a controlled system-prompt ladder, the strict equivalence judge, and candidate-pair generation, screening, and SABER solution summaries. It also reports prompt comparisons, including word-overlap results and the hard-tier ordering.

F The prompts, verbatim

  • The four-rung ladder changes only the system prompt: D1 is the benchmark’s Appendix-F template, D2 a close paraphrase, D3 a deliberately different style, and D4 an audience-focused restatement prompt.D1–D3 request equivalent variants and near-misses; D4 requests restatements only and writes no negatives by design.Source: F The prompts, verbatim · pp. 49–51
  • 10.30 hard R@1 over eight seeds with verified-arm counterexamples rises to 26.97 with the recipe arm’s own near-misses.The factorial comparison attributes the hard-tier collapse to missing negatives.Source: F The prompts, verbatim · p. 49
  • D2, D3, and D4 have word-set Jaccard similarities to D1 of 0.411, 0.192, and 0.259, respectively, so overlap does not follow ladder order.D4 shares more words with D1 than D3 does, yet its hard-tier score collapses because it asks for restatements and attaches no negatives.Source: F The prompts, verbatim · p. 49
  • D5 frames restatement as a survey author presenting another author’s problem, while the unrelated near-misses prompt adds minimal-edit companions to D4’s restatement task.The companion should look like the original but differ mathematically in one consequential feature.Source: F The prompts, verbatim · pp. 52–53
  • The judge treats problems as equivalent only when one is a mechanically translatable reformulation using the same method; changes that alter the answer or strategy fail the test.The judge was run through Qwen3-32B-AWQ, with Mistral-Small-24B-Instruct-2501 also used for the two-judge control.Source: F The prompts, verbatim · pp. 53–54
  • The reconstructed MELD prompt requests equivalent statements in two mathematical subfields, with minimal lexical overlap and content at basic graduate level.The 84/16 control recasts generation as seminar translation exercises; later calls avoid already-covered topics or vary their statements.Source: F The prompts, verbatim · pp. 54–55
  • The MELD screen checks each statement’s validity, pair equivalence, and whether the pair could be made less similar; it gates on validity and equivalence.The procedure records the dissimilarity answer, while judging equivalence strictly rather than accepting shared topics or one-way implication.Source: F The prompts, verbatim · p. 55
  • The prompt asks for the conceptual idea that makes a solution work, framed imperatively as a hint, while excluding how-to execution, multi-step reasoning, equations, and numeric computations.Source: F The prompts, verbatim · p. 56

Related papers

Loading 2609.31927v1…