Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot

Hui Mao

Source-linked AI summary

Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot

Hui Mao

arXiv:2608.16795v1Aug 17, 2026cs.CEcs.AI

TL;DR

Existing evaluations of scientific question-generation systems are subjective and non-falsifiable. This paper introduces historical backtesting and finds evidence-structure-first generation outperforms LLM-only prompting, while the outcome taxonomy—not the judge—fails validation.

Problem
Scientific question-generation systems lack an objective, falsifiable evaluation beyond subjective expert scores, LLM judgments, and curated case studies.
Method
Historical backtesting freezes questions generated from pre-cutoff literature and evaluates them against temporally isolated future literature using auditable outcomes.
Results
Evidence-structure-first generation yields 39% answered versus 15% for direct LLM prompting, while the outcome taxonomy fails validation despite judge models matching professional annotation.
Takeaways & Limitations
Historical backtesting makes claims about scientific question quality falsifiable, comparable across systems, and computable from frozen public data.
Takeaways & Limitations
Modern models may contain post-cutoff knowledge, because the protocol seals retrieval but not model weights, creating a fundamental contamination threat.

Abstract

from arXiv · show

Systems that generate scientific research questions are evaluated today by expert scores, LLM-as-judge ratings, or curated case studies -- all subjective, none falsifiable. We formalize historical backtesting as an alternative: a system generates questions from a corpus frozen at a historical cutoff, the questions are frozen before any access to later literature, and a temporally isolated future corpus then determines whether each question was subsequently answered, partially addressed, independently posed, or ignored, and whether its underlying premise was supported or refuted. The protocol is model-agnostic: any system that emits frozen questions can be scored. We release reproducible astronomy instances with temporally isolated corpora, frozen questions, auditable labels, four reference baselines, and a submission interface. Two findings result. First, evidence-structure-first generation outperforms LLM-only prompting: across a generator decomposition crossed with a four-cutoff stress test (2010-2024, 798 judged questions) whose last window postdates model training, LLM-only generation shows memorized relevance without specific foresight, while a generator using no model weights at all finds questions whose premises the future refutes in every era. Second, a seven-rater agreement study (two blinded human annotators, five judge models, 90 items) indicts the outcome taxonomy rather than the judge: two careful humans agree at kappa = 0.17, every judge model agrees with the professional annotator as well or better (0.17-0.26), and frontier models agree with one another at 0.60 -- certifying an LLM judge by model-model agreement would have overstated its reliability threefold. A prospective instance -- 200 questions frozen 2026-08-17, scored 2027-2030 -- is released so the central claims become contamination-free tests that time itself will grade.

1 Introduction

Historical backtesting evaluates frozen scientific questions against temporally isolated future literature, making subsequent engagement an observable, falsifiable proxy rather than a complete definition of value. The paper formalizes this model-agnostic benchmark, releases an astronomy instance and prospective test, and reports method and measurement-validity findings.

Motivation and scope: Future scientific engagement is an observable, falsifiable proxy for one dimension of question value, not a complete definition because popularity bias and resource constraints can cause valuable questions to be ignored.The protocol enables comparison from frozen public data without asking anyone’s opinion.
Protocol: Historical backtesting freezes questions generated from a historical corpus, then uses unseen future literature to label their outcomes and premise fates.The taxonomy separates question fate from premise fate, including answered, partially_addressed, posed_but_open, not_addressed, supported, refuted, weakened, still_plausible, and not_applicable.
Protocol: Any system that emits frozen questions—including evidence-graph pipelines, prompted LLMs, heuristics, or human scientists—can be evaluated under identical conditions.Because the protocol contains no question generator, it functions as a benchmark rather than an architecture-specific validation appendix.
Empirical findings: 39% vs. 15% answered and 13% vs. 0% premise refutation: evidence-structure-first generation outperformed direct LLM prompting across 125-question submissions (p = 3 × 10−5).The supported claim concerns a class of methods and holds across four historical cutoffs, while a ten-question evidence-graph example is explicitly not ranked against baselines.
Measurement validity: κ = 0.17 between humans, κ = 0.60 among frontier models: the seven-rater study indicates that the outcome taxonomy, rather than the judge, fails validation.The study included two blinded human annotators, five judge models, and 90 items; model–model agreement would have overstated judge reliability.
Prospective instance: Two hundred questions frozen at cutoff 2026-08-17 will be scored during 2027–2030, providing a contamination-free prospective test.The instance includes four generators, preregistered scoring, and published corpus manifests and hashes.

2 Related Work · 3 The Historical Backtesting Protocol

The paper replaces subjective evaluation of generated scientific questions with a reproducible historical backtesting protocol. It freezes questions and temporally isolates future evidence so systems can be compared on auditable outcomes under contamination-aware controls.

2 Related Work: Existing scientific-question evaluations rely mainly on expert preference or LLM scoring, measuring contemporaneous opinion rather than whether ideas later proved valuable.Prior work spans rule-based discovery, robot scientists, AI-scientist proposals, literature-based generation, agentic refinement, and end-to-end research automation.
2 Related Work: Historical backtesting adapts held-out-history and temporal-forecasting paradigms while addressing overfitting and data contamination through freezing, corpus isolation, and audited model-weight threats.The protocol completely controls the retrieval channel but treats post-cutoff training data as an explicitly declared threat.
3 The Historical Backtesting Protocol: The protocol evaluates frozen questions Q = {q1, . . . , qn} against a future corpus through six fully specified steps designed to yield identical measurements across groups.Each step is specified as part of the reproducible benchmark procedure.
3.1 Step 1: Choose a cutoff: A historical cutoff T divides past Corpus A from strictly later Corpus B; Astronomy v1 uses T = 2020-12-31 and recommends a cutoff at least 4 years old.The future window must allow the research community time to act while retaining a modern frontier in the past corpus.
3.2 Step 2: Generate questions: Any generator may consume only Corpus A evidence, and every question must record pre-cutoff source_evidence_ids or fail mechanical cutoff validation.Permitted generators include evidence graphs, prompted LLMs, citation heuristics, and human experts.
3.3 Step 3: Freeze: Questions are serialized with provenance and frozen: true before post-cutoff access, then never edited, preventing future knowledge from changing the prediction target.Freezing is the protocol’s load-bearing rule because otherwise question text can drift toward the future literature.
3.4 Step 4: Define the future window: Corpus B is independently manifested and isolated for 2021–2026, while fixed top-k retrieval selects k = 8 candidate documents for each question.The retrieved titles and abstracts are recorded so reviewers can inspect exactly what the judge saw.
3.6 Step 6: Assess outcomes: Judges assign independent question-fate and premise-fate labels under citation and conservatism constraints, allowing outcomes such as answered-but-refuted-premise to remain expressible.Premise refutation counts as question success; labels are provenance-tiered, and submissions pass through identical isolation, retrieval, judging, adjudication, metrics, and reporting pipelines.

4 Benchmark Metrics

The benchmark evaluates frozen questions through outcome and premise rates over the submitted set, separating community coverage from what happened after engagement. It also emphasizes premise refutations while treating lead time and calibrated community attention as limited or future metrics.

Outcome and premise rates: All rates are computed over the n frozen questions, and the four outcome rates sum to one.Each question receives outcome and premise labels, with independent post-cutoff papers recorded as supporting evidence.
Outcome and premise rates: Coverage measures whether the community reached a question’s topic, while answer, partial, and open rates describe what happened after engagement.Coverage is not reported alone because fashionable-topic questions can maximize it without demonstrating specific discovery value.
Premise outcomes: Premise refutation rate is prioritized because refutations identify accepted conclusions that later evidence showed to be wrong.The two-dimensional taxonomy records such a refutation without erasing that the question was answered.
Limitations: mean_lead_time_years: null is reported because identifying community first-posed dates requires review-literature annotation that is not yet available.The released records retain a weaker, well-defined alternative rather than asserting an indefensible number.
Limitations: A calibrated community-attention index remains future work, although v1 records supporting bibcodes, venues, and years and reports evidence strength.Raw attention is not treated as the headline metric.

5 The Astronomy v1 Instance · 6 Baselines

Astronomy v1 instantiates historical backtesting in exoplanet atmospheres with frozen, reproducible corpora, mechanically enforced temporal isolation, ten evidence-graph questions, and extensive released audit records. Four reference baselines use the same frozen-output evaluation pipeline, revealing saturated coverage, hindsight advantages, and a direct-LLM contamination signal.

5 The Astronomy v1 Instance: Astronomy v1 targets exoplanet atmospheres, combining fast-moving literature, structured catalogs, space-telescope archives, and a 2021–2026 JWST natural experiment.JWST data resolved questions that were unanswerable at the cutoff.
5 The Astronomy v1 Instance: Frozen ADS manifests rebuild both corpora reproducibly, while Corpus B adds targeted follow-up queries for five questions’ objects.Records without abstracts are dropped, and the targeted objects are HD 189733, HD 209458, WASP-12, WASP-121, and TRAPPIST-1.
5 The Astronomy v1 Instance: Six mechanical leakage checks enforce cutoff integrity across corpus contents, source evidence, retrieval, judging, record alignment, and Corpus B’s strictly post-cutoff window.The protocol excludes post-cutoff catalog updates and restricts judges to retrieved candidates.
5 The Astronomy v1 Instance: Ten frozen questions were generated from Corpus A by evidence_graph_v1, which extracts provenance-backed claims, types cross-paper tensions, and ranks falsifiable questions.The detected tensions include observational tensions, methodological challenges, single-dataset conclusions, and independent qualifications.
5 The Astronomy v1 Instance: The release includes frozen questions, corpus manifests, retrieval and adjudication records, rationales, computed metrics, per-question results, four frozen baseline submissions, and reports.Full ranked retrieval lists with scores are scheduled for v1.1, while the scaled v1L instance is also released.
6 Baselines: The four baselines bracket chance-directed attention, direct LLM prompting, community-written review questions, and citation prominence, with every output frozen before future-corpus access.B3 is expected to score highly on coverage, making premise refutation and lead time its discriminating metrics; B4 benefits from post-cutoff citation counts.
6 Baselines: Every non-random system achieves 90–100% future attention, whereas random claims achieve 70%, the only system with three not_addressed labels.Coverage therefore separates the random floor from the other systems but is not reported alone.
6 Baselines: The direct-LLM baseline reaches s1 = 0.732 versus 0.593–0.668 for every other system and 4.6 cited papers per question, yet records 0% answered, 0% refuted, and 90% partial.Its future-like phrasing is consistent with weights that had read the 2021–2026 literature, motivating the contamination-probe interpretation.

7 Pilot Results: Historical Validation

The ten frozen evidence_graph_v1 questions were engaged by all 2021–2026 literature, with two answered, seven partially addressed, one independently posed and open, and one premise refuted. Case studies show both successful null-result backtesting and instrument-timing confounds, while small-sample, rater-relative results limit interpretation.

Aggregate results: 100% coverage: all ten frozen questions were substantively engaged by 2021–2026 literature; two were answered, seven partially addressed, one independently posed and open, and one premise was refuted.There were 2.4 supporting papers per question on average, 60% of labels had at least two independent papers, and earliest engagement lagged the cutoff by a mean 2.9 years.
Case studies: Three independent 2021–2026 analyses performed the specified test for HD 209458 b and converged that its terminator water abundance was consistent with a retrieval artifact.The analyses used improved HST/JWST systematics treatment, Bayesian model averaging, independent retrieval frameworks, and ground-based high-resolution spectroscopy.
Case studies: q_002, the rank-1 question about terminator heterogeneity biasing WASP-12 b water abundance and C/O ratio, was independently posed but remains open for that planet.The broader methodological concern—inhomogeneous terminator biases in retrievals—has since been engaged without resolving the WASP-12 b-specific question.
Case studies: q_005 produced an answered null result: high-resolution spectroscopy placed stringent upper limits on NH3 and HCN, supporting the premise that HST data did not independently support their detection.Backtesting counts a cleanly resolved null result the same way it counts a positive result.
Limitations: q_011’s first engagement occurred in 2024 because JWST had to fly, and stellar contamination has so far prevented a definitive test.This makes engagement timing partly an instrument-schedule effect rather than purely a question-quality signal.
Limitations: [0.69, 1.0] is the exact 95% Clopper–Pearson interval for 10/10 coverage, underscoring that this one-domain, ten-question pilot has wide uncertainty and rater-relative absolute rates.Baseline rows are judge-only rather than adjudicated, and the generating system’s LLM components postdate the cutoff.

8 Scaling the Baselines: Astronomy v1L

Astronomy v1L scales the benchmark to 125 questions and enlarged corpora, revising two small-sample conclusions while preserving the direct-LLM contamination signature and the evidence-graph system’s standing. At scale, coverage separates systems and answered outcomes have a nonzero floor, but premise refutation is no longer exclusive to evidence-graph generation.

Astronomy v1L: Astronomy v1L expanded the corpora to 4,040 past and 5,754 frozen-window future records, respectively 1.8× and 2.4× the v1 corpora.The instance retained the same cutoff, retrieval, and judge settings while broadening manifests across six astronomy topics.
Coverage: At n = 125, engagement separated across random claims at 72.8%, citation leaders at 87.2%, and direct LLM at 96.0%.Random versus direct-LLM engagement differed at p < 10^-4, while random versus citation leaders differed at p = 0.007; the n = 10 reading was a small-sample artifact.
Premise refutation: At n = 125, citation leaders refuted four premises, yielding 3.2% [1,8], so premise refutation was not unique to the evidence-graph system.The four cases concerned HD 209458 b, K2-18 b, WASP-121 b, and ultracool-dwarf retrievals; no baseline refuted a premise at n = 10.
Answered outcomes: Random robustness templates were answered 11.2% [6,18] of the time, establishing a nonzero answered floor; citation leaders reached 25.6% and cleared it, whereas direct LLM reached 15.2%.Citation leaders exceeded the floor at p = 0.005, while direct LLM did not at p = 0.42.
Contamination signature: The direct-LLM baseline retained the highest future-literature similarity, s1 = 0.722, alongside 96% engagement, 94% multi-source rate, and 1.9-year mean engagement.Its cited evidence concentrated in 2021–2022, the years closest to its training distribution; scaling increased its answered rate to 15.2% from 0/10 at n = 10.
Corpus-relative metrics: Re-judging the ten v1 questions on the 2.4× larger corpus flipped four labels in both directions, showing that rates depend on the corpus, retriever, and judge triple.Two questions gained answered, one fell to not_addressed, one premise changed from weakened to still_plausible, and q_008’s answered + refuted label reproduced.

9 Separating Hindsight Memorization from Foresight

The section separates reasoning over pre-cutoff evidence from memorized future and topic prior by comparing generation pipelines and moving cutoffs across gpt-4.1’s training boundary. Evidence-structure-first pipelines show specific foresight, while LLM-only relevance does not reliably indicate it.

Experimental design: Backtest performance decomposes into reasoning over pre-cutoff evidence, memorized future, and topic prior, which a single retrospective instance cannot distinguish.The experiments vary the question generator at a fixed cutoff and vary the cutoff across the judge-model’s training boundary.
Fixed-cutoff decomposition: 24.8% vs. 15.2% answered and 3 vs. 0 refutations: weight-free structure-only generation beats LLM-only generation on the n = 125 decomposition.Adding the same LLM as a verbalizer raises answered rate to 39.2% vs. 24.8% for the same detected evidence structures, with p = 0.02 described as suggestive only.
Fixed-cutoff decomposition: 5% vs. 95–100%: LLM-only questions name specific catalogued objects far less often than structure-first questions, while achieving the highest engagement.The LLM-only pipeline produces broad questions that many papers touch but few settle.
Temporal contamination stress test: 14.7% pooled in-training vs. 16.0% post-training, p = 0.82: pipeline A’s answered rate remains statistically flat across the training boundary.Its engagement remains 92–98% across cutoffs, while specificity stays near the floor, indicating that relevance persists without specific foresight.
Temporal contamination stress test: 10.1% vs. 1.0%, p = 4×10−5: pipeline B refutes premises across all cutoffs, including the post-training era, whereas A’s only refutations occur at c2010.B’s answered margin over C persists post-training at +28 points at c2024; A shows a modest in-training phrasing-similarity peak of 0.717–0.724 versus 0.703 post-training.

10 Judge Validation

Judge validation finds meaningful internal checks but exposes substantial limitations: engagement responds to evidence swaps, while labels vary with prompts and models, and human agreement is too weak to provide a reliable reference. The paper’s headline ordering remains robust across judges, but reproducibility requires a judge that uses the full label space.

Internal validation: Engagement falls from 74% (67/90) on true pairings to 20% (18/90) on mismatched evidence (p < 10−4), with five of six systems individually significant.The 20% residual is treated as a generosity floor for engagement rates.
Internal validation: Identical reruns yield outcome agreement 96.7% and 93.3% (κ = 0.95, 0.91), while prompt rewrites reduce agreement to κ = 0.68 and neutral label codes to κ = 0.57.Temperature 0 leaves roughly 3–7 points of label noise, and neutral codes never produce posed_but_open.
Judge comparison: Cross-model agreement with gpt-4.1 is low—κ = 0.34 and 0.20 on outcome, and 0.19 and 0.05 on premise status—because weaker judges collapse toward conservative defaults.gpt-4-turbo assigns still_plausible 87 times and never uses refuted or weakened; gpt-4o assigns still_plausible 78 times.
Conclusion robustness: All three judges preserve the evidence-structure-first-over-LLM-only ordering and engagement ordering, while failed comparisons are ties at zero rather than reversals.Across all twelve judge–conclusion cells, no judge reverses a conclusion; the benchmark therefore requires judges that demonstrably use the full label space.
Human arbitration: Human–human agreement is κ = 0.17 on outcome and 0.17 on premise status (41–44% raw), and high annotator confidence does not improve judge agreement beyond κ = 0.11.The study concludes that the taxonomy lacks convergent labels, so no judge can be validated against a dependable human reference.

2. Every model clears the human–human bar with the expert — and none clears it

The five models match the professional expert about as well as another human does, but neither human nor model raters converge reliably on the outcome taxonomy. Model–model agreement can therefore overstate judge reliability, making absolute benchmark rates rater-relative while fixed-judge comparisons remain defensible.

Agreement with the expert: κ = 0.17–0.26 against the professional team, with gpt-4.1 highest at 0.26, but all values remain below conventional reliability thresholds.The frontier models reach κ = 0.21 despite two additional model generations, and newer models do not close the gap.
Model–model agreement: κ = 0.60 model–model agreement between the two frontier models exceeds the human–human figure threefold, with 67/90 identical labels.The passage notes that both frontier models use the full label space, although some other high-agreement cells reflect two-label collapse.
Rater disagreement: κ ≤ 0.17 marks the first annotator’s agreement with every other rater, alongside 33 answered and 9 not_addressed labels versus the expert’s 33/24.The expert’s marginal distribution closely tracks the strict judges’, while the non-expert labels more items answered and fewer not_addressed.
Interpretation and consequences: The evidence favors an underdetermined taxonomy: no human or model converges reliably, even though the judge is not distinguishably worse than a human rater.Absolute rates such as “answered 39%” are rater-relative, whereas comparative claims under a fixed judge remain defensible because headline orderings survived three judges with failures being ties.

11 Error Analysis

The error analysis uses curation and adjudication logs to identify failure modes that can distort question validity, counting, premise evaluation, and evidence judgments. These failures motivate benchmark rules and planned transparency improvements.

Curation failures: Near-duplicate questions can double-count one insight, so submissions require screening and an instance-level deduplication note.q_002/q_003 were merged during curation; mechanical similarity screening is planned for v1.1.
Question validity: Leading questions presuppose answers and cannot be cleanly refuted, requiring neutral reframing before outcome evaluation.Reframing q_008 enabled its later refutation to be expressed as attribution rather than as a presupposed abundance claim.
Question validity: Self-contradictory quantifiers can make questions unanswerable, motivating the benchmark’s clarity gate.q_003 asked whether detection was robust to biases exceeding an order of magnitude, where that magnitude implies non-robustness.
Question validity: Conflating statistical detection with physical conditions produced a low clarity score and required reframing as sensitivity analysis.q_010 combined atmospheric state with detection-pipeline factors and received the set’s lowest clarity-gate score, 6.0.
Premise evaluation: Inherited premises can yield questions that merely restate source claims, so tension typing and premise checks should seek both supported and refuted premises.The premise dimension is intended to prevent a portfolio from showing uniform support for its sources.
Evidence retrieval: Retrieval near-misses can support evidence judgments through methodological relevance without matching named targets, making full top-8 lists important for re-litigation.The q_007 adjudication log records this engagement-bar judgment, and full top-8 releases are planned for v1.1.

12 Threats to Validity

Historical backtesting reduces rater subjectivity but retains important confounds, including future attention, model-weight contamination, judge reliability, retrieval limits, small samples, curation hindsight, and domain-specific self-evaluation. The protocol makes several threats auditable or measurable, but retrospective instances cannot eliminate all contamination and limited statistical power remains consequential.

Attention and coverage: Future inattention can make excellent questions appear not_addressed, while fashionable topics can gain engagement without merit.Bounded windows and posed_but_open distinguish unresolved from ignored questions; coverage is not reported alone, and baseline B4 prices prominence.
Contamination: Modern model weights may contain post-cutoff knowledge, allowing generators or judges to use future information despite sealed retrieval.Source-evidence audits and submission metadata mitigate this threat, but none of the listed mitigations is complete.
Judge and retrieval reliability: Judge labels discriminate mismatched evidence, with engagement falling from 74% to 20%, but retain a 20% generosity floor and prompt sensitivity of κ = 0.57–0.72.Resampling stability is κ ≈0.9; abstracts rather than full texts constrain what a single judge can assess.
Statistical power and scope: At the observed effect size of 10% versus 3.2% premise refutation, 80% power at α = 0.05 requires n ≈209 per arm, exceeding the ten-question submission.The scaled instance uses n = 125 per system, and two of three small-sample conclusions did not survive.
Scope and curation: The evidence-graph system’s strongest signals depend on 37 annotated claims and 16 reviewed relation edges, making annotation labour—not compute—the supply constraint.The evaluation also remains limited by one domain, self-evaluation, judge-only baseline labels, and curation hindsight in retrospective instances.
Multiple comparisons: After conservative Bonferroni correction, headline contrasts survive, whereas contrasts at p ≈0.01–0.05 are labelled suggestive.Surviving examples include the structure-vs-LLM refutation gap at p = 4 × 10−5 and the mismatched-evidence control at p < 10−4.

13 Outlook: Discovery as Search, Language as Realization

The paper argues that scientific question discovery should be treated as evidence-structured search rather than bare prompting, while preserving a distinct role for language models in articulating candidate tensions as testable questions. This framing shifts the central engineering problem toward literature representation, configuration search, and ranking questions by their measured future outcomes.

Scope and interpretation: Bare prompting does not reliably perform problem discovery, although the experiments do not show that language models cannot originate scientific questions in principle.The evidence is limited to one prompting strategy, model family, and domain.
Capabilities separated: LLM-only generation exhibits topic priors, with 92–98% engagement but only 5% object anchoring versus 95–100% for structural generation.The passage presents these metrics as evidence that knowing likely future topics differs from asking specifically anchored questions.
Architecture: The proposed division of labour is machine-scale structural search over evidence, followed by candidate tensions, LLM articulation, and testable questions.The LLM is used to realize and articulate candidates rather than conjure novelty from nothing.
Architecture: Scientific question discovery becomes a computable search problem over explicit representations of literature claims, making representation, configuration enumeration, and ranking the key engineering questions.A literature of thousands of papers produces tens of thousands of claims and a much larger combinatorial space that machines can sweep in parallel.
Ranking and evaluation: The benchmark links each question’s generating signal to its measured fate, enabling tests of which structural patterns most often yield questions the future answers, advances, or refutes.Signals include tension type, object, method-dependency, and source claims; the initial object-co-mention and stance-cue probe is below the human-gated graph in precision.

14 Roadmap and Conclusion

The paper concludes that historical backtesting has been formalized and released through retrospective and prospective astronomy instances, while identifying reliability and scaling priorities for future work. Its decomposition and temporal stress test show that memorized relevance is not scientific foresight.

Prospective Instance: 200 questions were generated from a 2015–2026 corpus, frozen at cutoff 2026-08-17, and publicly committed with per-file SHA-256 digests.The prospective instance contains 50 questions from each of four automatic generators.
Roadmap: The roadmap prioritizes a reliability-gated v2 taxonomy with fewer labels, checkable conditions, and measured human–human κ before judge evaluation.It also calls for scaling the submission side with a 100+ question evidence-graph run on v1L.
Conclusion: The study formalized historical backtesting and released two retrospective astronomy instances plus one prospective instance with temporally isolated corpora and fully auditable labels.A ten-question pilot found every frozen question substantively engaged by literature the generating system never saw.
Conclusion: Scaling the baselines to 424 questions overturned two small-sample conclusions, established a measurable floor under a third, and left the qualitative refutation distinction unresolved.The distinction concerns refutation reached by specifying its test in advance.
Conclusion: The generator decomposition and temporal stress test show that memorized relevance is not scientific foresight.The result turns the benchmark’s deepest limitation into its sharpest finding.