Source-linked AI summary

Scientific reasoning does not reliably translate into scientific forecasting in frontier AI

Sean Wu, Pan Lu, Yupeng Chen, Jonathan Bragg, Yutaro Yamada, Peter Clark, David Clifton, Philip Torr, James Zou, Junchi Yu

arXiv:2605.22681v2cs.AI

TL;DR

It remains unclear whether strong scientific reasoning enables reliable forecasts of future scientific advances. Using CUSP to evaluate six frontier AI models, the paper finds that models identify plausible mechanisms but are substantially less reliable at forecasting feasibility, realization, and timing.

  • Problem

    Existing scientific reasoning evaluations provide limited evidence about whether AI models can reliably anticipate advances beyond their knowledge cutoff.

    Source: Introduction · p. 2
  • Method

    The study introduces CUSP, a temporally grounded suite of 4,760 verifiable events across eight disciplines and four dimensions of event-level forecasting.

    Source: Introduction · p. 2
  • Results

    Models consistently identify plausible mechanisms but are less reliable on feasibility, solution alignment, realization, and timing, with feasibility near chance and temporal errors systematic.

    Source: Introduction · p. 2
  • Takeaways & Limitations

    Scientific reasoning and scientific forecasting should be evaluated as distinct capabilities when assessing AI systems for research prioritization and decision-making.

    Source: Discussion · p. 11

    CUSP evaluates event-level forecasting rather than broader scientific foresight, so whether these limitations extend to long-term trajectories or paradigms remains open.

    Source: Discussion · p. 11

Abstract

from arXiv · show

AI systems are increasingly used to support forward-looking scientific judgment, but it remains unclear whether they can form reliable expectations about future scientific advances. Here we show that strong scientific reasoning does not reliably translate into accurate forecasting of future scientific advances. To study this question, we introduce CUSP, a temporally grounded evaluation suite for event-level scientific forecasting across eight scientific disciplines. Across six frontier AI models, we observe a striking asymmetry in forecasting performance together with systematic error patterns. Models often identify plausible mechanisms underlying future scientific advances, yet perform near chance on feasibility assessment, generate solution strategies that only weakly align with realized advances, and systematically predict scientific advances later than they become publicly observable. Providing additional pre-cutoff scientific knowledge improves performance but does not eliminate these forecasting limitations. These findings suggest that current AI systems possess substantial retrospective scientific competence but limited forward-looking predictive capability. Scientific forecasting should therefore be evaluated as a complementary dimension of AI scientific capability when deploying AI systems for research prioritization and scientific decision-making.

Introduction

This study introduces CUSP to evaluate whether frontier AI models can forecast scientific advances beyond their knowledge cutoff. Across four forecasting dimensions, models show strong mechanistic reasoning but limited, systematically biased predictive accuracy.

  • AI-assisted research prioritization and resource allocation require reliable judgments about which scientific advances are likely, yet inaccurate forecasts could misdirect scientific attention, resources, and effort.Source: Introduction · p. 1
  • CUSP evaluates event-level scientific forecasting using 4,760 verifiable events across 8 disciplines and four dimensions: feasibility, mechanisms, solution design, and timing.Source: Introduction · p. 2
  • Across six frontier models, mechanistic forecasting is strongest, while feasibility assessment, open-ended solution design, and temporal prediction are substantially weaker.Feasibility remains near chance, solution strategies weakly align with realized approaches, and temporal predictions show large systematic errors.Source: Introduction · p. 2
  • Additional pre-cutoff scientific knowledge consistently improves forecasting, but substantial limitations remain, indicating that reliable prediction requires more than retrieving and reasoning over existing knowledge.Source: Introduction · p. 2
  • Forecasting failures include persistent response biases, overconfidence, and a consistent tendency to predict advances later than their public observation.Together, the findings distinguish substantial retrospective scientific competence from limited forward-looking predictive capability.Source: Introduction · p. 2

Results

Across CUSP’s temporally grounded evaluation of scientific-event forecasting, frontier AI models identify plausible mechanisms more reliably than they predict feasibility, realized approaches, or observation timing. These limitations persist despite additional historical knowledge, differ by capability and domain, and reflect systematic biases rather than random error.

Overall forecasting performance on CUSP

  • Models perform best on mechanistic forecasting but near chance on feasibility, with weak solution design and temporal prediction.GPT-5.4 reaches 79.2% MCQ accuracy but 49.1% Binary accuracy and 5.04/10 on FRQ; all models’ predicted dates are later than realized dates.Source: Overall forecasting performance on CUSP · pp. 4–5, Table 1 · p. 4

Forecasting realized approaches is harder than generating plausible ones

  • Generated solution strategies are feasible but weakly aligned with the approaches underlying realized scientific advances.GPT-5.4’s alignment score is 3.3/10 versus feasibility scores of 4.8–6.0; from GPT-4o to GPT-5.4, specificity rises from 2.9 to 6.2 and novelty from 2.4 to 4.9, while alignment rises only from 2.2 to 3.3.Source: Forecasting realized approaches is… · p. 5

Knowledge alone is insufficient for forecasting scientific events

Human performance under identical information conditions

  • Under identical information conditions, humans outperform GPT-4o on feasibility, mechanistic, and temporal forecasting.Humans achieve 73.0% versus 52.5% on Binary, 67.0% versus 40.0% on MCQ, and 0.65 versus 0.54 on Date.Source: Human performance under identical… · p. 8

Forecasting errors are systematic rather than random

  • Forecasting errors are systematic: models show response priors, overconfidence, delayed predictions, and anchor-point effects.Feasibility calibration errors range from 0.16 to 0.31, predicted dates are four months to more than two years late, and a mean-date baseline outperforms every model, including GPT-5.4 at 0.857.Source: Forecasting errors are systematic… · pp. 8–9

Capability-dependent variation across scientific domains

  • Domain effects are strongest for mechanistic and temporal forecasting, whereas feasibility assessment and solution design remain difficult across domains.Mechanistic accuracy ranges from 48.1% in chemistry to 65.4% in environmental science, temporal scores from 0.198 in chemistry to 0.481 in AI, while feasibility remains 46.4%–51.6% and biology’s top solution-design score is 4.05.Source: Capability-dependent variation across… · p. 9

Discussion

Using CUSP across 4,760 events and eight disciplines, the study finds that frontier models can reason scientifically yet remain unreliable at forecasting future advances. These systematic limitations persist despite additional knowledge and have implications for research prioritization and scientific planning.

  • Across 4,760 events spanning eight disciplines, models identified plausible mechanisms and detailed solutions but remained unreliable about whether, when, and how advances would occur.These limitations persisted across scientific domains and reflected consistent systematic error patterns.Source: Discussion · pp. 9, 11
  • Scientific reasoning and knowledge access improved forecasting but were insufficient, as generated strategies often diverged from the approaches underlying realized advances.Additional pre-cutoff scientific knowledge consistently improved performance, yet substantial forecasting limitations remained.Source: Discussion · p. 11
  • The findings are limited to event-level forecasting, with open questions about broader foresight, event-selection effects, internal mechanisms, evaluation subjectivity, and residual familiarity in the human study.CUSP spans multiple disciplines, but automated free-response assessment is more subjective than deterministic tasks, and the controlled human study covers only AI-related advances.Source: Discussion · p. 11
  • Future research should develop better-calibrated forecasting systems and examine links between event-level forecasting, broader scientific foresight, and systematic failure mechanisms.Scientific forecasting may complement scientific reasoning as AI systems assume larger roles in scientific discovery and decision-making.Source: Discussion · p. 11

Methods · The CUSP suite

CUSP is a temporally grounded, event-level suite for forecasting scientific progress across disciplines, using controlled knowledge cutoffs and complementary tasks. It also adds prospective Time Capsule questions to assess prediction consistency and calibration when outcomes remain unresolved.

The CUSP suite

  • CUSP evaluates scientific forecasting with a temporally stratified corpus of verifiable events spanning January 2024 to March 2026 across diverse disciplines.Domain-specific inclusion criteria account for differing publication dynamics and evidentiary standards, with full criteria provided in Appendix J.Source: The CUSP suite · pp. 11–12
  • Its sources combine high-impact natural-science publications, community-curated AI papers, and time-resolved benchmark leaderboards, using earliest DOI-associated dates to prevent temporal leakage.Natural-science events come from Nature, Science, and Cell; AI coverage includes GPQA Diamond24, MMLU-Pro35, and Humanity’s Last Exam26.Source: The CUSP suite · p. 12

Question Types and Synthesis

  • CUSP operationalizes forecasting through binary feasibility and calibration, multiple-choice mechanism identification, free-response solution design, and date prediction tasks.Each accepted milestone receives four complementary evaluation formats probing distinct forecasting dimensions.Source: Question Types and Synthesis · p. 12
  • Tasks remove post-cutoff identifiers from structured problem, approach, and results components, then undergo independent-LLM and human-expert validation for faithfulness and objective verifiability.The construction pipeline is summarized in Fig. 1, with the taxonomy in Tab. S17 and validation details in Appendix B.Source: Question Types and Synthesis · p. 12

CUSP Time Capsule Construction

  • CUSP Time Capsule extends evaluation to prospective questions about measurable future events, supporting analyses of prediction consistency, confidence calibration, and cross-model agreement without accuracy scoring.Questions cover scientific benchmarks, institutional recognitions, technological events, and AI capability forecasts, and are curated with human experts.Source: CUSP Time Capsule Construction · p. 12

Key Statistics of CUSP

  • The suite contains 4,760 scientific events and 17,429 structured tasks across eight top-level domains and 4,245 specialized subcategories.Biology and artificial intelligence are the largest represented domains, with 1,234 and 1,141 papers respectively.Source: Key Statistics of CUSP · pp. 12–13

Comparison to Related Evaluation

  • CUSP provides a temporally stratified, event-level framework that separates scientific knowledge access from forecasting ability, addressing gaps in general-world forecasting and retrospective reasoning benchmarks.It remains substantially unsaturated across evaluated frontier systems, consistent with forecasting being harder than retrospective reasoning.Source: Comparison to Related Evaluation · p. 13, Table 2 · p. 13

1 Model Evaluation

CUSP evaluates scientific forecasting through deterministic outcome scoring and rubric-based reasoning assessment under a strict knowledge cutoff. Its framework combines task-specific accuracy measures with leakage-controlled evaluation of open-ended scientific proposals.

  • CUSP uses two complementary tracks to assess forecast correctness and whether scientific proposals are temporally consistent and scientifically plausible.The framework addresses concerns that correct answers may arise from flawed, unfaithful, or post-hoc reasoning processes.Source: §1 · p. 13
  • Deterministic scoring covers binary classification, multiple-choice, and date prediction, using exact or semantic answer matching and exp(−0.1d) temporal distance scoring.Binary scores average original and negation-perturbed variants to mitigate directional response bias; the framework scores only tasks present for each event.Source: §1 · pp. 13–14
  • Free-response proposals undergo separate web-based leakage detection, with post-cutoff information treated as contaminated and ineligible for forecasting credit.This separation prevents memorization from being conflated with scientific reasoning ability.Source: §1 · p. 14
  • Non-contaminated free responses receive rubric-based 0–10 scores for alignment, specificity, novelty, and feasibility, averaged into the final score.The protocol uses GPT-5.4-mini with agentic web search and includes a human expert correlation study to assess scoring reliability.Source: §1 · p. 14

Author contributions statement

The authors jointly conceived, designed, conducted, analyzed, and supervised the study, with all authors revising and approving the final manuscript.

Author contributions

  • Sean Wu and Pan Lu contributed equally; Wu led data curation, task synthesis, validation, and experiments, while Yu conceived the study and jointly supervised the project.Lu, Yu, and others contributed to evaluation-suite design, experiments, analysis, interpretation, manuscript drafting, scientific discussion, and evaluation guidance.Source: Author contributions statement · p. 15

Supplementary Information for Scientific reasoning does not reliably translate into scientific forecasting in frontier AI … A.3 Task synthesis procedure

CUSP constructs temporally grounded scientific-forecasting tasks from rigorously filtered publication and leaderboard milestones. Its pipeline standardizes milestone representations, removes post-cutoff terminology, generates five task formats, and validates answers through source checks, independent review, and human assessment.

A.1 Data acquisition and source construction

  • CUSP combines rigorously dated milestones from Nature, Science, and Cell with prominent AI papers and structured capability leaderboards.Natural-science milestones use the earliest observed manuscript DOI date across multiple academic APIs as the knowledge cutoff; AI papers are selected using community recognition and an age-adjusted impact score, while leaderboards test extrapolation of time-resolved progress.Source: §A.1 · p. 19

A.2 Corpus Extraction and Automated Filtering

  • The corpus extracts verifiable scientific milestones using only titles, metadata, and abstracts, yielding standardized and reproducible prediction targets.Abstracts summarize primary claims, methods, and quantitative outcomes while avoiding dependence on uneven full-text availability.Source: §A.2 · pp. 19–20
  • An agentic LLM pipeline retains only abstract entries containing at least one verifiable, measurable outcome and applies domain-specific acceptance criteria.Computational entries require validated benchmark advances or explicit performance metrics, whereas experimental entries require measurable quantities or defined physical or biological properties.Source: §A.2 · p. 20
  • The filtering process rejects descriptive, speculative, and review abstracts, producing concise acceptance summaries or rejection justifications while reducing subjective annotation bias.The procedure is designed for scalability and consistency, and high-impact publications can still fail the milestone criteria.Source: §A.2 · p. 20

A.3 Task synthesis procedure

  • Each accepted milestone is decomposed into a problem, technical approach, and measurable outcome, while source-specific terminology is removed to prevent post-cutoff identification.The fields separately represent the research limitation, mechanism or design, and quantitative result before task generation.Source: §A.3 · p. 20
  • GPT-4o generates binary, perturbed-binary, multiple-choice, free-response, and date-prediction tasks, with answers checked against the source publication by Grok-3 and human reviewers.The formats assess feasibility, plausible alternatives, approach identification, implementation strategies, and predicted realization month and year.Source: §A.3 · p. 20

A.4 Key Statistics and Distributional Properties

CUSP is a temporally grounded benchmark spanning 27 months, 17,429 validated tasks, eight scientific domains, and 4,245 subcategories. Its heterogeneous task availability, controlled labels, and varied linguistic complexity prioritize reliable, diverse evaluation of scientific forecasting.

Temporal Distribution

  • CUSP contains 4,760 milestones across all 27 months from January 2024 to March 2026, averaging 176.3 papers per month (± 51.5).This temporal coverage supports evaluation across short- and long-term forecasting horizons.Source: §A.4 · p. 21

Task Composition

  • The validated dataset contains 17,429 task instances across four task types, with 54.0% of papers admitting all four and the remainder admitting fewer tasks.Strict LLM-as-a-judge filtering removes tasks lacking verifiability, faithfulness, or logical consistency, prioritizing reliability over uniformity.Source: §A.4 · pp. 21–22

Label Distribution

  • Binary and perturbed binary questions yield an approximately 0.75 yes/no label distribution, while shuffled MCQ options ensure uniform correct-answer positions.Ground-truth dates for date-prediction tasks remain relatively stable across the dataset.Source: §A.4 · p. 22

Question Length and Complexity

  • Average task length increases from 29.2 words for binary questions to 36.6 for MCQs, 41.8 for FRQs, and 70.6 for problem statements.Problem statements provide rich scientific context, FRQs require open-ended methodological synthesis, and MCQs demand discriminative reasoning over expert-level distractors.Source: §A.4 · p. 22

Domain and Subcategory Diversity

  • CUSP spans eight top-level domains and 4,245 subcategories, led by biology with 1,234 papers and artificial intelligence with 1,141.The dataset also includes medicine, neuroscience, materials science, physics, environmental science, chemistry, and other domains.Source: §A.4 · p. 22

B Benchmark validation

CUSP uses independent LLM validation to verify benchmark question faithfulness and quality across task types. Structured judgments enable fine-grained filtering while preserving valid dataset coverage.

Validation framework

  • An independent Grok-3 judge validates GPT-4o-generated CUSP questions against source abstracts, reducing potential self-evaluation bias.Grok-3 is selected for its comparable frontier capability but different architecture and training distribution.Source: B Benchmark validation · p. 22

Binary and Perturbed Binary Questions

  • Binary questions are checked for faithfulness and verifiability, while perturbed versions must introduce meaningful unsupported modifications rather than trivial paraphrases.Validation examines entities, conditions, outcomes, quantitative details, operational clarity, thresholds, and added constraints.Source: B Benchmark validation · p. 22

Multiple-Choice Questions (MCQ)

  • MCQ validation checks faithful problem statements, technically supported correct answers, and plausible distractors that are not directly supported.This design aims to require mechanistic forecasting rather than superficial pattern matching.Source: B Benchmark validation · p. 22

Free-Response Questions (FRQ)

  • FRQ validation focuses on prompt context and background fidelity because open-ended answers may have multiple valid solutions.The problem description must accurately reflect the scientific problem in the source abstract.Source: B Benchmark validation · p. 23

Filtering and reliability

  • Separate structured LLM judgments support field-level filtering that removes invalid components, preserves valid samples, and removes dependent date tasks when binary questions fail.Human evaluation is also used to assess the reliability of the automated validation framework.Source: B Benchmark validation · p. 23

C Forecasting in a time capsule … E Extended Results

CUSP Time Capsule evaluates whether frontier models can forecast post-cutoff scientific and AI progress, revealing shared directional expectations but substantial disagreement in projected magnitudes. Across related benchmarks and AI-for-science settings, CUSP is positioned as a temporally grounded, unsaturated test of forward-looking scientific judgment rather than retrospective reasoning.

C Forecasting in a time capsule

  • CUSP Time Capsule tests frontier models’ open-ended forecasts of scientific breakthroughs and future AI capabilities beyond an April 2026 knowledge cutoff.The evaluation asks models to extrapolate future scientific and AI progress across benchmarked capability predictions.Source: C Forecasting in a time capsule · p. 24
  • Models uniformly forecast higher 2027 global CO2 emissions than in 2025, but differ substantially in projected growth and implied decarbonization pace.Claude S4.5, DeepSeek R1, and GPT-4o are comparatively conservative, GPT-5.4 predicts a larger increase, and LLaMA 3.3 forecasts the steepest growth.Source: C Forecasting in a time capsule · p. 24, Figure S3 · p. 24
  • GPT-5.4 makes the most optimistic capability forecasts, projecting Humanity’s Last Exam from 56.8% to 74% without tools and from 64.7% to 82% with tools by 2027-10.It also predicts near-saturation on GPQA Diamond and MMMLU, whereas DeepSeek R1 forecasts smaller gains and earlier plateaus.Source: C Forecasting in a time capsule · pp. 24–25
  • Forecasts for GPQA Diamond and MMMLU cluster near their upper bounds, while tool-augmented Humanity’s Last Exam elicits greater variation as a more open-ended benchmark.These patterns suggest models expect saturation on some benchmarks but perceive greater uncertainty in capability-sensitive evaluation settings.Source: C Forecasting in a time capsule · p. 25

D.2 AI for Science

  • CUSP frames scientific forecasting as a measurable capability for studying the implicit scientific priors and future-oriented world models embedded in frontier AI systems.The framework is intended to support scalable evaluation beyond factual recall and retrospective scientific reasoning.Source: C Forecasting in a time capsule · p. 25, §D.2 · p. 26

D.1 Comparison to Related Benchmarks

  • Existing scientific benchmarks test retrospective knowledge and reasoning, but several are near-ceiling while CUSP remains substantially unsaturated across evaluated frontier systems.Humanity’s Last Exam26, AstaBench25, PreScience28, ResearchBench27, and ScienceQA23 assess established scientific content rather than anticipation of future discoveries; MMLU-Pro, GPQA Diamond, and MedQA show near-ceiling performance, whereas CUSP remains unsaturated (Figure S8).Source: §D.1 · p. 26
  • CUSP extends forecasting benchmarks by targeting real peer-reviewed scientific discoveries, enforcing pre-milestone knowledge cutoffs, and combining feasibility, calibration, mechanistic, generative, and date-prediction tasks.Unlike general-world forecasting benchmarks such as ForecastBench36, FutureX37, FOReCAst38, and PROPHET39, CUSP explicitly targets scientific discovery and limits indirect exposure to future information.Source: §D.1 · p. 26

D.2 AI for Science

  • In AI for Science, prior systems generally depend on researchers to define problems and directions, leaving independent identification of promising, feasible research frontiers unresolved.CUSP operationalizes this capability by requiring predictions of concrete, verifiable discoveries emerging only after a strict knowledge cutoff.Source: §D.2 · p. 26

E.1 Full Web-search Results … F.1 Human evaluation of Dataset Validation

Across supplementary analyses, models showed constraint-sensitive forecasting failures, domain-dependent FRQ alignment, temporal overestimation, and overconfidence, while human evaluation found LLM judges rigorous for dataset validation.

E.1 Full Web-search Results

  • Web search still produced false positives when models matched broad scientific themes but ignored explicit constraints or extrapolated beyond the evidence.GPT-5.4 overlooked a <2 nm particle-size requirement and treated evidence near 85% as support for a rigid 90% threshold.Source: §E.1 · pp. 28–29

E.2 FRQ Sub-dimension Score Analysis

  • FRQ evaluation decomposed responses into alignment, specificity, and novelty, with the specificity–alignment gap identifying technically detailed but methodologically misaligned answers.All sub-dimensions were scored from 0–10, and positive gaps indicated plausible-sounding hallucination.Source: Table S5 · p. 31, §E.2 · p. 31

GPT-OSS

  • Chemistry and Physics had consistently lower FRQ alignment, reflecting greater domain specificity and less overlap with general pretraining data.The research-area analyses included MCQ, date prediction, binary merged accuracy, FRQ, and sub-domain predictability measures.Source: Table S6 · p. 32, Table S7 · p. 32, Table S8 · p. 32

GPT-4o

  • DeepSeek R1 correctly anticipated GLM-4.5’s RL-plus-MoE direction and mechanism but predicted its realization nearly a year late.It predicted 2026-07 versus the 2025-08 ground truth and identified RL as a driver of future capability improvements.Source: GPT-4o · p. 39

E.4 Model Bias and Confidence

  • Models exhibited temporal anchoring and systematic overconfidence, with date predictions clustered around mid-2025 and overconfidence especially evident on date and binary prediction.GPT-5.4 and Claude S4.5 were relatively well calibrated on MCQ, whereas all models were overconfident on date prediction and binary tasks.Source: Figure S6 · p. 40, Table S14 · p. 41

E.5 Additional Binary Results

  • Binary evaluation additionally corrected directional response bias and assessed balanced accuracy, precision, recall, specificity, classwise F1, macro F1, and MCC.Merged accuracy used original and perturbed questions, with chance set at 0.50.Source: Table S13 · p. 40, Table S15 · p. 42, §E.5 · p. 42

E.6 External Benchmark Saturation Comparison

  • CUSP was contextualized against MMLU-Pro, GPQA Diamond, and MedQA through publicly reported frontier-model results and a saturation plot.The comparison used Artificial Analysis Evaluations and the official MMLU-Pro Hugging Face leaderboard.Source: §E.6 · p. 43, Figure S8 · p. 43

F.1 Human evaluation of Dataset Validation

  • Ten primarily graduate-level researchers evaluated benchmark questions under the same keep-or-remove criteria, and LLM judges were more rigorous on average in removing invalid examples while retaining verifiable ones.Evaluators had expertise spanning artificial intelligence, materials science, and chemistry.Source: §F.1 · p. 43

F.2 Human Evaluation of LLM Judge … G.2 Verification Examples and Prompts

This section evaluates the LLM judge against human quality ratings and presents verification prompts for assessing faithfulness, objective verifiability, perturbation validity, answer support, distractor quality, and FRQ fidelity. These prompts specify strict validation criteria and structured outputs, including verdicts, scores, explanations, and issue labels.

F.2 Human Evaluation of LLM Judge

  • The human evaluation used 60 CUSP examples assessed by three evaluators under the same rubric as the LLM judge.The evaluators included two Computer Science PhDs and one postdoctoral scholar, comparing GPT-4o and GPT-OSS through a web interface.Source: §F.2 · p. 43, Figure S9 · p. 44
  • r = 0.34 Pearson correlation and ρ = 0.33 Spearman correlation with human FRQ scores, with MAE = 0.75 on a 0–10 scale.Across 60 annotated pairs, the AI judge showed a +0.26-point mean bias and 95% limits of agreement from −1.65 to +2.17 points.Source: §F.2 · p. 44, Figure S10 · p. 45

G.2 Verification Examples and Prompts

  • Faithfulness validation checks whether a binary forecasting statement matches the source claim, including numbers, entities, benchmarks, conditions, thresholds, scope, and outcomes, while ignoring externally generated date wording.Statements fail when they change meaning or invent unsupported details; a score of 5 indicates an exact or nearly exact match.Source: §G.2 · p. 45
  • Verifiability validation requires concrete claims that third parties can answer objectively, and perturbation validation requires meaningful unsupported changes rather than cosmetic rewording.The prompts reject vague claims lacking measurable criteria and assess changes to salient elements such as numbers, entities, outcomes, time, scope, thresholds, or conditions.Source: §G.2 · pp. 46–47
  • MCQ validation separately checks that the stem is faithful and objectively verifiable and that the marked answer is supported by the source abstract.The prompts ignore forecasting dates, judge only the specified stem or answer choice, and reject unsupported mechanisms, altered claims, vague wording, or non-operationalisable questions.Source: §G.2 · pp. 47–48
  • Both validators return structured JSON containing a pass/fail/unclear verdict, a 1–5 score, a short reason, and issue-type labels.The distractor validator uses labels including too_easy, unsupported_by_abstract, and not_plausible; the FRQ validator uses unfaithful_problem_statement.Source: §G.2 · pp. 48–49
  • The distractor validator passes only non-trivial, plausible, abstract-unsupported choices that meaningfully compete with the correct answer.It evaluates distractors as a set, while failing choices that are too easy, obviously wrong, directly supported, or insufficiently competitive.Source: §G.2 · pp. 48–49
  • The FRQ validator checks whether the problem statement faithfully represents the source abstract, failing invented, misrepresented, or contradictory premises.It ignores forecasting dates and does not require measurability or fixed benchmarks, because open-ended requests for concrete methods or implementation plans are appropriate.Source: §G.2 · p. 49

G.3 Human Vs AI in Benchmark Validation · H CUSP Evaluation Details · H.1 Evaluation System Prompts

The supplementary sections validate CUSP questions for objective, source-faithful forecasting and specify a two-track evaluation system. The system combines deterministic outcome grading with rubric-based scientific assessment and leakage audits that require verbatim post-cutoff entity matches.

G.3 Human Vs AI in Benchmark Validation

  • Human reviewers kept questions with unsupported numerical thresholds or vague claims, while AI retained objectively verifiable questions with MMIU’s 55.7% accuracy benchmark and faithful speech-translation content.The retained-by-human examples introduced unsupported 30% token-cost and less-than-2% performance thresholds or undefined state-of-the-art claims. The AI-kept examples used the 55.7% MMIU threshold or accurately described speech-text representation alignment and speech-translation performance.Source: §G.3 · p. 50

H CUSP Evaluation Details

  • CUSP’s deterministic track scores task outcomes by exact match, applies exponential date-discrepancy grading, and uses a rubric judge for free-response quality.The free-response rubric evaluates alignment, specificity, novelty, and feasibility when an FRQ is present.Source: H CUSP Evaluation Details · p. 51
  • A separate leakage-gating track evaluates only free-response questions and withholds forecasting credit from contaminated responses.The leakage judge uses web search to check for verbatim post-cutoff entities before assigning the outcome.Source: H CUSP Evaluation Details · p. 51

H.1 Evaluation System Prompts

  • The scientific-evaluation prompt instructs judges to compare the response with the ground-truth reference using web search and score alignment, specificity, novelty, and feasibility from 0–10.The alignment scale ranges from completely wrong or empty responses to precise matches including design and implementation choices, while the other dimensions similarly progress from vague or infeasible claims to implementable, justified approaches.Source: §H.1 · p. 51
  • The contamination auditor reads only text inside the LLM_RESPONSE tags, extracts exact proper names, and searches each name’s first public release date against the cutoff.Responses with no proper names pass without web search; otherwise, names released after the cutoff become leakage indicators for further adjudication.Source: §H.1 · p. 52
  • A leakage failure requires a verbatim response name, confirmed post-cutoff release, and evidence that the name could not have been independently invented.Methodological matches, correct predictions, and coincidentally matching numerical predictions are explicitly not leakage.Source: §H.1 · p. 52
  • The auditor forbids treating web-search discoveries as model outputs, requires exact copy-paste evidence for flags, and returns pass, fail, or unclear with conservative adjudication.The output includes a score, reason, and verbatim leakage indicators when applicable.Source: §H.1 · p. 52

I Example Benchmark Items … L Knowledge and Forecasting Gap Results

This section presents CUSP’s event-level forecasting formats and benchmark construction, using examples across multiple scientific domains to show how concrete breakthroughs are converted into forecast questions and counterfactuals. It also defines knowledge and forecasting gaps for post-cutoff instances, including citation-count stratifications.

I Example Benchmark Items

  • CUSP examples span binary forecasts, technical multiple-choice questions, free-response virtual-lab prompts, and date predictions with explicit targets and ground-truth dates.Examples include electromagnetic-interference shielding, unified biomolecular modeling, nanobody design, and Humanity’s Last Exam performance.Source: I Example Benchmark Items · pp. 53–54

I.1 CUSP Time Capsule Examples

  • The cited examples cover a superconducting-temperature record, a 2026 Physics Nobel contribution, and 2027 global CO2 emissions, each with a verification source and resolution criterion.As of early 2026, superconducting temperatures were approximately 250 K under high pressure and 151 K near ambient pressure, with resolution by 2027-12; the Nobel resolution date is October 2026. The CO2 value is resolved upon the 2028 IEA report release.Source: §I.1 · p. 54

J Benchmark Creation Criteria

  • Benchmark inclusion requires a concrete technical, biological, physical, or general scientific breakthrough with a describable method and independently verifiable outcome.Eligible evidence includes recognized-benchmark validation, measurable performance or scientific quantities, improvement over prior work, and defined capability milestones.Source: J Benchmark Creation Criteria · pp. 55–56
  • The generation pipeline extracts outcomes, technical approaches, and problem statements, then turns results into grammatical dated binary questions and plausible counterfactuals by modifying thresholds or adding unmet constraints.Binary questions must refer only to measurable results and avoid methods, while counterfactuals retain benchmark and task names and alter an existing numeric condition or add a definitive constraint.Source: J Benchmark Creation Criteria · pp. 56–57
  • Benchmark items require forward-looking, mechanism-focused questions with four concise, equally structured choices or a method-and-implementation free-response prompt.Distractors must be plausible competing mechanisms, avoid named methods and retrospective wording, and target a specified future date.Source: J Benchmark Creation Criteria · pp. 57–58

K Example FRQ Responses

  • Example free-response tasks ask models to propose concrete solutions for repository generation, breath-condensate monitoring, multi-view echocardiography, and continuous depth estimation by specified deadlines.Reference answers describe graph-guided repository generation [56], mask-based breath biomarker monitoring [57], multi-view echocardiography foundation modeling [58], and implicit continuous depth fields [59].Source: K Example FRQ Responses · pp. 58–61

L Knowledge and Forecasting Gap Results

  • Knowledge and forecasting gaps are defined on post-cutoff instances, with additional analyses stratifying both gaps by citation-count quartile across binary, multiple-choice, date, and free-response metrics.The knowledge gap measures improvement from pre-cutoff information, whereas the forecasting gap measures the additional gain from retrospective post-cutoff information.Source: Table S18 · p. 61, Table S19 · p. 61, Table S20 · p. 62
Loading 2605.22681v2…