PaperlayerView exact revision on arXiv

Source-linked AI summary

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

Sy-Tuyen Ho, Minghui Liu, Huy Nghiem, Furong Huang

arXiv:2605.30329v1cs.LG

TL;DR

Existing benchmarks rarely test whether language models can judge a research proposal’s methodological soundness before costly implementation. SoundnessBench evaluates this capability on 1,099 reconstructed ICLR proposals, finding that models often over-approve weak proposals and become unstable under stricter prompting.

  • Problem

    Existing benchmarks provide limited evidence about whether models can judge a research idea’s methodological soundness before researchers spend substantial labor and computational resources.

  • Method

    SoundnessBench reconstructs 1,099 ICLR machine-learning proposals, labels them using reviewer soundness sub-scores, and evaluates 12 frontier LLMs on results-masked proposals.

  • Results

    Across models, standard prompting produced frequent false approval of low-soundness proposals, with a mean false-positive rate of 74.0%, while aggressive prompting shifted errors toward false negatives.

  • Takeaways & Limitations

    Current frontier LLMs are not yet reliable standalone first-gate critics for machine-learning research proposals, so proposal-stage judgment may require calibration, targeted training, or human review.

  • Takeaways & Limitations

    The benchmark uses imperfect reviewer-derived proxies and covers a bounded slice of ICLR machine-learning research, not exact full-paper outcomes or scientific soundness generally.

Abstract

from arXiv · show

Autonomous AI research agents aim to accelerate scientific discovery by automating the research pipeline, from hypothesis generation to peer review. However, existing benchmarks rarely test a fundamental bottleneck: whether Large Language Models can judge the methodological viability of a research idea before expending time and computational resources. We introduce SoundnessBench, a curated benchmark of 1,099 machine-learning research proposals reconstructed from ICLR submissions, labeled with reviewer soundness sub-scores, and audited against source papers. SoundnessBench should be interpreted as a benchmark for recoverable proposal-stage soundness rather than exact prediction of full-paper review outcomes. Across 12 frontier LLMs, we find a pervasive optimism bias: under standard prompting, models frequently rate low-soundness proposals as sound, while aggressive prompting largely shifts errors from false positives to false negatives. Additional controls for public-corpus contamination, paper-identifying phrases, surface features, and human audit quality suggest that this behavior is not explained by a single confounder. Our results indicate that current LLMs are not yet reliable as standalone first-gate evaluators for scientific rigor.

1. Introduction

SoundnessBench targets the overlooked pre-execution judgment of whether machine-learning research proposals are methodologically sound. Across 12 frontier LLMs, it finds substantial optimism bias that persists after controls, limiting their reliability as standalone first-gate evaluators.

  • Motivation: Scientific triage is a first-gate decision: researchers must assess a proposal’s methodological soundness before spending substantial labor and computation.This judgment concerns whether an experimental design can rigorously test its hypothesis, not eventual impact, novelty, acceptance, or universal validity.
  • The Soundness Gap: Existing research-agent benchmarks mainly evaluate execution or post-hoc outcomes, leaving pre-execution methodological-soundness judgment largely untested.SoundnessBench combines pre-execution evaluation, direct methodological-soundness judgment, and proposal-only input.
  • Benchmark: 1,099 hypothesis–experiment proposals were curated from over 35,209 ICLR submissions and 137,940 expert reviews into a benchmark for proposal-stage soundness.The benchmark spans multiple years of top-tier ML submissions and 16 subfields, with scope explicitly limited to machine-learning research.
  • Main Results: 74.0% mean false-positive rate under standard prompting indicates pervasive optimism, whereas aggressive prompting reduces false positives to 19.9% but lowers high-soundness recall to 36.1%.The tested aggressive prompt shifts errors toward false negatives rather than resolving the evaluation problem.
  • Robustness: Controls for leakage, contamination, identifiers, surface features, writing quality, and adversarial injection do not eliminate the optimism bias.The controls include an ICLR 2026 split and a preliminary human audit of leakage and agreement with assigned labels.

2. SoundnessBench: Benchmark Reconstruction

SoundnessBench isolates pre-execution scientific rigor assessment by presenting LLMs with results-masked proposals and grounding soundness labels in audited ICLR reviewer judgments. Its five-step reconstruction yields 1,099 traceable proposals after filtering, labeling, extraction, and verification.

  • Overview and Motivation: SoundnessBench tests whether LLM agents can reject weak research designs before implementation, addressing benchmarks’ limited coverage of pre-execution rigor assessment.The benchmark targets the first research gate, before substantial engineering effort and compute are expended.
  • Challenges and Design Choices: The benchmark combines peer-review-grounded labels, proposal-only inputs, and evidence-audited extraction to improve task faithfulness and traceability.Models receive results-masked proposals, while each instance remains traceable to source evidence.
  • Benchmark Reconstruction: Mean reviewer soundness scores ≥3 define high-soundness pairs, scores ≤2 define low-soundness pairs, and middle-score cases are excluded.Labels use ICLR reviewer soundness sub-scores as proxies for recoverable proposal-stage methodological validity rather than absolute post-execution research quality.
  • Benchmark Reconstruction: 66.93% of candidates pass the verification filter, producing a benchmark of auditable hypothesis–experiment designs.The audit uses retrieval-backed atomic-claim support to check extraction faithfulness against source PDFs.
  • Benchmark Reconstruction: 1,099 research proposals comprise the final dataset, including 458 low-soundness and 641 high-soundness instances.The corpus spans major machine-learning areas across 16 subfields and begins with ICLR releases from 2022 because earlier releases lack consistent soundness-related scores.

3. Evaluation

Across 12 frontier LLMs, SoundnessBench reveals strong optimism bias under standard prompting: models often approve low-soundness proposals, while stricter prompting mainly trades false positives for false negatives. Robustness controls support interpreting this as a prompt-sensitive limitation in proposal-stage judgment rather than a single leakage, memorization, or surface-feature artifact.

  • Evaluation protocol: The evaluation classifies results-masked proposals as high or low soundness using fixed standard and aggressive prompts, targeting recoverable proposal-stage methodological soundness.The benchmark does not aim to predict exact full-paper review outcomes.
  • Standard prompting: 26.0% mean low-soundness recall under standard prompting corresponds to a 74.0% false-positive rate, while mean high-soundness recall is 91.8%.Nine of twelve models exceed a 70% false-positive rate on low-soundness proposals.
  • Standard prompting: GPT-5.4 and GPT-5.4-mini reduce optimism but have high-soundness recall of 74.6% and 76.6%, respectively, exposing a calibration trade-off.Stricter filtering suppresses false approvals while increasing rejection of promising ideas.
  • Robustness controls: Robustness analyses indicate the optimism bias is unlikely to arise solely from leakage, memorization, simple style features, or a narrow topic slice.Removing paper-identifying phrases changes the aggregate confusion matrix by roughly one percentage point, while simple baselines over-reject high-soundness proposals.
  • Overall interpretation: Across the evaluation, current LLMs are not yet reliable as standalone gatekeepers for scientific rigor in proposal-stage soundness judgment.The results support a calibrated interpretation of SoundnessBench as an imperfect but audited proxy.

4. Related Work

Prior work has developed autonomous AI research agents, evaluated LLM-generated ideas and completed-paper reviewing, and documented sycophancy and fragile judgment. SoundnessBench extends these lines of work toward proposal-stage methodological soundness, where important strengths or weaknesses may remain hidden before experimentation.

  • Autonomous AI Research Agents: Recent AI scientist agents automate parts of the research pipeline, including idea generation, coding, experimentation, and paper drafting.The AI Scientist, The AI Scientist-v2, and Agent Laboratory represent early systems, with Agent Laboratory organizing literature review, experimentation, and report writing.
  • Evaluating Research Agents and Idea Quality: LLM-generated ideas may be novel yet less feasible, while proposal promise can diverge further from outcomes after implementation.These findings motivate caution about proposal-only judgment because experiments can reveal weaknesses or strengths that were not previously visible.
  • LLMs as Peer Reviewers: Existing LLM-reviewer research primarily evaluates critiques and judgments of completed papers against human assessments.LLM assistance has also been reported to improve review clarity and usefulness in human review workflows.
  • LLM Sycophancy and Fragile Judgment: Studies of sycophancy show that LLMs can follow user framing over ground truth and validate plausible but weak claims.Efforts to suppress sycophancy may instead produce unstable skepticism or excessive refusal, while small prompt changes can alter model decisions.

5. Conclusion

SoundnessBench evaluates whether AI research agents can identify methodologically sound ML proposals before expensive experiments. The conclusion finds prompt-sensitive optimism and over-rejection, while positioning the benchmark as a recoverable pre-execution signal rather than an exact predictor of full-paper outcomes.

  • Contribution: SoundnessBench evaluates whether autonomous ML research agents can judge the methodological soundness of hypotheses and experimental designs before expensive experiments.It targets an early decision gate in the research pipeline.
  • Findings: Under standard prompting, many models over-approve weak proposals, whereas stricter prompting can push some models toward broad over-rejection.The conclusion characterizes judgments as sensitive to prompt framing.
  • Findings: Controls for public-corpus contamination, identifier recognition, surface features, and slice-level concentration do not remove the observed pattern.The result supports diagnosing calibration failures beyond these tested confounds.
  • Implications: At the early research gate, unstable judgments can misallocate compute and attention, motivating SoundnessBench as a tool for safer, more useful research agents.The benchmark is also presented as a way to diagnose where model calibration breaks down.
  • Limitations: SoundnessBench measures recoverable pre-execution soundness signals rather than exact full-paper review prediction or definitive post-execution research quality.Its ground truth uses reviewer soundness sub-scores, which are expert signals but imperfect proxies because reviewers saw full papers and related qualities.

Supplementary … A.4. Preliminary Human Audit of Extraction Quality and Label Validity

The supplementary material specifies a proposal-extraction pipeline that excludes outcomes, decomposes proposals into atomic claims, and verifies them against source evidence. A preliminary human audit independently checks leakage, acceptance clues, and soundness-label validity.

  • A.1. Prompt to Extract Proposal: The extraction prompt captures proposal-level hypotheses and experimental designs before execution while excluding results and conclusions.It requires structured JSON, source-grounded atomic claims, and proposal language such as “we propose” rather than outcome-revealing phrasing.
  • Related Work: Experiments are represented by distinct planned groups containing goals, datasets, models, baselines, setups, and evaluation metrics without metric values.The instructions preserve exact dataset, task, and baseline names and prohibit merging distinct experiments or revealing measured performance.
  • Related Work: Related-work extraction retains prior work, research gaps, explicit limitations, problem motivation, and proposed methods while excluding improvement claims and invented risks.The format also includes a paper title, short hypothesis, abstract, experiments, and explicit limitation sentences.
  • A.2. Prompt to Decompose Atomic Claims from Proposal: The proposal is decomposed into atomic, independently verifiable claims covering methods, hypotheses, planned evaluations, datasets, tasks, baselines, and metrics.Claims must be complete, literal, close to source wording, and free of synthetic experiment labels or commentary.
  • A.2. Prompt to Decompose Atomic Claims from Proposal: Result and outcome claims, including improvement, outperforming, and state-of-the-art assertions, are excluded from atomic-claim decomposition.This preserves the distinction between a research plan and confirmed findings.
  • A.3. Prompt to Verify Atomic Claims: Each claim–evidence pair receives an explicit YES/NO verification judgment from a rigorous fact-checking prompt using provided paper excerpts.Gemini-2.5-Flash performs this verification, returning a one-sentence explanation with each decision.
  • A.4. Preliminary Human Audit of Extraction Quality and Label Validity: Two annotators independently audit a balanced subset of low- and high-soundness proposals for result leakage, outcome language, acceptance clues, and label support.The audit is a preliminary quality-control check rather than a full expert-ceiling study; disagreements and unsupported labels guide future filtering.

A.5. Example of Extracted Proposals … B.1. Evaluation Prompt Templates

The supplementary material illustrates high- and low-soundness extracted proposals, verifies their atomic claims, and publishes the standard and aggressive evaluation prompts. The examples pair methodological detail and limitations with soundness labels, while the prompts require structured JSON classifications under either ordinary or stricter evidence standards.

  • A.5. Example of Extracted Proposals: A.5. Example of Extracted Proposals contains 641 high-soundness and 458 low-soundness instances, with proposals formatted following Yamada et al. (2025).The section provides one example from each soundness category.
  • A.5. Example of Extracted Proposals: The high-soundness Gaussian-process proposal studies parameter identifiability for holomorphic kernels and reports Soundness 3.25, Support Ratio 1.0, and a high rigor bucket.Its stated limitations include that identifiability does not guarantee maximum-likelihood consistency and that extensions to non-stationary or non-holomorphic kernels remain open.
  • A.5. Example of Extracted Proposals: The low-soundness proposal evaluates an absolute-value neural activation against ReLU and Leaky ReLU on MNIST classification and autoencoder generation tasks.The classification experiment varies batch sizes 32, 64, and 128, while the generation experiment compares image quality against a Leaky ReLU baseline.
  • A.5. Example of Extracted Proposals: Its limitation notes that the proposed individualization may cause unstable generalization, with accuracy potentially varying across datasets; the example is labeled Soundness 1.0 and low rigor.The reported Support Ratio is 1.0.
  • A.6. Example of Extracted Atomic Claims and Verification: A.6. Example of Extracted Atomic Claims and Verification decomposes the Gaussian-process proposal into 17 atomic claims, all verified as supported, yielding a support ratio of 1.0.The claims cover the holomorphic-kernel framework, applicable kernel families and combinations, the Matérn-related gap, and synthetic-data evaluation.
  • B.1. Evaluation Prompt Templates: B.1. Evaluation Prompt Templates require models to provide a scientific rationale before assigning a low-or-high rigor bucket and confidence from 1 to 5.The standard prompt defines high rigor through clear hypotheses, controls, baselines or ablations, suitable metrics, and rigorous experiments, with JSON output constraints.
  • B.1. Evaluation Prompt Templates: The aggressive prompt changes the decision rule to default to low unless concrete evidence demonstrates strong controls, baselines or ablations, valid metrics, and a credible evaluation plan.It retains the same low/high classification and 1–5 confidence range while explicitly penalizing missing or weak methodological evidence.

B.2. Does Scale Resolve the Optimism Bias? · B.3. Is the Optimism Bias Caused by Instruction Tuning?

Scaling does not resolve Qwen3.5’s optimism bias: standard-prompt scaling improves high-soundness recall while worsening low-soundness recall, and aggressive prompting shows no consistent size benefit. Base and instruction-tuned variants behave similarly under standard prompting, while instruction tuning offers only modest robustness under aggressive prompting.

  • B.2. Does Scale Resolve the Optimism Bias?: Six Qwen3.5 models spanning 2B to 122B parameters were evaluated under standard and aggressive prompting to isolate scale effects.The comparison controls for model family and training recipe.
  • B.2. Does Scale Resolve the Optimism Bias?: 71.8% to 92.8% high-soundness recall improves monotonically from 2B to 35B under standard prompting, while 31.0% to 19.2% low-soundness recall decreases.Larger models therefore become more permissive toward weak proposals rather than correcting optimism.
  • B.2. Does Scale Resolve the Optimism Bias?: Under aggressive prompting, high-soundness recall ranges from 1.4% at 4B to 32.4% at 27B, with no monotonic trend across model sizes.All six models tend toward near-always-low behavior, while 27B achieves better balance than smaller and larger variants.
  • B.2. Does Scale Resolve the Optimism Bias?: The optimism bias is not a small-model artifact: it persists with scale under standard prompting and remains unresolved under aggressive prompting at every tested size.The results indicate that reliable soundness judgment likely requires interventions beyond simply increasing model size.
  • B.3. Is the Optimism Bias Caused by Instruction Tuning?: Base and instruction-tuned Qwen3.5-35B-A3B variants were compared under both prompting conditions to test whether instruction tuning or RLHF causes optimism.The hypothesis is that instruction tuning or RLHF may encourage agreeableness and avoidance of negative feedback.
  • B.3. Is the Optimism Bias Caused by Instruction Tuning?: Under standard prompting, the base model records 19.0% low-soundness recall and 92.5% high-soundness recall, versus 19.2% and 92.8% for the instruction-tuned model.The near-indistinguishable results suggest instruction tuning is unlikely to be the sole driver of optimism.
  • B.3. Is the Optimism Bias Caused by Instruction Tuning?: Under aggressive prompting, the base model reaches 98.7% low recall and 3.4% high recall, compared with 89.3% low recall and 17.2% high recall for the instruction-tuned variant.Both shift toward over-conservatism, but instruction tuning provides modest robustness to prompt pressure without preserving calibrated judgment.
  • B.3. Is the Optimism Bias Caused by Instruction Tuning?: Resolving optimism may require interventions at or before pre-training, such as targeted training on scientific judgment, rather than post-hoc instruction tuning alone.Instruction tuning improves robustness under aggressive prompting but does not eliminate the underlying calibration problem.

B.4. Controls for Public-Corpus Contamination

The benchmark treats public-corpus contamination as a validity threat that cannot be fully eliminated. Accordingly, SoundnessBench is presented as a reduced-contamination-risk analysis, interpreted alongside additional controls and future private or continuously updated variants.

  • Contamination risk: Public ICLR submissions and OpenReview discussions may have exposed evaluated models to source papers during pretraining or supervised fine-tuning.The paper treats this potential exposure as a threat to validity rather than claiming contamination is fully eliminated.
  • Contamination risk: Perfect contamination control is difficult for any benchmark built from public corpora.
  • Interpretation: SoundnessBench is therefore a reduced-contamination-risk analysis rather than a contamination-free benchmark.The authors recommend interpreting it together with the 2026 split, identifier-removal analysis, and future private or continuously updated benchmark variants.

B.5. Identifier-Removal Robustness · B.6. Surface-Feature Baselines

Masking method names, framework names, and self-identifying phrases changes the aggregate confusion matrix by only roughly one percentage point. Surface-feature and trivial baselines diverge from LLM behavior, indicating that shallow structural statistics alone do not explain the optimism bias.

  • B.5. Identifier-Removal Robustness: Paper titles are excluded from the main evaluation, and an anonymized variant masks method names, framework names, and self-naming phrases while preserving scientific meaning.The intervention targets recognizable identifiers and self-identifying language without changing the hypothesis or experiment design.
  • B.5. Identifier-Removal Robustness: ~1 percentage point: masking paper-identifying phrases has a negligible effect on the aggregate confusion matrix.Table 5 reports mean prediction percentages for this intervention.
  • B.6. Surface-Feature Baselines: Always-high and always-low baselines predict every proposal as high or low soundness, respectively.These trivial baselines test whether label priors alone can reproduce the observed behavior.
  • B.6. Surface-Feature Baselines: Mean-threshold classifiers use proposal length, experiment count, and risk-factor count, with thresholds set midway between per-class means.These no-training baselines probe obvious structural features without fitting evaluation labels.
  • B.6. Surface-Feature Baselines: The baselines avoid test-label model selection, while a fully supervised text classifier is reserved for a training-split extension.The supervised classifier would measure learnable signal from labeled SoundnessBench examples rather than the intended no-training control.
  • B.6. Surface-Feature Baselines: Structural thresholds over-reject high-soundness proposals, whereas LLMs over-approve low-soundness proposals, producing qualitatively opposite errors.The comparison with the average LLM under standard prompting is summarized in Table 6; richer supervised confounder models remain a future control.

B.7. Adversarial Injection of Methodological Flaws · B.8. Breakdowns by Year, Subfield, and Writing Quality

Adversarially injected hypothesis–experiment mismatches sharply reduce model approval, showing that models can detect severe methodological inconsistencies. Yet optimism bias persists across years, subfields, and writing-quality bands, indicating a systemic failure on subtler naturally occurring flaws.

  • B.7. Adversarial Injection of Methodological Flaws: 77.0% to 1.0%: GPT-5.4’s approval rate collapses when dataset names and evaluation metrics are replaced to create severe hypothesis–experiment mismatches.The hypothesis and other content remain fixed during injection.
  • B.7. Adversarial Injection of Methodological Flaws: The adversarial result rules out the strongest version of the claim that models ignore methodological content entirely.Models can detect severe methodological inconsistencies.
  • B.7. Adversarial Injection of Methodological Flaws: Models remain insufficiently critical of subtler naturally occurring flaws in low-soundness proposals.The control is preliminary; future versions will test missing baselines and data leakage.
  • B.7. Adversarial Injection of Methodological Flaws: The adversarial experiment is reported as an initial control rather than a comprehensive test of methodological flaw types.The authors plan to extend it to additional flaws, including missing baselines and data leakage.
  • B.8. Breakdowns by Year, Subfield, and Writing Quality: Low-soundness false-positive rates show optimism bias across all examined year slices.Table 8 reports the low-soundness false-positive rate by year.
  • B.8. Breakdowns by Year, Subfield, and Writing Quality: Optimism bias also appears across subfields, including analyses restricted to subfields with more than 10 pairs.Table 9 reports low-soundness false-positive rates by subfield under this restriction.
  • B.8. Breakdowns by Year, Subfield, and Writing Quality: The bias spans writing-quality bands as well as years and subfields, suggesting it is systemic rather than concentrated in a narrow benchmark subset.Table 10 reports the writing-quality breakdown.

B.9. Qualitative Examples of False and True Positives

Qualitative examples show that all three evaluated models can assign high-soundness judgments to proposals with reviewer-identified methodological weaknesses, illustrating optimism-bias false positives. They also correctly identify a high-soundness proposal whose method and evaluation plan received favorable reviewer assessment.

  • False positives: Low-soundness proposals were predicted as high soundness by Claude Opus 4.6, GPT-5.4 Thinking, and Gemini 3.1 Pro despite concerns about comparisons, missing analyses, and novelty.This first false-positive example illustrates the optimism-bias failure mode.
  • False positives: A second low-soundness proposal was likewise predicted as high soundness by all three models despite overclaims, insufficient experimental support, and evaluation-task pretraining data weakening the comparison.The example shows that false positives can persist when reviewer concerns target both evidence and evaluation validity.
  • True positives: A high-soundness proposal was correctly predicted as high soundness by Claude Opus 4.6, GPT-5.4 Thinking, and Gemini 3.1 Pro, matching favorable reviewer assessment of its method and evaluation plan.This example represents a true positive rather than an optimism-bias error.
Loading 2605.30329v1…