Source-linked AI summary
Predicting Empirical AI Research Outcomes with Language Models
Jiaxin Wen, Chenglei Si, Yueh-han Chen, He He, Shi Feng
TL;DR
Predicting which empirical research ideas will work could help prioritize experiments that otherwise require substantial labor and compute to validate. The paper builds a benchmark and a retrieval-augmented, fine-tuned language-model system; it reports 77% test accuracy and outperforms human experts on an NLP subset. It also tests the system on unpublished ideas and identifies limits to its current black-box use.
Problem
Predicting which research idea will perform better without implementing the ideas could help prioritize experiments whose validation costs human labor and computational resources.
Source: §1 · p. 2Method
The paper builds a human-verified benchmark from conference-paper idea pairs and combines a fine-tuned GPT-4.1 with a paper retrieval agent.
Source: §1 · p. 2Results
The system achieves 77% test-set accuracy and, on the NLP subset, scores 64.4% versus 48.9% for the human-expert baseline.
Source: §1 · p. 2Takeaways & Limitations
The system’s 63.6% accuracy on unpublished projects supports its potential use as an idea ranker or reward model for AI-generated ideas.
Source: §1 · p. 2The system is a black box, limits human-AI cooperation to idea prioritization, and may rely on spurious features despite robustness checks.
Source: §8 · p. 9
Abstract
from arXiv · showhide
Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial for accelerating empirical AI research, a skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare LMs with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. We scrape ideas and experimental results from conference papers, yielding 1,585 human-verified idea pairs published after our base model's cut-off date for testing, and 6,000 pairs for training. We then develop a system that combines a fine-tuned GPT-4.1 with a paper retrieval agent, and we recruit 25 human experts to compare with. In the NLP domain, our system beats human experts by a large margin (64.4% v.s. 48.9%). On the full test set, our system achieves 77% accuracy, while off-the-shelf frontier LMs like o3 perform no better than random guessing, even with the same retrieval augmentation. We verify that our system does not exploit superficial features like idea complexity through extensive human-written and LM-designed robustness tests. Finally, we evaluate our system on unpublished novel ideas, including ideas generated by an AI ideation agent. Our system achieves 63.6% accuracy, demonstrating its potential as a reward model for improving idea generation models. Altogether, our results outline a promising new direction for LMs to accelerate empirical AI research.
1 Introduction
The paper frames empirical-idea outcome prediction as a way to prioritize costly experiments, then builds and tests a retrieval-augmented, fine-tuned language model. It reports benchmark and expert comparisons, robust behavior on stress tests, and accuracy on unpublished ideas.
1 Introduction
- Predicting which idea will perform better on benchmarks could help prioritize experiments and refine ideas before incurring implementation costs.The paper frames this as more actionable than evaluating subjective qualities such as novelty or excitement [14] [9] [4].Source: §1 · p. 2
- 77% test-set accuracy; on 45 NLP idea pairs, the system scored 64.4% versus 48.9% for the expert baseline.The NLP baseline aggregated predictions from ensembles of five experts, who collectively spent over 45 minutes per prediction.Source: §1 · p. 2
- On human-designed and LM-proposed stress tests, the system showed robust behavior against superficial features such as idea recency and complexity.The tests included three human-designed tests and hundreds proposed by language models.Source: §1 · p. 2
- 63.6% accuracy on 35 unpublished projects, including AI-generated ideas, supports testing the system as an idea ranker or reward model.The projects had experiments implemented by NLP researchers and had never been publicly released online.Source: §1 · p. 2
2 Benchmark
The benchmark turns empirical idea comparisons into pairwise prediction examples with labels derived from benchmark outcomes. Its construction controls comparison conditions and includes human verification of extracted examples.
2 Benchmark
- Each example compares two research ideas for a research goal, with a binary label for which idea wins across multiple benchmarks.The paper uses pairwise evaluation to reduce ambiguity and noise from evaluating individual ideas [14] [9] [4].Source: §2 · p. 2, Figure 2 · p. 3
2.1 Benchmark Construction
- The extraction pipeline collects empirical conference papers, summarizes ideas without including results, pairs ideas only within papers, and labels winners by majority vote across benchmarks.Pairs tied or evaluated on fewer than three benchmarks are removed; within-paper pairing avoids confounds such as differing hyperparameters or preprocessing.Source: §2.1 · p. 3
- The train/test split uses GPT-4.1’s June 1, 2024 cutoff so test ideas include work published after the model’s knowledge cutoff.Data statistics are reported in Table 2, but the supplied passage does not provide the table values.Source: §2.1 · p. 3
2.2 Human Verification of Extracted Examples
- Human annotators checked extracted examples against original PDFs, correcting win-condition errors and removing fabricated or incomparable results; cross-validation mislabels fell from 11% to 2.5%.The 16 annotators were trained and experienced in reading AI papers, while the expert-baseline annotators had greater domain expertise.Source: §2.2 · pp. 3–4
3 Building a System for Research Outcome Prediction
The system retrieves literature for indirect or transferable insights about novel ideas, summarizes relevant papers, and fine-tunes a language model to predict outcomes. Full-paper summaries improve accuracy over reusing abstracts.
3.1 Retrieval
- For novel idea pairs, retrieval queries target transferable insights or related sub-components rather than searching directly for identical comparisons.The agent iteratively decides whether it has enough information and generates distinct queries when further research is needed.Source: §3.1 · p. 4
- On 600 test examples, full-paper summaries raised accuracy from 38.8% to 53.0% compared with reusing abstracts.The authors report that abstracts alone bias predictions toward old ideas, while full-paper summaries alleviate that bias.Source: Table 3 · p. 4, §3.1 · p. 5
- The system retrieves up to 15 relevant papers per query from arXiv, limiting results to papers published before June 1, 2024, to prevent information leakage.It filters initially retrieved papers using a language model’s relevance judgments.Source: §3.1 · pp. 4–5
3.2 Fine-tuning
- Fine-tuning trains a language model to predict outcome labels from the research goal, idea pair, and retrieval results.The authors also explore generating and selecting self-produced chain-of-thought examples because high-quality task-specific CoTs are unavailable.Source: §3.2 · p. 5
4 Evaluation on Human-written Ideas
The evaluation compares retrieval-augmented language models with fine-tuned models and human researchers on predicting which research idea performs better. Fine-tuned GPT-4.1 outperforms expert researchers on the NLP subset, while the tested prediction task remains difficult for both people and off-the-shelf models.
4.1 Experiment Setup
- Predictions count as correct only when swapping the order of the two ideas produces consistent answers; inconsistent predictions count as incorrect.The evaluation uses GPT-4.1 and other frontier models whose knowledge cutoffs precede June 1, 2024.Source: §4.1 · p. 5
4.2 Automatic Evaluation
- Fine-tuning GPT-4.1 raises accuracy on future ideas from 51.9% to 77%, while GPT-3.5-turbo rises only from 45.0% to 53.7%.The authors suspect GPT-4.1’s stronger reasoning and more recent AI-research knowledge contribute to its performance; they also report limited GPT-3.5 gains as evidence against trivial shortcuts.Source: §4.2 · p. 6
- Neither randomly sampled nor LM-selected self-augmented chain-of-thoughts improves on the fine-tuning baseline.The authors suspect GPT-4.1’s near-chance performance produces unsound generated reasoning that offers little useful fine-tuning signal.Source: §4.2 · p. 6
4.3 Comparison with Expert Researchers
- Researchers evaluated 45 NLP idea pairs across five topics, producing 222 valid annotations after excluding three labels from annotators who had read both ideas.Five researchers with relevant backgrounds were recruited per topic; 68% had published on their assigned topics.Source: §4.3 · p. 6
- 64.4% accuracy for the system exceeds majority-vote NLP experts’ 48.9% and the 60.0% ceiling human baseline.The ceiling baseline aggregates the best-performing researcher per topic, selected on the same test set.Source: §4.3 · p. 6
5 Stress-testing Our System’s Robustness
The paper tests whether the system relies on superficial idea features through hand-designed and LM-designed robustness checks. Accuracy is generally robust across the tested features, though the system shows some sensitivity to complexity and specialization.
5 Stress-testing Our System’s Robustness
- The stress tests compare accuracy on subsets where a feature is supported versus unsupported, testing whether performance depends on superficial features.The authors hand-design three tests and use an LM to propose more, aiming to test biases including idea complexity and stylistic cues.Source: §5 · p. 6, §5.1 · p. 7
5.1 Human-Designed Tests
- The hand-designed tests find no sensitivity to idea recency, a slight length bias, and resilience when famous lab names are added to losing ideas.For length, the system can still identify complicated but ineffective ideas.Source: §5.1 · p. 7
5.2 LM-Designed Tests
- LM-designed tests generate hypotheses about differences between model-preferred and model-dispreferred ideas, verify them across the test set, then split examples into supported and unsupported subsets.The pipeline samples five idea pairs at a time to propose hypotheses and filters verified hypotheses to validity scores between 25% and 75%.Source: §5.2 · p. 8
- 68.5% average unsupported-subset accuracy across 289 verified hypotheses, with 255 (over 88%) cleared under the authors’ 62% threshold.The table reports results on LM-proposed hypothetical biases; a hypothesis is flagged when unsupported-subset accuracy falls below 62%.Source: §5.2 · p. 8, Table 6 · p. 8
- Even across the 34 flagged hypotheses, average unsupported accuracy is 61.3%; complex or specialized ideas are among the challenging cases.The authors describe predicting complex or specialized but ineffective ideas as inherently more challenging, while reporting accuracy above 61% on these tests.Source: Figure 5 · p. 7, §5.2 · p. 8
6 Evaluation on Novel Unpublished Ideas
The authors test their system on unpublished ideas with empirical outcomes, including projects generated by an AI ideation study. It reaches 63.6% accuracy on 33 labeled idea pairs, supporting its potential as a reward model.
6 Evaluation on Novel Unpublished Ideas
- The evaluation addresses a key bottleneck: testing predictions on unpublished ideas requires obtaining ground truth by implementing them.Researchers implemented the ideas at an average cost of 103.4 hours per idea.Source: §6 · p. 8
- The dataset began with 35 unpublished ideas from an AI ideation study, half generated by its language-model agent and half by recruited NLP experts.The ideas concern novel prompting methods to improve language-model capabilities.Source: §6 · p. 8
- After pairing each idea with its baseline and excluding ties, the authors retained 33 labeled idea pairs.Labels were derived from empirical outcomes documented in papers describing the experiments.Source: §6 · p. 9
- 63.6% accuracy on unpublished ideas supports the system’s potential as a reward model in an automated research pipeline.The authors suggest using it to guide sampling at inference time or optimize idea-generation models with reinforcement learning.Source: §6 · p. 9
7 Related Work
The paper distinguishes predicting empirical outcomes from peer-review-style evaluation and prior outcome-prediction setups. It frames its approach as forecasting and draws on prior work using retrieval and scale.
Accelerating Research with AI
- Prior work has explored AI across literature review, ideation, validation, writing, and review, but selecting promising ideas for execution remains costly.The passage describes one fully AI-generated paper accepted at an ICLR workshop after manual selection of three from 256 candidates.Source: §7 · p. 9
Research Evaluation
- Unlike peer-review-style evaluation, this work predicts objective empirical effectiveness, whose ground truth can be established reliably.The authors note that review scores can be subjective and may favor mathematically complex but ineffective ideas.Source: §7 · p. 9
Research Outcome Prediction
- The task predicts which of two independently written idea summaries without empirical results will be more effective, rather than detecting altered results in published abstracts.The prior setup cited in the passage achieved over 80% accuracy with fine-tuned Llama-2-7B, but the authors characterize it as easier.Source: §7 · p. 9
8 Discussion
The discussion presents the system as a tool for prioritizing research ideas while acknowledging limits to interpretability and robustness. It also identifies advanced modeling and reward-model use as future directions.
8 Discussion
- The current system is a black box, limiting its use in human-AI cooperation to prioritizing research ideas, and possible reliance on spurious features remains unresolved.Source: §8 · p. 9
- Future work could investigate advanced modeling such as inference-time experiment simulation, and use the system as a reward model for automated ideation.The authors describe their current fine-tuning method as straightforward but effective.Source: §8 · p. 9
- The conclusion reports that the system outperforms human experts on direct predictions of AI research outcomes and generalizes to novel unpublished ideas, including AI-generated ones.The authors also report robustness to biases such as recency.Source: §8 · p. 9
A Human Verification of Test Examples
Table 7 presents typical errors made by human annotators during test-example verification.
A Human Verification of Test Examples
- Table 7 presents typical errors found by human annotators during verification of test examples.Source: A Human Verification of Test Examples · p. 11
B LM-Designed Tests
Table 8 presents examples of LM-proposed hypotheses.
B LM-Designed Tests
- Table 8 presents examples of LM-proposed hypotheses.Source: Table 8 · p. 12