Source-linked AI summary
Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
Chenglei Si, Diyi Yang, Tatsunori Hashimoto
TL;DR
The paper asks whether LLMs can generate novel, expert-level research ideas, a capability not established by prior evaluations. It conducts a controlled head-to-head study using standardized ideas and blind reviews by over 100 NLP researchers, finding that AI ideas are judged more novel than human ideas while exposing limitations in self-evaluation and generation diversity.
Problem
Prior work does not establish whether LLMs can produce novel, expert-level research ideas, the first step toward autonomous research.
Method
The study compares human and LLM ideas using standardized topics and review criteria, blind expert evaluation, and an LLM agent with retrieval, generation, and ranking.
Results
AI ideas receive higher novelty ratings than human ideas, while remaining comparable on other metrics; reviewers’ overall scores correlate strongly with novelty and excitement.
Takeaways & Limitations
The findings provide statistically significant evidence that current LLMs can generate ideas judged more novel than expert-written ideas.
Takeaways & Limitations
Naively scaling idea generation has significant limitations, including failures of LLM self-evaluation and limited generation diversity.
Abstract
from arXiv · showhide
Recent advancements in large language models (LLMs) have sparked optimism about their potential to accelerate scientific discovery, with a growing number of works proposing research agents that autonomously generate and validate new ideas. Despite this, no evaluations have shown that LLM systems can take the very first step of producing novel, expert-level ideas, let alone perform the entire research process. We address this by establishing an experimental design that evaluates research idea generation while controlling for confounders and performs the first head-to-head comparison between expert NLP researchers and an LLM ideation agent. By recruiting over 100 NLP researchers to write novel ideas and blind reviews of both LLM and human ideas, we obtain the first statistically significant conclusion on current LLM capabilities for research ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility. Studying our agent baselines closely, we identify open problems in building and evaluating research agents, including failures of LLM self-evaluation and their lack of diversity in generation. Finally, we acknowledge that human judgements of novelty can be difficult, even by experts, and propose an end-to-end study design which recruits researchers to execute these ideas into full projects, enabling us to study whether these novelty and feasibility judgements result in meaningful differences in research outcome.
1 Introduction
The paper asks whether LLMs can generate expert-level research ideas, addressing the lack of large-scale, controlled evidence for this first step of autonomous research. It compares human and LLM ideas through standardized, blind expert evaluation.
- Motivation: LLM research ideation remains untested despite progress in scientific applications such as mathematical problem solving, proof writing, retrieval, coding, and pattern discovery.The paper frames ideation as a more creative and challenging capability than these supporting applications.
- Main finding: AI ideas are judged significantly more novel than human ideas, while the conditions are comparable on other reported metrics.The comparison uses blind expert review rather than lower-cost evaluation surrogates.
- Evaluation challenge: Evaluating research ideas is difficult because expert researchers are hard to recruit at scale and quality criteria are subjective.Even experts may struggle to judge idea quality reliably.
- Study design: The study recruits over 100 NLP researchers to write human baseline ideas and conduct blind reviews of human and LLM ideas.The design includes 49 ideas from each condition and 79 expert reviewers.
- Study design: Human and LLM ideas are matched on topic distribution and standardized in style to reduce confounding variables during comparison.The agent also uses retrieval augmentation and inference-time overgeneration with reranking.
2 Problem Setup
The experiment separates idea content, writeup, and expert evaluation, then controls topic, proposal format, and review criteria to compare human and LLM ideas fairly. Three conditions distinguish human ideas, automatically ranked AI ideas, and human-reranked AI ideas.
- Experiment design: Research idea evaluation is defined as three components: the idea itself, the writeup communicating it, and expert evaluation of that writeup.The design focuses on confounders across all three components.
- Ideation scope: The study defines seven NLP topics and gives human and LLM participants identical topic descriptions, templates, and demonstration examples.Human participants may select a preferred topic from the common list.
- Idea writeup: Both groups use a proposal template covering the title, problem, motivation, method, experiment plan, examples, and fallback plan.The template is intended to standardize content and detailedness across writeups.
- Review and evaluation: Reviews score novelty, excitement, feasibility, expected effectiveness, and overall quality on calibrated 1-10 scales.Reviewers also provide free-text rationales for the numerical scores.
- Experiment conditions: The comparison includes human-written ideas, directly selected top-ranked AI ideas, and AI ideas manually selected by a human expert from the agent’s generations.The human reranking condition estimates an upper-bound quality of AI ideas relative to automatic ranking.
3 Idea Generation Agent
The ideation agent combines paper retrieval, candidate generation, and idea ranking in a minimalist pipeline designed to test current LLM ideation capabilities. It uses retrieval augmentation, repeated generation, deduplication, and pairwise ranking to select proposals.
- Agent architecture: The agent has three components: paper retrieval, idea generation, and idea ranking.The design intentionally prioritizes measuring LLM capability over agent novelty.
- Paper retrieval: Retrieval-augmented generation grounds ideation in topic-relevant papers obtained through LLM-generated Semantic Scholar API calls.The agent retains and reranks retrieved papers using relevance, empirical focus, and inspiration criteria.
- Idea generation: The agent generates many candidate ideas, prepends titles and abstracts from 10 retrieved papers, and asks the LLM to avoid repeating prior ideas.The pipeline then deduplicates the resulting candidate pool.
- Idea ranking: The automatic ranker is trained as a proxy from 1,200 ICLR 2024 language-model submissions and their review outcomes.The authors found pairwise judgments more useful than direct score or decision prediction.
- Idea ranking: 71.4% accuracy is achieved by Claude-3.5-Sonnet in zero-shot accepted-versus-rejected pair prediction, exceeding GPT-4o’s 61.1% and Claude-3-Opus’s 63.5%.The chosen ranker assigns proposal scores through repeated Swiss-system pairwise comparisons.
- Reranking comparison: 32 of 49 human-reranked AI ideas differ from the automatically selected AI ideas, showing disagreement between the LLM ranker and human reranking.The two conditions overlap on the remaining 17 ideas.
4 Expert Idea Writing and Reviewing
The human study recruits a highly qualified NLP participant pool to generate and review ideas under balanced, anonymized assignments. Writers and reviewers contribute substantial effort, while each idea receives multiple independent reviews.
- Participants: The study recruits 49 expert idea writers and 79 expert reviewers, totaling 104 participants across the two tasks.Twenty-four reviewers also wrote ideas, but no reviewer evaluated their own idea.
- Expert qualifications: Idea writers come from 26 institutions, while reviewers come from 32 institutions and are mostly PhD students and postdocs.The participant pool is characterized using publication, citation, and related Google Scholar metrics.
- Expert qualifications: 72 of 79 reviewers have previously reviewed for major AI conferences or journals.The authors use this and other participant statistics to characterize the reviewers as experienced.
- Idea writing: Writers spend an average of 5.5 hours producing 902-word ideas and report topic familiarity of 3.7 and task difficulty of 3 on 1-5 scales.These statistics indicate substantial effort on the writing task.
- Review assignment: Reviewers select two preferred topics and a reviewing load of 2 to 7, then receive anonymized ideas with balanced human and AI assignments.Each idea is reviewed by 2 to 4 reviewers, with cross-institution safeguards against contamination.
- Review quality: Reviewers report average topic familiarity and review confidence of 3.7 out of 5, spending 32 minutes per review and writing about 232 words.The confidence level matches the reported average for relevant ICLR 2024 reviews.
5 Main Result: AI Ideas Are Rated More Novel Than Expert Ideas
Across three statistical tests, AI-generated ideas were judged more novel than human expert ideas, while remaining comparable on other evaluated metrics.
- Main finding: AI ideas received higher novelty scores than human ideas consistently across three statistical tests.The analyses account for review-level, idea-level, and reviewer-level datapoint choices.
- Test 1: 5.64±1.76 for AI Ideas and 5.81±1.66 for AI Ideas + Human Rerank exceeded 4.84±1.79 for Human Ideas on overall score in Test 1.Comparisons used two-tailed Welch’s t-tests with Bonferroni correction.
- Test 1: AI ideas were significantly better than human ideas in novelty and excitement while comparable on all other metrics in Test 1.The table caption reports the metric-level comparison for review-level analysis.
- Test 2: AI ideas were significantly better than human ideas in novelty while comparable on all other metrics in Test 2.Test 2 averaged scores by idea before comparison.
- Test 3: Reviewer-level analysis also found AI ideas rated more novel than human ideas, with p<0.05 for both AI conditions.Test 3 compared within-reviewer condition differences using one-sample t-tests.
6 In-Depth Analysis of the Human Study
The human-study analysis examines idea-writing behavior, metric relationships, and reviewer agreement, finding that overall scores track novelty and excitement more than feasibility and that agreement is limited.
- Human idea generation: 37 of 49 participants generated their ideas during the study, while 12 had them beforehand.Participants also rated their submitted ideas relative to past research ideas.
- Metric relationships: The overall score correlates with novelty at r=0.725 and excitement at r=0.854, but has almost no correlation with feasibility at r<0.1.These correlations suggest reviewers’ overall assessments align more closely with novelty and excitement than feasibility.
- Reviewer agreement: Reviewer agreement was 56.1%, above random agreement of 50% but below the 66.0% NeurIPS 2021 baseline and 71.9% ICLR 2024 baseline.The study attributes the lower agreement most likely to the subjectivity of evaluating ideas without experiment results.
- Reviewer agreement: The authors characterize idea review as inherently subjective, potentially more so than reviewing executed papers.This motivates examining inter-reviewer agreement directly.
7 Limitations of LLMs
The agent’s over-generate-and-rank strategy is constrained by duplicate ideas and unreliable automatic evaluation, limiting inference-time scaling and confidence in LLM-as-a-judge.
- Idea diversity: Only 200 of 4000 generated seed ideas were non-duplicate, as newly generated batches became increasingly repetitive.Duplicates were identified using cosine similarity above 0.8 with previously generated ideas on the same topic.
- Idea diversity: The accumulated number of non-duplicate ideas eventually plateaued, so generating more candidates primarily produced repetitions.This creates a bottleneck for inference-time scaling.
- Automatic evaluation: The authors report that LLM evaluators had lower agreement than expert reviewers’ scores.This challenges the reliability of using LLM-as-a-judge for research ideas.
- Automatic evaluation: The best LLM evaluator, a Claude-3.5 pairwise ranker, achieved only 53.3% accuracy, below the 56.1% inter-reviewer consistency.The comparison concerns ranking top and bottom 25% of collected human and AI ideas.
8 Qualitative Analysis and Examples
Qualitative reviews portray human ideas as more grounded and feasibility-oriented, while AI ideas are often more novel but can be vague, poorly benchmarked, unrealistic, or resource-demanding.
- AI idea strengths: Reviewers frequently described AI ideas as novel across in-context learning, iterative LLM approaches, cross-lingual prompting, and safeguarding.These examples reinforce the quantitative finding that AI ideas tend to receive higher novelty judgments.
- AI idea weaknesses: Common AI-idea failure modes include vague implementation details, unsuitable datasets, missing or incomparable baselines, and unrealistic assumptions.Reviewers also identified weak motivation and resource demands in some proposals.
- AI idea weaknesses: Some AI proposals remained too resource-demanding despite explicit feasibility instructions, including fine-tuning BLOOM with 176B parameters.Reviewers noted that such requirements could cause substantial delays.
- Human idea strengths: Human ideas generally build on existing research and practical considerations, but may be less innovative.Examples include applying known techniques to new problems, making incremental changes, or combining established methods.
- Human idea strengths: Human ideas often target common problems or datasets and sometimes prioritize feasibility and effectiveness over novelty and excitement.Reviewers described some as likely to work despite not being groundbreaking.
- Case studies: The sampled case studies span uncertainty, multilinguality, factuality, and coding topics, with average overall scores ranging from 3.3 to 6.7.The examples include both AI and human ideas, with authorship revealed separately.
9 Related Work
Prior work has proposed several techniques to improve research-idea generation, but these efforts generally optimize generation methods without comparing against human expert baselines.
- Prior methods improve idea generation through iterative novelty boosting, multi-agent collaboration, or multi-module retrieval and revision.
- Some prior systems share components with this paper’s ideation agent, but compare against vanilla prompting rather than expert researchers.
- Earlier work therefore emphasizes generation-method improvements without establishing how generated ideas compare with human expert ideas.
10 Discussion
The discussion reports that AI ideas were judged more novel than expert ideas, while emphasizing that idea-only evaluation leaves feasibility and real-world research outcomes unresolved.
- Discussion: Expert reviewers rated AI-generated research ideas as statistically more novel than expert ideas.
- Discussion: Idea submissions may not represent researchers’ best ideas because many participants developed them spontaneously within a short period.
- Discussion: Ideas that sound novel and exciting may not become successful projects, and the study found feasibility trade-offs for AI ideas.
- Discussion: The planned next phase will recruit researchers to execute AI- and human-generated ideas into full projects and compare evaluations with project outcomes.
- Discussion: The execution agent uses a template codebase to load data, implement baselines and proposed methods, evaluate them, and report performance.
11 Ethical Considerations
The ethical discussion highlights unresolved attribution, misuse, and homogenization risks from AI-generated research ideas, alongside calls for transparency, safety research, and greater diversity.
- Publication Policy: AI-assisted idea generation makes credit attribution less clear across model developers, framework designers, and researchers integrating generated ideas.
- Publication Policy: The paper advocates transparent documentation of models, data sources, frameworks, and human involvement rather than overhauling legal and academic norms.
- Potential for Misuse: AI-generated ideas may be misused in ways that produce harmful or destabilizing outcomes.
- Idea Homogenization: Current LLMs lack diversity in idea generation, raising concerns that widespread adoption could homogenize research perspectives and outputs.
- Idea Homogenization: The evaluation primarily measures typical generated ideas and may miss a long tail of unique or transformative ideas.
Positionality Statement
The study discloses authors’ anticipated outcomes before the experiment and describes the shared topic, proposal format, and evaluation setup used for idea generation and review.
- The authors disclosed their anticipated study outcomes before data collection: two expected no difference, while one expected AI to outperform humans.
- Human and LLM participants received the same seven research topics, spanning bias, coding, safety, multilinguality, factuality, math, and uncertainty.
- Both groups were instructed to write proposals using a shared template covering the title, problem, motivation, method, experiments, examples, and fallback plan.
- The demonstration proposal was adapted from an existing paper and shown to both the AI agent and human idea writers.
- The evaluation compared three conditions using numerical ratings for novelty, excitement, feasibility, expected effectiveness, and overall score.
- AI ideas were rated more novel than human ideas in mixed-effects analyses, while the example proposal received feasibility 8 and expected effectiveness 6.
Q Example Idea: Semantic Resonance Uncertainty Quantification
Semantic Resonance Uncertainty Quantification (SRUQ) estimates uncertainty from agreement among multiple LLM reasoning paths, using cross-evaluation and a resonance graph. The proposal evaluates it against standard uncertainty baselines across reasoning datasets, while reviewers view its novelty and potential effectiveness positively but identify substantial dependencies on cross-evaluation and prompt design.
- Motivation: The proposal targets uncertainty that statistical measures may miss in complex reasoning tasks, where semantic coherence and reasoning-path agreement provide additional signals.Its motivation contrasts SRUQ with softmax probabilities, entropy measures, and ensemble disagreement.
- Proposed Method: SRUQ generates multiple solution attempts, cross-evaluates their logical consistency and mutual support, and maps graph-based resonance to calibrated uncertainty.Its pipeline constructs a resonance graph, computes connectivity or centrality-based scores, and maps them to uncertainty estimates.
- Experiment Plan: The experiments use GSM8K, EntailmentBank, and HotpotQA, comparing SRUQ with softmax probabilities, Monte Carlo Dropout, and ensemble disagreement.Evaluation includes calibration, uncertainty ranking, error correlation, sensitivity analyses, and cross-dataset generalization.
- Illustrative Example: A train example reports a fully connected resonance graph, resonance score 0.95, calibrated uncertainty 0.05, and a correct 150-mile answer.The example illustrates multiple consistent solution paths producing low calibrated uncertainty.
- Review Assessment: Reviewers rated the proposal’s novelty around 5–6 and expected effectiveness 6, while noting similarity to self-consistency and dependence on properly executed cross-evaluation.One review also questions whether the method would outperform its baselines, and another calls for broader prompting-strategy analysis.
S Example Idea: Linguistic Pivot Constellation: Enhancing Cross-Lingual Transfer for Low-Resource Languages and Dialects
Linguistic Pivot Constellation (LPC) uses related languages or dialects as task-specific prompting pivots for low-resource cross-lingual transfer. The proposal combines language-similarity-based pivot selection with evaluation across translation and question-answering settings, receiving strong novelty judgments but mixed feasibility and effectiveness assessments.
- Motivation: The proposal addresses weak cross-lingual transfer for low-resource languages and dialects, where parallel data or multilingual pretraining may be limited.Its motivation draws on using similarities among known languages to connect models with less-resourced linguistic contexts.
- Proposed Method: LPC identifies conceptually similar languages or dialects, generates prompts in those pivots, and guides the model to triangulate responses across multiple linguistic perspectives.The proposed constellation can include related languages, regional lingua francas, and etymologically connected languages.
- Experiment Plan: The study plans translation and question-answering experiments using FLORES-101 and TyDi QA, with few-shot and zero-shot transfer methods as baselines.It varies pivot-language counts and compares performance across language families and resource levels using BLEU and F1.
- Review Assessment: Reviewers judged LPC highly novel, with novelty scores of 8–9, especially for dynamically constructing task-specific prompt constellations from related languages.The assessments distinguish this prompting use of language similarity from prior work on cross-lingual transfer and machine translation.
U Example Idea: Semantic Divergence Minimization: Reducing Hallucinations in Large Language Models through Iterative Concept Grounding
Semantic Divergence Minimization (SDM) constrains each generated reasoning step by grounding it against semantic concepts extracted from the original input. The proposal evaluates this iterative self-correction approach on multi-hop reasoning and mathematical problem solving, but reviewers question its scalability, threshold selection, and likely general effectiveness.
- Motivation: The proposal targets hallucinations caused by divergence from an input’s core semantic content, especially in complex reasoning tasks.Its motivation contrasts SDM with standard and chain-of-thought prompting, which do not explicitly constrain semantic drift.
- Proposed Method: SDM generates a next reasoning step, extracts input concepts, measures semantic similarity, and regenerates steps below a threshold until convergence or an iteration limit.The method is designed to reduce semantic drift during chain-of-thought-style reasoning.
- Experiment Plan: Experiments use HotpotQA and GSM8K, compare SDM with standard and chain-of-thought prompting, and measure task performance alongside semantic-similarity and qualitative hallucination analyses.The plan tunes similarity thresholds and iteration limits and examines cases where SDM succeeds or fails.
- Review Assessment: Reviewers rated SDM’s novelty from 4 to 8, with some viewing semantic-similarity-constrained reasoning as new and others comparing it to constrained decoding or rejection sampling.The reviews also note overlap with existing work on self-consistency and factuality control.
- Review Assessment: Expected effectiveness was rated 3, and reviewers identified manually tuning task-sensitive similarity thresholds as a major scalability concern.One review specifically questions whether the threshold is meaningful for some tasks, including elementary mathematical problems.
V Example Idea: Autoprompting: Generate Diverse Few-shot Examples for Any Application
Autoprompting automatically creates diverse few-shot examples for software applications by extracting documented functions, generating natural-language prompts, executing generated code, and refining failures. The proposal targets reduced manual effort and broader coverage, but reviewers identified vagueness and incomplete baselines as concerns.
- Motivation: Manual few-shot prompt creation is time-consuming and may not ensure comprehensive coverage.
- Proposed Method: Autoprompting extracts user-facing functions with documentation and tests, generates diverse prompts, produces code, executes functions, and retries failures.Incorrect results are revised using unit-test insights or general reasoning.
- Experiment Plan: The experiment uses applications, few-shot examples, extracted metadata, natural-language and execution modules, storage, and evaluations across models.Planned measures include goal completion and error rates, beginning with a strong model and later testing weaker and open-source models.
- Evaluation: Reviewers rated the idea’s novelty from 4 to 7, with positive assessments emphasizing automatic natural-language mapping and system checks.
- Evaluation: Reviewers criticized vague evaluation details, missing or unsuitable baselines, unclear contributions, and weak connections between the experiment plan and test cases.
- Evaluation: One reviewer gave Expected Effectiveness 7, reasoning that external systems, context enrichment, and error feedback could help the method work well.
W Example Idea: Temporal Dependency Unfolding: Improving Code Generation for Complex Stateful Systems
Temporal Dependency Unfolding addresses code generation for stateful systems by explicitly modeling states and temporal relationships before staged implementation. Reviewers considered the problem highly novel and potentially impactful, while noting that practical gains still require concrete validation.
- Problem and Motivation: Generating code for complex stateful systems remains difficult because existing approaches often overlook temporal dependencies and state changes.The limitation affects applications such as distributed systems, games, and real-time software.
- Proposed Method: Temporal Dependency Unfolding identifies changing states, constructs a temporal graph, generates code in stages, verifies consistency, and integrates the resulting segments.
- Experiment Plan: The evaluation compares direct prompting and Chain-of-Thought prompting with GPT-4 across multi-threaded applications, game logic, and distributed systems.The proposed method is assessed using correctness, temporal consistency, code quality, execution efficiency, and human ratings.
- Evaluation: Reviewers rated the idea’s novelty 6 to 10, citing the temporal-graph construction and the underexplored problem of temporal dependencies in LLM code generation.
- Evaluation: Reviewers rated excitement 8 and the overall score 9, while requiring stronger justification of use cases and concrete evidence that each method step works.
Y Attempt on Idea Execution Agent
The execution agent converts full project proposals into executable Python files using a supplied template, then evaluates correctness and implementation style. Although over half of generated files execute, manual inspection found that only a handful outperform baselines and exposed serious evaluation pitfalls.
- Execution Agent: The execution agent receives a full project proposal and outputs a Python file executable with a specified command.A manually crafted code file is provided as a template for prompting-based research pipelines.
- Execution and Evaluation: The template implements baseline and proposed methods, evaluates correctness against reference answers, checks style components, and compares performance.
- Evaluation: The agent was tested on 30 safety prompting ideas and 30 factuality prompting ideas.
- Results: Over half of generated code files executed successfully, but only a handful of proposed methods outperformed their baselines.
- Limitations: Manual checking revealed that an agent-generated baseline for malicious-prompt detection used a trivial keyword-based approach that any LLM method could beat.
- Limitations: The execution agent evaluated original sentiment-classification accuracy but omitted the proposed adversarial-prompt-detection task.The authors therefore caution against blindly trusting executed results and call for more careful verification.