Source-linked AI summary
Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
Zhiyu Chen, Keyu Zhao, Jigao Fu, Dong Liang, Yanbiao Wu, Jiaoyang Li, Haidong Xue, Xinhua Zeng, Yuanyi Zhen, Fengli Xu, Yong Li
TL;DR
Evaluating LLM-generated research ideas is difficult because scientific value lacks a single reference answer and requires judgments about novelty, feasibility, significance, and related properties. Ideation Arena addresses this with shared closed-context literature, double-blind expert pairwise comparisons, and leaderboard construction across models and agents. The resulting benchmark exposes architecture-dependent performance and shows that current LLM judges remain poorly aligned with experts, with the best judge reaching 72.56% Soft Accuracy on Overall Quality.
Problem
Research-idea quality lacks a single reference answer and requires expert judgment of novelty, feasibility, significance, and scientific value.
Method
Ideation Arena provides shared literature contexts to 14 LLMs and five agent architectures, evaluates proposals through over 6,000 double-blind expert comparisons, and aggregates preferences into an Elo leaderboard.
Results
Agent effects vary by architecture, while the best automated judge reaches 72.56% Soft Accuracy on Overall Quality and current judges remain unreliable substitutes for expert labels.
Takeaways & Limitations
Human-grounded evaluation remains necessary for measuring progress in automated research ideation because current LLM judges do not reliably reproduce expert preferences.
Takeaways & Limitations
The rankings are a time-sensitive snapshot focused mainly on computer science and should not be treated as permanent capability estimates.
Abstract
from arXiv · showhide
Evaluating research ideas generated by LLMs is difficult because their scientific value cannot be fully determined by objective criteria, and no single reference answer specifies what counts as a good idea. To address this challenge, we introduce Ideation Arena, a battle style platform that evaluates research ideas through pairwise human assessment. Ideation Arena evaluates ideas generated by 14 frontier LLMs and 5 research agent architectures built on 2 base models. To ensure a common starting point, Ideation Arena builds shared literature contexts from papers familiar to the participating researchers and provides the same contexts to all LLMs and agents. We collect over 6,000 double blind pairwise comparisons from 105 active computer science researchers and construct an Elo rating leaderboard of proposal-stage expert preferences in computer science under a shared closed-context protocol. We validate the rankings through interrater agreement and robustness analyses, showing that the leaderboard remains stable under changes in annotator composition and domain coverage. Our results show substantial variation in agent effectiveness, with some frameworks improving ideation quality over their backbones and others offering little benefit or even underperforming their base models. We further construct Ideation Arena Eval, a benchmark for assessing whether automated evaluators align with human preferences in research ideation. Experiments with current LLM judges show that they still cannot reliably reproduce expert preferences, with the best judge reaching 72.56% Soft Accuracy on Overall Quality. Our code, data, and leaderboards are available at https://github.com/foss12138/Research-Ideation-Arena.
1 Introduction
Ideation Arena addresses the difficulty of evaluating open-ended research ideas by combining shared literature contexts with double-blind expert comparisons. It produces an Elo-based leaderboard and shows that agent effectiveness and automated-judge alignment vary substantially.
- LLM-generated research ideas require expert assessment because no single reference answer or correctness criterion captures novelty, feasibility, significance, and scientific value.
- Ideation Arena uses shared literature contexts and double-blind pairwise comparisons to evaluate ideas under comparable information conditions.
- Over 6,000 comparisons from 105 researchers across five dimensions support an Elo-based leaderboard of 14 frontier LLMs and five research-agent architectures.
- Agent effects are architecture dependent: AI-Researcher with DeepSeek V3.2 ranks 2nd at 1276.5, while DeepSeek V3.2 alone ranks 7th at 1099.5.
- Ideation Arena-Eval finds that current automated judges remain misaligned with human preferences, with the best judge reaching 72.56% agreement on Overall Quality.
2 Related Work
Prior benchmarks broaden research-idea evaluation but often depend on reference alignment or LLM judges. Ideation Arena instead combines expert-aligned inputs, human preferences, and an audit benchmark for automated evaluators.
- Existing systems generate research ideas end to end, but reliable standards for judging their quality remain limited.
- AI Idea Bench emphasizes alignment with target papers, making it closer to reconstructing known ideas than evaluating divergent ideation.
- IdeaBench and LiveIdeaBench include novelty and feasibility but rely heavily on LLM-as-a-Judge, which may miss technically infeasible innovations.
- Human pairwise arenas have expanded across multiple tasks, but earlier platforms primarily target general-purpose assistant responses and broad user preferences.
- Ideation Arena aligns model inputs with evaluator expertise through citation-based construction and uses human preferences to audit automated idea evaluators.
3 Data Construction: An Expert-Guided Retrospective Pipeline
The paper constructs standardized retrospective ideation contexts from expert-linked literature. Models must identify gaps and propose methods using only the resulting closed context.
- The expert-guided pipeline uses domain expertise to acquire literature and construct standardized model inputs.
- The retrospective paradigm uses preset queries and citation titles and abstracts to provide consistent background for methodological idea generation.
- Participant DOIs and selected research fields define individualized background and domain profiles for literature construction.
- The pipeline retrieves and samples recent papers, then applies impact and additional filtering to form a quality-controlled candidate corpus.
- 3.3 Problem Formalization: For seed paper Pseed, the context C concatenates titles and abstracts of its references, representing the model’s observable knowledge boundary.
- 3.3 Problem Formalization: Models receive an instruction-context tuple and must generate a proposal solely from C, without external knowledge bases or internet retrieval.
4 The Ideation Arena Platform
The Ideation Arena platform compares models and agents under a shared closed-context protocol, then aggregates expert votes into Elo ratings with uncertainty estimates. Reliability analyses indicate that the resulting rankings are stable across evaluator and domain perturbations.
- Model Pool: The model pool includes eight proprietary LLMs, six open-source models, and five research agents evaluated with GPT-4o and DeepSeek V3.2 backbones.
- Closed-Context Protocol: Agents retain reasoning, refinement, and proposal-generation components while external retrieval and preconstructed knowledge sources are disabled.
- Pairwise Evaluation: Experts anonymously compare proposals from two systems on Novelty, Feasibility, Significance, Specificity, and Overall Quality.
- ELO-Based Ranking System: Bradley-Terry modeling converts sparse pairwise votes into latent capability coefficients and Elo leaderboard scores.
- ELO-Based Ranking System: Ties and Both Bad outcomes are encoded as draws with y_ij = 0.5 in the Arena dataset.
- ELO-Based Ranking System: Bootstrap resampling with 1,000 samples produces 95% confidence intervals for leaderboard scores.
- Reliability: 1,675 queries, covering 3,888 pairwise battles, received independent review from at least two experts.
- Reliability: Average Spearman correlations reach 0.980 under 50% annotator subsampling and 0.998 under leave-one-domain-out re-estimation.
5 Ideation Arena Analysis
The leaderboard shows strong stratification and substantial variation in how agent architectures affect ideation quality. Expert preferences also correlate across dimensions, while proposal length and seed-paper similarity have limited relationships with rankings.
- GPT-5.1, AI-Researcher with DeepSeek V3.2, Claude 4.5 Opus, and Claude 4.5 Sonnet all score above 1200 and lead the overall leaderboard.
- AI-Researcher with DeepSeek V3.2 raises DeepSeek V3.2 from 7th with 1099.5 to 2nd with 1276.5, a +177-point increase.
- 119k tokens does not ensure quality: ResearchAgent with DeepSeek V3.2 incurs the highest computational cost but ranks 17th.
- Elo ratings correlate at r > 0.95 across evaluation dimensions, while Overall Quality is especially aligned with Specificity.
- Length-adjusted rankings correlate 0.7504–0.8835 with original rankings, and seed-proximity margins correlate −0.041 with Overall Quality Elo and −0.082 with Novelty Elo.
- High-ranking models produce structured proposals with mathematical formulations, detailed algorithms, derivations, and concrete experimental plans, unlike low-ranking counterparts.
6 Meta-Evaluation: The Ideation Arena-Eval Benchmark
Ideation Arena-Eval tests whether automated judges reproduce expert preferences using expert-annotated idea pairs and a soft scoring protocol. Current judges remain imperfect, with weak alignment particularly on feasibility and sensitivity to proposal length.
- Ideation Arena-Eval contains over 6,000 idea pairs annotated with expert preferences across five dimensions.
- The benchmark formulates judging as win, loss, or tie classification and gives each tied ground-truth candidate 0.5 Soft-Accuracy credit.
- Overall Quality Soft Accuracy ranges from 59.52% to 72.56% across 14 judges, while Feasibility has the lowest agreement.
- A 500-comparison A/B-swap analysis yields a 0.942 Spearman correlation between original-order and swapped-order Overall Quality rankings.
- Across comparable-strength battles, judges select the longer proposal in 50.9%–62.0% of decisive predictions across dimensions.
7 Conclusion
Ideation Arena establishes an expert-preference benchmark for proposal-stage research ideation and shows that agent architectures can reshape model quality under a shared closed-context protocol. Its rankings are time-sensitive and should complement, not replace, human scientific judgment.
- Over 6,000 expert votes from 105 active researchers benchmark 14 foundation models and 5 agent systems under shared closed-context constraints.
- The leaderboard is a controlled snapshot because foundation models and agent frameworks continue to evolve.
- The main benchmark focuses on computer science, while research-agent comparisons retain reasoning and proposal-generation components under shared closed-context conditions.
- Generated ideas should be treated as preliminary hypotheses requiring expert scrutiny, methodological validation, and empirical verification.
A.2 Evaluation Criteria for Human Review
Human reviewers compare proposals using explicit criteria spanning overall research value, novelty, feasibility, significance, and specificity. The evaluation draws on a panel with varied research backgrounds and seed-paper contexts organized across research areas.
- Evaluation criteria: Reviewers assess proposals by Overall Quality, Novelty, Feasibility, Significance, and Specificity.Overall Quality reflects willingness to invest time and resources; Novelty emphasizes uniqueness and substantive depth.
- Expert panel: The expert panel includes PhD students, postdoctoral researchers, faculty members, and industry researchers.
- Expert panel: Expertise coverage is reported across primary areas and frequently reported secondary research interests, with experts allowed to select multiple areas.The resulting tables represent overlapping research-area coverage.
- Context coverage: The seed-paper pool assigns each context one secondary-field label mapped to one of eight primary research areas.
- Evaluation setup: Agent workflows are characterized by the components retained, removed, or replaced for proposal generation under the shared closed-context configuration.
A.6 Proposal-Length Analysis
The proposal-length analysis tests whether response length is associated with human preferences and whether adjusting for length changes the Elo leaderboard. It also examines seed-specific similarity and alternative treatment of “Both Bad” outcomes.
- Length preference: The analysis first compares longer-proposal selection in battles between systems whose anchored ratings differ by at most 100 points.
- Length adjustment: A length-adjusted Bradley–Terry model is fit separately for each evaluation dimension, with proposal character counts entering the analysis.
- Seed-proximity diagnostic: SPECTER2 measures proposal proximity to seed papers, and the normalized margin captures seed-specific proximity beyond similarity induced by shared topic and literature context.The reported means are 0.9346 for proposal-to-seed-summary similarity and 0.9326 for within-task model-proposal similarity.
- Seed-proximity diagnostic: Across both rankings, group means remain close to zero and show no monotonic increase with rank in the similarity-based diagnostic.
- Outcome sensitivity: The sensitivity analysis keeps Tie as 0.5 but converts each Both Bad label into two comparisons against a virtual acceptable-quality anchor.The resulting rankings are compared with the original rankings.
A.10 LLM-Judge Order and Length Diagnostics
The LLM-judge diagnostics examine sensitivity to proposal order and length. Order-swapped Overall Quality rankings are generally consistent, while judges select longer proposals at variable rates.
- Order effects: Ten judges other than GPT-4o show Overall Quality order-swap consistency between 70.6% and 92.5%, whereas GPT-4o reaches 45.0%.
- Length effects: The analysis also reports how often judges select the longer proposal among decisive predictions in comparable-strength battles.
A.11 Elo Rating Confidence Intervals
The appendix provides statistical diagnostics for Elo rankings, annotation distributions, and judge evaluation reliability. These materials examine confidence intervals, ranking sensitivity, agreement structure, and preference baselines.
- Confidence intervals: Figure 3 presents the Elo leaderboard with error bars, while accompanying analyses assess the statistical validity of the rankings.
- Agent workflows: The agent leaderboard records the original workflow and the components retained, removed, or replaced in each evaluated research-agent variant.
- Confidence intervals: The confidence-interval analysis reports disjoint intervals between AI-Researcher with DeepSeek V3.2 and its base model, supporting the observed +177-point gain as distinct from stochastic evaluation noise.
- Ranking sensitivity: The analysis also compares rankings when Both Bad outcomes are modeled separately and reports Both Bad involvement rates by original Overall Quality ranking group.
- Preference distributions: The appendix analyzes marginal ground-truth label distributions across five dimensions because tie labels alter the expected random-agreement baseline.The reported tie rate is 10.62%.
- Reliability diagnostics: Additional appendix materials report pairwise agreement, bootstrap confidence intervals, judge Soft Accuracy, order-swap consistency, and longer-proposal selection rates.
- Annotation procedure: Expert annotators were recruited through academic social-media platforms, screened for active research experience and domains, and compensated according to expected workload.
B Qualitative Case Study Analysis
The case studies present several proposed frameworks that target knowledge injection, multimodal reasoning, self-supervised perception, prompt tuning, and agent-based evaluation. Across these examples, the proposals emphasize explicit structure, uncertainty handling, relational consistency, and modular or interactive mechanisms.
- RELAX: RELAX combines relation-specific adapters, entity embeddings, bidirectional relational training, elastic regularization, and retrieval-assisted knowledge injection.Its stated targets include continual updates, relational consistency, and long-tail factual coverage.
- DynaProg: DynaProg integrates dynamic program induction with modular verifiers and process-supervised reinforcement learning for structured multimodal reasoning.The framework defines a domain-specific language and selects a consistent induced program through clustering.
- PBCMA: PBCMA replaces unidirectional supervision with symmetric probabilistic embedding alignment to propagate uncertainty and reduce corruption from low-confidence pseudo-labels.The planned evaluation uses terrain datasets, self-supervised and supervised baselines, and metrics including mIoU, calibration, label efficiency, robustness, and inference speed.
- HCPT: HCPT adapts frozen vision-language models through shared global coordination prompts and modality-specific prompts that are dynamically fused.The proposal separates cross-modal coordination from modality-specific adaptation.
- EEIAF: EEIAF uses multiple reader-persona agents trained with reinforcement learning to provide feedback to a central event-extraction controller.The agents focus on aspects such as causality, relevance, and temporality, while the controller adapts extraction priorities to context.
- Other proposal patterns: The case studies also describe adaptive benchmarking and multimodal knowledge-graph systems, while the requested output format emphasizes concrete mechanisms, bottlenecks, and three main contributions.These examples illustrate a recurring preference for technically specified proposals rather than broad performance goals.