Source-linked AI summary
InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem
Shuofei Qiao, Yunxiang Wei, Xuehai Wang, Bin Wu, Boyang Xue, Ningyu Zhang, Hossein A. Rahmani, Yanshan Wang, Qiang Zhang, Keyan Ding, Jeff Z. Pan, Huajun Chen, Emine Yilmaz
TL;DR
Scientific idea generation has expanded faster than evaluation, which remains constrained by narrow knowledge, biased single-judge assessment, and flattened criteria. InnoEval addresses this with heterogeneous evidence search, diverse review personas, and decoupled multi-dimensional evaluation, outperforming baselines across point-wise, pair-wise, and group-wise tasks while aligning with human judgments.
Problem
Scientific idea evaluation remains costly and subjective, with narrow knowledge horizons, biased LLM-as-a-Judge assessment, and flattened evaluation dimensions.
Method
InnoEval retrieves and grounds literature, web, and code evidence, then combines dimension-specific evaluators with an innovation review board of diverse academic personas.
Results
InnoEval outperforms baselines in point-wise, pair-wise, and group-wise evaluation, with scores highly correlated with human and peer-review judgments across five dimensions.
Takeaways & Limitations
InnoEval provides a knowledge-grounded, multi-perspective framework whose evaluation patterns and consensus align well with human innovation evaluation.
Takeaways & Limitations
InnoEval is currently limited to AI-domain, text-based ideas and requires about half an hour to evaluate one sample, despite supporting batch parallelization.
Abstract
from arXiv · showhide
The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.
1. Introduction
InnoEval frames scientific idea evaluation as a knowledge-grounded, collective, multi-criteria reasoning problem. It combines heterogeneous evidence, specialized evaluators, and human-like review consensus to improve evaluation across tasks.
- LLM-driven idea generation has outpaced evaluation, while human review remains costly, subjective, and limited in scope.
- Ideal scientific evaluation requires knowledgeable grounding, collective deliberation, and multi-criteria assessment across attributes such as novelty, feasibility, and impact.
- Existing automated methods face narrow knowledge horizons, biased LLM-as-a-Judge review, and flattened evaluation dimensions.
- InnoEval searches literature, web, and code sources, then performs multi-dimensional decoupled evaluation with dedicated evaluator agents and customizable metrics.
- 16.18% F1, roughly 5% accuracy, and 7.56% accuracy improvements over the strongest baseline are reported for point-wise, pair-wise, and group-wise tasks, respectively.
2. Problem Definition
The paper represents research ideas as structured, time-indexed objects and defines point-wise, pair-wise, and group-wise evaluation outputs. These outputs combine knowledge, assessments, decisions, and revision guidance.
- An idea I is a structured six-tuple containing a summary, motivations, research questions, methods, experimental settings, and expected results, with some fields optional.
- Each idea also has a timestamp t that specifies the temporal standpoint for evaluation, defaulting to the latest time when unspecified.
- Point-wise evaluation produces background knowledge K, multi-dimensional assessment results Epoint, and revision suggestions V for a single idea.
- Group-wise evaluation produces individual reports, comparative reasoning, and a final ranked list for a group of n ≥ 2 ideas.
- Point-wise decisions use Reject, Poster, Spotlight, and Oral labels, while group-wise output is a complete ranking dgroup.
3. InnoEval
InnoEval converts raw ideas into grounded evidence through iterative heterogeneous search, then uses diverse reviewers and dimension-specific evaluators to generate point-wise and group-wise reports. Its reports synthesize evidence, evaluations, rankings, and revision guidance.
- Framework Overview: InnoEval extracts structured idea parts before searching heterogeneous online sources and grounding retrieved knowledge for subsequent evaluation.
- Knowledge Search: The search engine covers literature, web content, and code repositories, using timestamps to account for knowledge available before and after the evaluation standpoint.
- Knowledge Search: Fast retrieval generates expanded queries and ranks candidates with semantic similarity, reranking, and model-based relevance and quality scores.
- Knowledge Search: Slow search enriches selected literature, web, and code results, while iterative query refinement repeats retrieval and enrichment N times.
- Knowledge Grounding: A grounding agent extracts evidence that supports or contradicts idea parts and supplies relevance analyses to the evaluation module.
- Multi-dimensional Multi-perspective Evaluation: Five initial dimensions are assessed by dedicated agents, with five distinct academic personas randomly assigned to each idea and scoring each criterion from 0 to 10.
- Report Generation: Point-wise synthesis combines enriched knowledge, revision suggestions, reviewer evaluations, a meta-review, a final score, and a final decision into a report.
- Report Generation: Group-wise synthesis compares point-wise reports across the evaluation dimensions and produces comparative reasoning with a ranked list of ideas.
4. Experimental Settings
The experiments use peer-reviewed ideas and construct datasets for single-idea evaluation, pairwise comparison, and group ranking. InnoEval is compared with diverse baselines under standardized implementation settings.
- Dataset Construction: Ideas are extracted from NeurIPS25 and ICLR25 peer-reviewed submissions and stratified by Reject, Poster, Spotlight, and Oral decisions.
- Dataset Construction: Pairwise and groupwise datasets are built from topic-similar papers, with easy and hard pairs defined by the gap between their labels.
- Dataset Construction: 372 pairwise samples comprise 172 easy pairs and 200 hard pairs, evaluated with Accuracy.
- Baselines: Baselines include CoT, RAG, ResearchAgent, InternAgent, GraphEval, and ScholarEval, using DeepSeek-V3.2 as the common backbone.
- Implementation: The implementation uses bge-base-en-v1.5 retrieval, bge-reranker-base reranking, α = 0.2, at most three refinement rounds, and a cost of $0.42 per sample.
5. Experiment Results
InnoEval consistently outperforms baselines across idea-evaluation tasks and produces reviews aligned with human and peer-review judgments. Analyses attribute these gains to evidence grounding, diverse perspectives, heterogeneous retrieval, and multi-dimensional feedback.
- Quantitative Results: InnoEval achieves state-of-the-art performance across point-wise, pair-wise, and group-wise evaluation tasks.Its point-wise F1 scores reach 75.74% and 74.56%, while several baselines exhibit label collapse.
- Qualitative Results: Over 70% win-rate in Overall Quality distinguishes InnoEval’s comprehensive reports from all baselines.The reports also exceed 90% win-rate in Depth and 80% in Constructiveness against most or all baselines, respectively.
- Human Evaluation: Correlation coefficients ≥0.5 across all five dimensions show strong alignment between InnoEval scores, human experts, and peer-review judgments.Significance correlates relatively less strongly than other dimensions because it is more complex to assess.
- Ablation Studies: Removing grounding, personalization, or web-and-code retrieval degrades performance, with literature-only retrieval especially harming pair-wise and group-wise tasks.Grounding filters noise, personalized evaluation mitigates LLM-as-a-Judge bias, and richer retrieval supplies background knowledge.
- Multi-perspective Test-time Scaling: Personalized test-time scaling outperforms vanilla scaling and continues improving as sample counts increase, whereas vanilla scaling gradually plateaus.The comparison supports consensus from diverse academic personas rather than fabricated divergent opinions from one reviewer.
- Search Module Evaluation: Only the heterogeneous deep knowledge search engine simultaneously preserves relevance, coverage, and diversity.ScholarEval prioritizes relevance at the expense of diversity, while AutoSurvey prioritizes diversity at the expense of topic coverage.
- Idea Generation: InnoEval’s actionable revision suggestions significantly improve generated ideas across problem formulation, methodology, and experimental design.The improvement is attributed to multi-dimensional evaluation and feasible feedback integrated into ResearchAgent’s iteration pipeline.
- Metrics Influence: Novelty is the strongest predictor of acceptance, while highlight status requires well-rounded strength across all evaluation dimensions.After acceptance, feasibility becomes more prominent because comprehensive experiments must demonstrate the proposed methodology.
6. Related Work
Prior work on LLM-assisted scientific discovery uses language models for literature reviewing, idea generation, and evaluation. Existing idea-evaluation approaches remain constrained by limited search breadth and depth, while simple LLM-as-a-Judge methods introduce bias and subjectivity.
- LLMs for Scientific Discovery: LLMs increasingly support automated scientific discovery, including literature reviewing, idea generation, and idea evaluation.The related-work landscape reflects the expanding role of LLM reasoning in these scientific workflows.
- LLMs for Idea Generation and Evaluation: Knowledge is treated as indispensable for effective idea generation and evaluation, but prior methods rely mainly on insufficient internal parametric knowledge.Recent systems add paper-search tools, yet their searched resources remain limited in breadth and depth.
- LLMs for Idea Generation and Evaluation: Simple LLM-as-a-Judge evaluation introduces systematic bias and subjectivity while overlooking review consensus needed for fair scientific evaluation.This limitation motivates evaluation procedures that incorporate broader knowledge and multiple perspectives.
7. Conclusion
InnoEval is a deep idea evaluation framework for multi-dimensional, multi-perspective innovation assessment grounded in heterogeneous knowledge. The paper benchmarks it across point-wise, pair-wise, and group-wise assessment settings.
- InnoEval performs multi-dimensional and multi-perspective innovation assessment grounded in heterogeneous knowledge.
- The authors construct an idea evaluation dataset supporting point-wise, pair-wise, and group-wise assessment.
- The evaluation combines quantitative, qualitative, and human evaluation strategies.
Impact Statement
InnoEval may accelerate innovation evaluation, reduce human effort, and optimize academic resource allocation, but its use requires attention to ethical and societal considerations. The authors recommend human-AI collaboration rather than reliance on the system alone.
- LLM-generated content may contain hallucinations and unreliable information despite efforts to mitigate LLM-as-a-Judge biases.
- The authors position InnoEval as an auxiliary tool for human decision-making rather than a replacement for human expertise.
- Users should scrutinize generated content, while open-sourcing the code framework and evaluation data supports system transparency.
A. Limitations
The paper’s limitations concern disciplinary scope, evaluation efficiency, and modality. InnoEval currently targets AI-domain, text-based ideas, while individual evaluations require substantial time.
- InnoEval currently focuses exclusively on innovation evaluation within the AI domain.The authors plan to extend it to biology, medicine, physics, geography, oceanography, and other disciplines.
- Evaluating a single sample requires about half an hour because of the multi-source search and multi-perspective, multi-dimensional methodology.The method supports parallel batch processing at approximately 100 samples per hour.
- The current system is limited to text-based ideas rather than flowcharts, slides, or videos.The authors plan to explore evaluation for additional modalities.
B.1. Point-wise Dataset
The point-wise dataset is built from sampled NeurIPS 2025 and ICLR 2025 submissions stratified by final decision. It supports binary and three-way classification evaluated with Accuracy and macro F1 score.
- Dataset Construction: The dataset collects NeurIPS 2025 and ICLR 2025 papers through the official OpenReview API2.Withdrawn submissions without review comments and placeholder papers lacking full text are filtered out.
- Dataset Construction: Remaining papers are partitioned into Reject, Poster, Spotlight, and Oral strata before stratified sampling.Every conference submission track is included within each stratum.
- Tasks and Metrics: Binary classification predicts Reject versus Accept, with Accept including Poster, Spotlight, and Oral.
- Tasks and Metrics: Three-way classification predicts Reject, Poster, or Highlight, with Highlight combining Spotlight and Oral.
- Tasks and Metrics: Accuracy and macro F1 score are used as evaluation metrics.
B.2. Group-wise Dataset
The group-wise dataset organizes ideas with similar topics into groups and evaluates both selecting the best idea and reproducing the complete gold ranking.
- Dataset construction: 172 group-wise instances form Dgroup after retrieving, reranking, and selecting similar papers across decision-label strata.Abstract embeddings retrieve 800 papers, reranking reduces them to 120, and the highest-similarity paper from each label stratum forms a group.
- Tasks and metrics: Group-wise evaluation has two objectives: identify the single best idea and produce the exact full ranking of all group ideas.
- Tasks and metrics: Best-idea selection is evaluated by Accuracy, while ranking uses Longest Increasing Subsequence (LIS) score and Accuracy.
- Tasks and metrics: The LIS score measures the length of the longest correctly ordered subsequence relative to group size.For a four-item group with one adjacent swap, the example LIS score is 3/4 = 0.75.
- Tasks and metrics: Ranking Accuracy equals 1 only when the predicted ranking exactly matches the gold ranking; otherwise it equals 0.
B.3. Pair-wise Dataset
The pair-wise dataset is sampled from grouped ideas at easy and difficult label contrasts, then evaluated with Accuracy against multiple baseline categories.
- Pair construction: Easy pairs have markedly different labels, while difficult pairs have similar labels and are harder to distinguish.
- Evaluation: Pair-wise tasks use Accuracy as the evaluation metric and report easy- and difficult-pair results separately.
- Baselines: The comparison includes chain-of-thought, retrieval-augmented generation, idea-generation, end-to-end discovery, and idea-evaluation baselines.The baselines include ResearchAgent, InternAgent, GraphEval, and ScholarEval, alongside CoT and RAG variants.
- Baselines: All baselines use DeepSeek-V3.2 as the backbone, with o4-mini additionally tested for InnoEval robustness across backbones.
D. Human Evaluation
Human evaluation compares InnoEval’s dimension-wise scoring with judgments from AI-domain experts and publicly available peer reviews.
- Human scoring: Five AI-domain experts score 60 sampled instances on clarity, novelty, feasibility, validity, and significance.The panel includes three Ph.D. students, one professor, and one algorithm engineer.
- Human scoring: Experts may use search tools and combine searched knowledge with prior domain expertise when assigning ratings.
- Review scoring: Public review records for the same 60 instances provide individual comments and meta-reviews for review-based scoring.
- Review scoring: An LLM extracts latent cues from review text and converts them into scores on the same five dimensions.The reviews rarely contain explicit dimension-wise scores, so the dimensions are inferred from textual discourse.
E. Search Metrics
Search quality is assessed through relevance, coverage, diversity, and resource quality, while persona conditioning introduces distinct academic reviewing perspectives and expertise-dependent evidence access.
- Search metrics: Relevance Density divides highly relevant retrieved resources by the total number retrieved.This normalization prevents methods from winning simply by retrieving more items.
- Search metrics: Topic Coverage measures how much the retrieved resources’ topic keywords overlap with the idea’s topic keywords.
- Metric relationships: Pairwise metric plots fit linear trends after outlier removal and report slope, Pearson r, Spearman ρ, and inlier fit goodness R2.Figure 5 provides the complete set of metric-pair correlations.
- Search metrics: Diversity measures the proportion of topic keywords beyond those already covered in the idea that appear in retrieved documents.
- Search metrics: Quality is the average LLM-assigned score across all retrieved resources.Each resource receives an individual quality score before averaging.
- Persona-based evaluation: InnoEval uses personas with distinct backgrounds, preferences, and reviewing tendencies to mitigate LLM-as-a-Judge bias.Persona backgrounds span theoretical, industry-focused, conservative, creative, empirical, senior, mentoring, service-heavy, and domain-specialist reviewers.
- Persona-based evaluation: Persona descriptions are embedded into dimension-specific prompts, while search-result masking reflects each persona’s background knowledge.Different knowledge dimensions govern access to corresponding literature, methodology, application, or frontier-related search results.
G. Case Study
The case study evaluates Mamba as a selective state space model designed to combine content-based reasoning with linear-time sequence modeling. InnoEval’s reviewers largely rate the idea highly, while identifying risks around long-sequence stability, hardware dependence, and unvalidated scaling assumptions.
- Structured Idea: Mamba introduces input-dependent selection in state space models to enable content-based reasoning while maintaining linear-time sequence modeling.The case study contrasts this design with fixed-dynamics SSMs and quadratic-scaling Transformers.
- Novelty: The selective mechanism differentiates Mamba from fixed-dynamics SSMs and attention-based Transformers through content-aware state transitions and linear time complexity.The reviewers also emphasize the hardware-aware scan algorithm and the combination of selective parameterization with a homogeneous architecture.
- Validity: Mamba is judged conceptually sound and experimentally well specified, with proposed evaluations spanning language, DNA, audio, and speech.The principal theoretical risk is instability caused by highly dynamic parameter changes during training or inference.
- Feasibility: Mamba is considered feasible with standard deep-learning components, although its hardware-aware fused selective-scan kernel requires custom CUDA programming.An open-source implementation, standard datasets and metrics, and realistic A100 GPU requirements support the feasibility assessment.
- Review Consensus: The reviewers assign high overall evaluations, including an 8.80/10 average and a final meta-review score of 8.5 with a spotlight decision.The meta-review highlights novelty, theoretical foundations, multimodal validation, empirical gains, and practical efficiency improvements.
- Risks/Feasibility Flags: The evaluation flags unresolved boundaries involving Transformer-level reasoning without attention, hardware-general efficiency, larger-scale validation, and long-sequence training stability.Further concerns include possible regressions across tasks and the need for deeper validation on continuous-valued data such as audio and genomics.
H. Prompts Used in Our Paper
The prompts require atomic, independently understandable extraction of research ideas, motivations, questions, methods, and experiments, followed by evidence-aware evaluation and multi-dimensional comparison. They also specify reproducible experiment descriptions and explicit theoretical-risk reporting.
- H.1. Prompts for Extraction Agent: The extraction agent must produce atomic, independently verifiable scientific claims rather than broad or composite statements.Each claim should be self-contained and technically understandable without consulting the original paper.
- H.1. Prompts for Extraction Agent; H.4. Prompts for Evaluation Agents; H.5. Prompts for Report Agent: The extraction and evaluation workflow prioritizes independent understandability, reproducibility, and evidence-based assessment of research ideas.The supplied prompts emphasize implementable experimental descriptions and use missing evidence or underspecified methods as grounds for caution or downgrade.
- H.1. Prompts for Extraction Agent: The extraction format covers an optional TL;DR, motivation, research questions, and atomic method components.Motivations and questions are split into distinct claims so each rationale or question remains independently answerable.
- H.1. Prompts for Extraction Agent: Method descriptions must explain procedural steps and core design ideas with sufficient context for domain experts to understand and potentially implement them.Paper-specific terminology, abbreviations, and novel components require explicit explanation rather than vague references.
- H.1. Prompts for Extraction Agent: Experimental settings begin with datasets, baselines, metrics, and hardware, followed by grouped main and analysis experiments.The required structure separates shared benchmark protocols from analyses such as ablations, sensitivity tests, and diagnostics.
- H.1. Prompts for Extraction Agent: Main experiments sharing a protocol should be merged into comprehensive descriptions that list all datasets, baselines, metrics, and implementation details.The example specifies benchmark datasets, proprietary and open-source baselines, pass@1 and pass@3, a judge model, and eight A100 GPUs.
- H.1. Prompts for Extraction Agent: Analysis experiments should isolate distinct variables and report their type, tested variable, measurement, and experimental context without overlapping factors.Examples include data scaling, training strategies, hyperparameter sensitivity, component contributions, evaluation robustness, scaling laws, and convergence.
- H.4. Prompts for Evaluation Agents; H.5. Prompts for Report Agent: Evaluation-agent prompts require concrete critique of theoretical risks, while the report agent compares ideas across clarity, novelty, validity, feasibility, and significance.The comparison is explicitly multi-dimensional rather than a single undifferentiated judgment.