InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

Motivation and gap

• LLM-generated ideas have outpaced evaluation capacity, while expert-dependent assessment remains costly, subjective, narrow, and vulnerable to overlooking valuable ideas.The paper identifies narrow knowledge horizons, overlooked review consensus, and flattened evaluation dimensions as central limitations of existing automated tools.Source: §1 · pp. 1–2

Knowledge grounding

• InnoEval uses heterogeneous deep search over literature, web opinions, and code repositories, combining iterative query refinement, hybrid scoring, fast search, and slow reading.The resulting evidence base supplies living, cross-domain knowledge for grounding and evaluation.Source: §1 · p. 2, Figure 1 · p. 2

Collective deliberation

• An innovation review board samples reviewers with distinct backgrounds, expertise, and preferences, partially masking knowledge by role to mitigate single-judge bias.Reviewers independently assess ideas to simulate deliberation among diverse perspectives.Source: §1 · p. 2

Multi-dimensional evaluation

• Specialized evaluator agents independently assess customizable criteria—Clarity, Novelty, Feasibility, Validity, and Significance—and aggregate findings into cited, actionable reports.Reports include structured analyses and scores, a meta-review with decisions, and concrete improvement directions.Source: §1 · p. 2

Evaluation results

• InnoEval outperforms the strongest baseline by 16.18% F1 in point-wise prediction, roughly 5% accuracy in pair-wise comparison, and 7.56% accuracy in group-wise ranking.Its reports also achieve over 70% win rate in Overall Quality, while scores highly correlate with human judgments across all metrics.Source: §1 · p. 2

• An idea I is represented as a six-tuple covering its summary, motivations, research questions, methods, experimental settings, and expected results.The summary, experimental settings, and expected results are optional for less mature ideas.Source: §2 · pp. 2–3
• Each idea includes a timestamp t specifying the temporal standpoint for evaluation, defaulting to the latest time when unspecified.This accounts for the time-sensitive nature of research ideas.Source: §2 · p. 3
• Point-wise evaluation searches background knowledge, assesses the idea across multiple dimensions, and provides concrete revision suggestions in an individual report.Its measurable final decision is one of Reject, Poster, Spotlight, or Oral.Source: §2 · p. 3
• Group-wise evaluation produces individual reports for ideas on the same topic together with a final ranked list of all ideas.The ranked list constitutes the group-wise decision output dgroup.Source: §2 · p. 3

3. InnoEval

• InnoEval extracts structured idea components, grounds retrieved knowledge, evaluates ideas across dimensions and perspectives, and synthesizes actionable reports with meta-reviews and revision suggestions.Its framework combines specialized evaluator agents with an innovation review board of diverse academic personas.Source: §3 · p. 3

3.1. Heterogeneous Deep Knowledge Search

• The search module combines academic, web, and code sources with timestamped retrieval, hybrid fast-slow search, ranking, filtering, enrichment, and iterative query refinement.It uses online sources to incorporate emerging knowledge, distinguishes pre-knowledge for evaluation from post-knowledge for revision suggestions, and repeats refinement for N iterations.Source: §3.1 · p. 3

3.2. Knowledge Grounding

• Knowledge grounding distills evidence that specifically supports or contradicts each idea component and provides detailed relevance analyses for downstream evaluation.The grounding results are passed into the subsequent evaluation module.Source: §3.2 · p. 4

3.3. Multi-dimensional Multi-perspective Evaluation

• The evaluation module scores ideas independently on Clarity, Validity, Novelty, Feasibility, and Significance using dedicated evaluators and five randomly assigned academic personas.Each reviewer receives familiarity-dependent masking of knowledge sources and produces a [0, 10] score with a reasoning narrative; users may add dimensions.Source: §3.3 · p. 4

3.4. Report Generation

• Report generation combines enriched knowledge, future information, grounding, and reviewer outputs into point-wise reports containing meta-reviews, final scores, decisions, and revision suggestions.For groups of at least two ideas, it compares every idea across the evaluation dimensions and produces reasoning with a comprehensive ranking.Source: §3.4 · pp. 4–5

4.1. Datasets and Metrics.

• The benchmark comprises 217 point-wise samples, 372 pair-wise samples, and 172 group-wise instances derived from stratified NeurIPS25 and ICLR25 submissions and similarity-based grouping.Point-wise tasks predict binary acceptance or ternary labels, pair-wise tasks compare easy or hard pairs, and group-wise tasks select or rank ideas.Source: §4.1 · p. 5
• Evaluation uses Accuracy and macro F1 for point-wise classification, Accuracy for pair-wise comparison and best selection, and Longest Increasing Subsequence Match plus Accuracy for ranking.Pair-wise data include 172 easy and 200 hard pairs, while group-wise instances contain automatically ranked ideas based on paper labels.Source: §4.1 · p. 5

4.2. Baselines and Implementation Details

• InnoEval is compared with CoT, RAG, ResearchAgent, InternAgent, GraphEval, and ScholarEval, using DeepSeek-V3.2 uniformly as the backbone model.The baselines span naive, idea-generation, end-to-end scientific-discovery, and dedicated idea-evaluation methods; o4-mini is additionally tested in §5.2.Source: §4.2 · p. 5

5.1. Main Results

• 75.74% and 74.56% F1 scores show InnoEval disperses label predictions and achieves state-of-the-art performance across point-wise, pair-wise, and group-wise tasks.Most baselines exhibit label collapse, while GraphEval performs poorly on pair and group tasks and ResearchAgent struggles with relatively new ideas.Source: §5.1 · p. 5
• InnoEval exceeds 90% win-rates in Depth, over 80% in Constructiveness, and over 70% in Overall Quality against baselines, while ScholarEval remains competitive on several dimensions.InnoEval’s multi-source search improves Rationality and Supportiveness, whereas ScholarEval’s limited dimensions and evidence-free suggestions constrain Constructiveness.Source: Table 2 · p. 6, §5.1 · p. 6
• Correlation coefficients ≥0.5 across all five dimensions show that InnoEval’s scores align strongly with both human experts and peer-review judgments.Clarity has the highest correlation, whereas Significance has relatively lower correlation because of its greater complexity.Source: Figure 2 · p. 6, §5.1 · p. 6

5.2. Ablations

• Removing grounding, personalization, or heterogeneous retrieval degrades performance, while replacing DeepSeek-V3.2 with o4-mini causes only slight declines and still surpasses the strongest baseline.The largest personalization effects occur on point-wise and group-wise tasks, and literature-only retrieval particularly harms pair-wise and group-wise evaluation.Source: Figure 2 · p. 6, §5.2 · pp. 6–7

5.3. Analysis

• Personalized test-time scaling consistently outperforms vanilla scaling, continues improving with more samples, and supports using five randomly selected personas as an efficiency-effectiveness compromise.A single persona can underperform no-persona ranking because it transforms model bias rather than producing reviewer consensus; the five-persona setting is not the performance ceiling.Source: Figure 3 · p. 7, §5.3 · p. 7
• The heterogeneous search engine uniquely preserves relevance, coverage, and diversity, while InnoEval’s actionable feedback improves generated ideas and reveals novelty as the strongest acceptance predictor.After acceptance, feasibility becomes more prominent for achieving highlight status; clarity has the weakest acceptance effect, and highlight status requires strength across dimensions. Dimension correlations further show that significance, validity, feasibility, novelty, and clarity exhibit interpretable relationships aligned with scholarly cognition.Source: Figure 3 · p. 7, §5.3 · pp. 7–8

LLMs for Scientific Discovery

• LLMs now support literature reviewing, idea generation, method implementation, manuscript writing, and broader automated scientific discovery pipelines, while dedicated idea evaluation remains rare.Because innovation is central to scientific discovery, idea quality conditions every downstream phase.Source: §6 · p. 8

LLMs for Idea Generation and Evaluation

• Prior idea-generation and evaluation methods rely mainly on LLMs’ internal parametric knowledge, which is insufficient for effective knowledge-grounded operation.Relevant background knowledge is described as indispensable for both idea generation and evaluation.Source: §6 · p. 8
• Paper-search tools broaden access to evidence, but limited resource breadth and depth still produce narrow knowledge horizons.Recent methods invoke paper-search tools, including Baek et al. (2025), Su et al. (2025), Wang et al. (2025; 2024a), Si et al. (2025), Li et al. (2024), Zhao et al. (2025), Ju et al. (2025), Tang et al. (2025), Zhang et al. (2025; 2026).Source: §6 · p. 8
• Simple LLM-as-a-Judge evaluation introduces systematic bias and subjectivity, overlooking the review consensus required for fair scientific evaluation.This limitation is associated with Tang et al. (2025), Zhang et al. (2025), Si et al. (2025), Shahid et al. (2025), and Moussa et al. (2025).Source: §6 · p. 9

Conclusion

• InnoEval enables multi-dimensional, multi-perspective idea evaluation grounded in heterogeneous knowledge.The framework targets innovation assessment through diverse knowledge grounding.Source: §7 · p. 9
• The authors construct a dataset supporting point-wise, pair-wise, and group-wise assessment with quantitative, qualitative, and human evaluation strategies.This dataset broadens benchmarking across complementary evaluation modes and evidence types.Source: §7 · p. 9
• A comprehensive experiment suite analyzes InnoEval’s performance and behavior, yielding findings intended to guide future research.The experiments provide insights for the research community beyond a single evaluation setting.Source: §7 · p. 9

Impact Statement

• InnoEval should augment rather than replace human expertise because LLM-generated content may contain hallucinations and unreliable information despite efforts to mitigate LLM-as-a-Judge biases.Users should carefully scrutinize its generated content; the authors will open-source the code framework and evaluation data to enhance transparency.Source: Impact Statement · p. 9

A. Limitations

• The framework currently evaluates innovation ideas exclusively within the AI domain, with future plans to cover biology, medicine, physics, geography, and oceanography.Source: A. Limitations · p. 13
• Evaluating one sample takes about half an hour because of heterogeneous deep search and multi-perspective, multi-dimensional evaluation, although batch processing reaches approximately 100 samples per hour.Source: A. Limitations · p. 13
• The method is currently limited to text-based ideas, with future work targeting flowcharts, slides, videos, and other modalities.Source: A. Limitations · p. 13

Dataset Construction

• Dpoint contains 217 manually verified ideas—136 from ICLR 2025 and 81 from NeurIPS 2025—stratified by Reject, Poster, Spotlight, and Oral decisions.The dataset includes 138 Reject, 66 Poster, 9 Spotlight, and 4 Oral decisions, corresponding to 61.3%, 29.3%, and 9.4% for Reject, Poster, and Highlight categories.Source: §B.1 · p. 13

Tasks and Metrics

• The dataset supports binary Reject-versus-Accept classification and three-way Reject-versus-Poster-versus-Highlight classification, evaluated with Accuracy and macro F1.Accept includes Poster, Spotlight, and Oral; Highlight combines Spotlight and Oral.Source: §B.1 · p. 13

Dataset Construction

• The construction pipeline retrieves 800 similar papers, reranks them to 120, selects the highest-similarity paper per label stratum, and produces 172 group-wise instances.The same extraction and verification pipeline is applied to retrieved papers before forming Dgroup.Source: §B.2 · p. 13

Tasks and Metrics

• Best Selection identifies the single best idea in each group and is evaluated with Accuracy.The objective is to select one best idea within every group.Source: §B.2 · p. 13
• Ranking requires the exact full ordering of group ideas and is evaluated with LIS score and Accuracy.LIS is the longest increasing subsequence length divided by group size, while Accuracy equals 1 only for an identical predicted and gold ranking.Source: §B.2 · p. 13

B.3. Pair-wise Dataset

• Pairs are sampled within each Dgroup group, with a group of size n providing the basis for pair construction.The sampling distinguishes markedly different labels from similar labels to create two difficulty levels.Source: §B.3 · p. 13
• After screening and filtering, Dpair contains 372 samples: 172 easy pairs and 200 difficult pairs, evaluated separately with Accuracy.Easy pairs contrast Highlight with Reject, while difficult pairs use similar labels such as Poster versus Reject or Highlight versus Poster.Source: §B.3 · pp. 13, 15

C. Baselines and Reproduction Details

• Four baseline categories are selected for comprehensive comparison with InnoEval: reasoning, idea generation, end-to-end scientific discovery, and idea evaluation methods.The supplied reproduction details cover CoT and RAG, ResearchAgent, InternAgent, GraphEval, and ScholarEval.Source: C. Baselines and Reproduction Details · pp. 15–16
• CoT removes grounding and personalized settings from InnoEval’s evaluation agent, whereas RAG adds fast-retrieved arXiv papers and their metadata as references.Both methods use the evaluation agent Mψ and report-agent Mr, with RAG retrieving the top-m relevant papers.Source: C. Baselines and Reproduction Details · p. 15
• ResearchAgent (Baek et al., 2025) scores an idea’s research problem, scientific method, and experiment design on five criteria before synthesizing a final decision.The reproduction optimizes the final-decision prompt and adapts ResearchAgent’s evaluations for point-wise, pairwise, and group-wise results.Source: C. Baselines and Reproduction Details · p. 16
• InternAgent (Zhang et al., 2025) is reproduced through its survey and assessment modules, with outputs adapted to the evaluation tasks.Its survey module retrieves relevant literature, while its assessment module performs evaluation.Source: C. Baselines and Reproduction Details · p. 16
• GraphEval (Feng et al., 2025) predicts labels with an officially implemented GNN, while ScholarEval (Moussa et al., 2025) synthesizes literature-grounded soundness and contribution reviews.GraphEval ranks ideas using predicted labels and confidence-based tie-breaking; ScholarEval reviews are adapted into final decisions.Source: C. Baselines and Reproduction Details · p. 16
• All baselines use DeepSeek-V3.2, while o4-mini is additionally tested in §5.2 for backbone robustness.The shared backbone standardizes baseline comparisons, and the additional backbone test applies to InnoEval.Source: C. Baselines and Reproduction Details · p. 16

Human Scoring

• Human experts scored 60 instances across clarity, novelty, feasibility, validity, and significance using InnoEval’s protocol, enabling dimension-wise Pearson-correlation comparison.The panel comprised three Ph.D. students, one professor, and one algorithm engineer, who could use search tools alongside their prior expertise.Source: D. Human Evaluation · p. 16

Review Scoring

• LLM-extracted scores from reviewers’ comments and meta-reviews on the same five dimensions were correlated with InnoEval scores to mirror authentic peer-review judgments.Because public reviews rarely provide explicit dimension-wise scores, the LLM inferred latent cues from textual discourse while following InnoEval’s scoring protocol.Source: D. Human Evaluation · p. 16

E. Search Metrics

• Relevance density divides highly relevant retrieved resources by all retrieved resources, preventing search methods from benefiting merely by returning more items.Highly relevant resources are identified with an LLM, and the metric is |R_h|/|R|.Source: E. Search Metrics · p. 16
• Topic coverage measures overlap between an idea’s topic keywords and those covered by retrieved resources, while diversity measures additional topic keywords appearing in retrieved documents.Together, these metrics capture retrieval comprehensiveness and information beyond the idea’s stated topics.Source: E. Search Metrics · pp. 16–17
• Figure 5 analyzes relationships among search metrics with outlier-removed linear fits, reporting slopes, Pearson r, Spearman ρ, and inlier goodness-of-fit R2.The figure presents scatter plots for each metric pair, with the red dashed line representing the fitted relationship after outlier removal.Source: Figure 5 · p. 17, E. Search Metrics · p. 17
• Quality averages LLM-assigned scores across retrieved resources, using DeepSeek-V3.2 for all metric computations.The quality metric is based on each resource’s score s_ri.Source: E. Search Metrics · p. 17

F. Innovation Review Board

• The innovation review board uses domain-specific personas whose backgrounds, knowledge, goals, and reviewing constraints are embedded into dimension-specific agent prompts.Personas represent academic roles such as professors, PhD students, and algorithm engineers, with background knowledge scored across four dimensions from 1–10.Source: F. Innovation Review Board · p. 17
• Persona expertise is simulated by randomly masking search evidence according to knowledge scores, while increasing persona count consistently improves evaluation performance.Literature Familiarity masks literature results, Frontier Sensitivity masks web findings, and Application Experience masks code evidence; a score of 8 leaves 80% of literature grounding accessible.Source: F. Innovation Review Board · p. 18
• The board spans complementary reviewing perspectives, including senior and domain experts, empirical and theoretical scientists, industry practitioners, conservatives, creatives, and detail-oriented or clarity-focused researchers.These personas differ in priorities such as novelty, rigor, reproducibility, deployment, conceptual creativity, communication clarity, and implementation practicality.Source: Personas · pp. 18–19

Structured Idea

• Mamba makes ∆, B, and C input-dependent, enabling content-aware selection while using associative scans, kernel fusion, recomputation, and a simplified homogeneous architecture for efficient computation.The model targets Transformers’ quadratic scaling and prior SSMs’ inability to perform content-based reasoning, and is evaluated across language, DNA, audio, speech, and synthetic tasks.Source: # Final Report · p. 21

Reviewer Evaluations

• Reviewers scored the idea highly overall: Reviewer 1 gave 8/10, Reviewer 2 gave 7/10, Reviewer 3 gave 8.60/10, and the average was 8.80/10.Reviewer 1 emphasized clarity, mechanism definition, and goal-method alignment, while noting limited proprietary-baseline comparability and underspecified ablations.Source: ## 3. Evaluation Results · pp. 23–25

Novelty

• The proposed contribution was judged novel because selective parameterization and hardware-aware scanning distinguish Mamba from fixed-dynamics SSMs, attention-based Transformers, and related adaptive models.The novelty assessment assigned a score of 9 and described the combination as a substantial advancement if its empirical claims are validated.Source: ## 3. Evaluation Results · p. 24

Validity and Feasibility

• The idea was judged valid and feasible, with concrete parameterization, coherent computational methods, standard datasets and metrics, realistic A100 resources, and an available open-source implementation.Potential concerns include instability from dynamic parameters and the engineering difficulty of implementing the fused CUDA kernel.Source: ## 3. Evaluation Results · pp. 24–25

Final Review Meta-review

• The final meta-review awarded Mamba 8.5 and a spotlight decision, citing unanimous reviewer praise, strong empirical gains, and efficiency improvements across language, DNA, and audio.Mamba was characterized as a transformative contribution to sequence modeling with input-dependent selection and linear-time scaling.Source: ## 3. Evaluation Results · p. 25

Revision Suggestions

• Recommended next steps include profiling across sequence lengths and hardware, testing Long-Range Arena and noisy real-world data, and exploring MoE, bidirectional, pruning, and binarization variants.Risks include long-sequence training instability, added implementation complexity, task-dependent regressions, and uncertain selection effectiveness for continuous audio and genomic data.Source: ## 5. Revision Suggestions · p. 26

H.1. Prompts for Extraction Agent

• The extraction prompt requires atomic, independently verifiable claims that are self-contained, clearly defined, and understandable without the original paper.It prohibits unexplained paper-specific terminology and requires method implementation and experimental reproduction to remain unambiguous.Source: Prompt for Extraction Agent · p. 27
• The extraction schema covers TL;DR, motivation, research questions, atomic method components, experimental setting, and optional expected results.Motivations and research questions must be separated into focused claims, while methods must include procedural steps and core design ideas.Source: Prompt for Extraction Agent · p. 27, §2 · p. 27, §3 · p. 27
• The experimental-setting output must begin with datasets, baselines, metrics, and hardware, followed by grouped main experiments and distinct analysis experiments.Main experiments merge setups sharing methodology and purpose, whereas analysis experiments separately describe variables, measurements, and contexts.Source: §4 · p. 28
• Analysis experiments are organized around distinct dimensions such as data scaling, training strategies, hyperparameters, component contributions, evaluation robustness, model scaling, and convergence.The prompt instructs the extractor to avoid overlapping variables and to cover all major analysis types mentioned in the paper.Source: §4 · p. 28
• Across prompt types, outputs are constrained to strict machine-readable formats, including JSON arrays for extraction and bracketed pipe-separated query lists for search.These constraints prohibit explanations or extra text and preserve consistent downstream processing.Source: RULES FOR ATOMIC CLAIMS · p. 29, OUTPUT FORMAT · p. 29, Prompt for Search Agent (Query… · p. 30

H.2. Prompts for Search Agent

• The literature-search prompt generates 6–10 title queries from only the idea’s core essence, primary gap, and decisive technical components, prioritizing precision over breadth.It expands core concepts into title-level synonyms and uses OR-rich queries with AND only for indispensable concept combinations.Source: Prompt for Search Agent (Query… · pp. 29–30
• The web-search prompt synthesizes 3–5 targeted queries from an idea’s motivation and methodology to find related works, evidence, criticisms, and diverse viewpoints on specified research-oriented websites.Queries use only AND and OR operators, exclude site: filters and other Boolean operators, and contain 1–3 keywords or phrases.Source: Prompt for Search Agent (Query… · p. 31
• The code-search prompt targets runnable GitHub implementations across related pipelines, supporting frameworks, and baseline, benchmark, or dataset code.Queries must include site:github.com and exclude collections, surveys, paper lists, and other repositories without real code.Source: Prompt for Search Agent (Query… · pp. 31–32
• Search queries should use concise, single-angle phrases covering core tasks, methods, frameworks, benchmarks, datasets, or baselines, with strict source and negative filters for GitHub searches.Initial GitHub searches require 8–12 queries, each including site:github.com and the filters -awesome -survey -paper -list -collection.Source: Prompt for Search Agent (Query… · pp. 32–33
• Literature relevance is scored from 0 to 10 using methodological relevance, problem-domain match, technical contribution, and conceptual similarity.The prompt maps scores from complete irrelevance at 0–2 to extreme relevance at 9–10 and requires one integer output.Source: Prompt for Search Agent (Search… · p. 33
• Web-content relevance is scored from 0 to 10 using content relevance, practical utility, information quality, and domain alignment.The output is a single integer representing relevance to the research idea, with 9–10 reserved for direct answers or critical insights.Source: Prompt for Search Agent (Search… · p. 33
• Code-repository scoring prioritizes directly reusable implementations, frameworks, and experimental baselines or datasets across categories A, B, and C.High scores require substantial, well-documented, reproducible, and reasonably maintained code that helps implement the idea or reproduce experiments.Source: Prompt for Search Agent (Search… · pp. 34–35
• Repositories clearly belonging to A/B/C, with plausible reuse for implementing the idea or experiments, meaningful code, some documentation, and a serious-project appearance, receive scores of 7–8.Scores are single integers from 0 to 10; unrelated repositories and personal toy projects with little reuse value should not receive high scores.Source: Prompt for Search Agent (Search… · pp. 34–35
• Refined queries reuse high-signal terminology, generalize over-specific fragments, remove noisy terms, and introduce adjacent terminology without drifting off-topic.The prompts require refinement rather than verbatim duplication and aim to improve recall while maintaining precision.Source: Prompt for Search Agent (Query… · pp. 35–36
• Code-query refinement checks category A/B/C coverage and repository quality, then broadens constrained searches when categories or quality signals are under-covered.Quality checks include stars, maintenance recency, real code, documentation, reproducibility, and alignment with the experimental setting.Source: Prompt for Search Agent (Query… · p. 37
• Search-agent prompts refine queries from strong and weak results, target missing buckets, and require concise GitHub-focused outputs with standardized exclusions.Queries must focus on one implementation, framework/toolkit, or baseline/benchmark/dataset angle and produce 8–12 pipe-separated queries.Source: Prompt for Search Agent (Query… · pp. 37–38

H.3. Prompts for Grounding Agent

• Literature-grounding prompts extract a related paper’s alignment, differences, evidence, and support or contradiction for a specific research-idea part.They request a factual 2–5-sentence summary and a 0–10 relevance score using independently applied bands from no relevance to strong alignment.Source: Prompt for Grounding Agent - Literature · pp. 38–39
• Across grounding prompts, agents must evaluate independently, remain critical and balanced, avoid overstating weak relationships, and align scores with human-reviewer patterns.These instructions are applied to literature, web content, and repositories according to each source’s relevance or support strength.Source: Prompt for Grounding Agent - Literature · p. 39, Prompt for Grounding Agent - Web · p. 40
• Web-grounding prompts capture analytical viewpoints, evidence, criticisms, limitations, and alternatives while excluding tutorials and weak or tangential connections.The output combines a 2–5-sentence perspective summary with a 0–10 relevance score tied to evidence strength and directness.Source: Prompt for Grounding Agent - Web · pp. 39–40
• Code-grounding prompts assess repositories through concrete implementations, frameworks, experimental resources, implementation quality, completeness, and usability rather than documentation alone.They produce a 2–5-sentence contribution summary and a 0–10 score ranging from unrelated code to directly usable implementations.Source: Prompt for Grounding Agent - Code · pp. 40–41

H.4. Prompts for Evaluation Agents Prompt for Evaluation Agent - Clarity

• Clarity evaluation checks goal–method alignment, mechanism definition, ambiguity, and whether expected metrics measure the research question.The prompt evaluates preliminary ideas rather than finished manuscripts, avoids penalties for brevity or missing graphs, and requests logical gaps for scores below 7.Source: Prompt for Evaluation Agent - Clarity · pp. 41–42
• Novelty evaluation distinguishes prior art from self-retrieval and assesses differentiation from baselines, tailored combinations, and uniqueness against the state of the art.It uses a 0–10 scale spanning groundbreaking innovation, incremental work, derivative ideas, and redundancy.Source: Prompt for Evaluation Agent - Novelty · pp. 42–43
• Feasibility evaluation judges real-world implementability without penalizing absent repositories and permits internal engineering knowledge when references are sparse.The prompt specifically considers whether the mathematics and logic can be implemented with standard libraries such as PyTorch, TensorFlow, and Scikit-learn.Source: Prompt for Evaluation Agent -… · p. 43

H.4. Prompts for Evaluation Agents

• Evaluation agents assess feasibility through compute realism, engineering complexity, and library support, then assign a 0–10 score with justification and core-loop pseudocode.Feasibility scores range from turnkey implementation to impossible proposals, with calibration targets across score bands.Source: Prompt for Evaluation Agent -… · p. 44
• Validity prompts test miracle assumptions, theoretical alignment, and baseline fairness while evaluating conceptual soundness rather than requiring complete mathematical proofs.The agent must provide a score, critique, and the most likely theoretical failure mode; calibration spans theoretically flawless to invalid.Source: Prompt for Evaluation Agent - Validity · pp. 44–45
• Significance prompts evaluate problem relevance, generalizability, and nontriviality, assuming the idea works, and require a 0–10 justification identifying the primary beneficiary community.The scale ranges from transformative impact on major bottlenecks to trivial or useless contributions.Source: Prompt for Evaluation Agent -… · pp. 45–46

H.5. Prompts for Report Agent

• The meta-review prompt combines the idea specification with five reviewer reports, treats their average as a nonbinding signal, and adjusts decisions using explicit evidence-first rules.Missing datasets, metrics, protocols, or mechanisms can justify downgrading, while upgrades require concrete formulations, baselines, or theoretical grounding.Source: Prompt for Report Agent (Meta-Review) · p. 46
• The report agent maps adjusted scores to Reject, Poster, Spotlight, or Oral decisions and sets confidence high only when concrete evidence and consistent reasoning support the outcome.The required JSON includes the score, decision, reviewer delta and justification, final reasoning, confidence, and key evidence snippets.Source: Prompt for Report Agent (Meta-Review) · pp. 47–48
• The revision-suggestion prompt requires specific, actionable future-work advice grounded only in the current idea and enriched future papers, covering methods, experiments, data, risks, and next steps.Suggestions must prioritize feasible high-impact actions and avoid generic recommendations.Source: Prompt for Report Agent (Revision… · pp. 48–49
• The pair-comparison prompt selects the better of two ideas through multi-dimensional analysis of clarity, novelty, validity, feasibility, and significance.It also requires strengths, weaknesses, risks, trade-offs, complementarity, reviewer consensus, comparative insights, and a justified A-or-B selection.Source: Prompt for Report Agent (Pair… · p. 49
• The prompt assigns the agent to rank four research ideas from best to worst using comprehensive multi-dimensional analysis.Source: Prompt for Report Agent (Group Ranking) · p. 50
• The input combines each idea’s basic idea, motivation, research question, method, experimental setting, and summaries from multiple reviewers.Source: Prompt for Report Agent (Group Ranking) · p. 50
• The agent must analyze all four ideas jointly and produce one global ranking from best to worst.Source: Prompt for Report Agent (Group Ranking) · p. 50
• Evaluation considers clarity, novelty, validity, feasibility, and significance while emphasizing relative strengths, weaknesses, and trade-offs.Source: Prompt for Report Agent (Group Ranking) · p. 51
• The ranking analysis must explain the ordering, identify each idea’s key strengths and weaknesses, and emphasize the factors driving the decision.Source: Prompt for Report Agent (Group Ranking) · p. 51
• The index list must contain each integer from 1 to 4 exactly once, ordered from highest-ranked to lowest-ranked, without additional text.Source: Prompt for Report Agent (Group Ranking) · p. 51