Source-linked AI summary
On the Limits of LLM-as-Judge for Scientific Novelty Assessment
Soumitra Sinhahajari, Navonil Majumder, Soujanya Poria
TL;DR
Reliable scientific-novelty assessment is needed as LLMs increasingly generate and judge research ideas. The paper introduces RQ-Bench to compare model-generated research questions with author-anchored questions from real papers, finding a novelty mirage: LLM judges favor generated questions while experts prefer the references.
Problem
The reliability of LLM-as-judge systems for evaluating the scientific novelty of research questions remains uncertain.
Method
RQ-Bench reconstructs author-anchored research questions from cited backgrounds, gaps, and contributions to compare them with model-generated questions.
Results
LLM judges rate model-generated questions as highly novel and favor them comparatively, whereas domain experts prefer author-anchored questions for their broader scope and non-obviousness.
Takeaways & Limitations
Contradictory LLM and expert evaluations question the reliability of using LLMs to assess research-question novelty, particularly when narrowness is overlooked.
Takeaways & Limitations
RQ-Bench covers recent computer-science papers from arXiv, and broader, more diverse human studies are still needed.
Abstract
from arXiv · showhide
LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.
1 Introduction
The paper isolates research-question novelty as a simpler upstream evaluation target and introduces RQ-Bench with author-anchored questions reconstructed from real papers. Across standalone and comparative LLM judging, models favor generated RQs, whereas human experts prefer the author-anchored references and detect narrow or source-bound scope.
- Motivation: The study evaluates research questions because judging complete ideas also requires assessing methods, feasibility, experiments, baselines, and likely impact.An RQ opens an investigative space without committing to a method or answer, preceding the proposed idea.
- RQ-Bench: RQ-Bench reconstructs author-anchored reference questions from cited works, local citation contexts, and the gaps addressed by each target paper.These references represent questions pursued by real authors, while acknowledging that the same background can support many valid RQs.
- Findings: LLM judges assign high novelty to model-generated RQs and prefer them over author-anchored RQs in standalone evaluation.The paper frames this as a reliability test of LLM-as-judge for scientific novelty assessment.
- Findings: Comparative LLM judging amplifies the preference for generated RQs, producing a sharp increase in their win rates.The evaluation compares standalone scoring, comparative scoring, and human expert judgments.
- Findings: Human experts prefer author-anchored RQs far more often than LLM judges and find generated RQs narrow or source-bound.Explicitly evaluating narrowness aligns the results with experts’ assessment.
2 Related Work
Prior work spans autonomous scientific discovery, benchmarks for generated scientific artifacts, and LLM-based judging. This paper focuses on research-question formulation as a distinct pre-ideation object for evaluating scientific novelty.
- Autonomous Scientific Discovery: Automated scientific discovery has progressed from rule-based systems and domain-specific machine learning to foundation-model systems acting as AI scientists.Recent systems can propose ideas, write code, run experiments, and draft papers.
- LLM Ideation and Research Benchmarks: Existing benchmarks mainly evaluate downstream artifacts such as generated ideas, hypotheses, and data-driven reasoning.IdeaBench, AI Idea Bench, and LiveIdeaBench evaluate generated ideas, while ResearchBench and DiscoveryBench assess hypothesis composition and data-driven reasoning.
- LLM-as-Judge for Scientific Novelty: LLM-as-judge methods are widely used for open-ended generations, but scientific novelty is high-level, partly subjective, and sensitive to framing.Human peer review can also disagree, so a single novelty score should not be treated as absolute.
- Research Question Formulation: Research-question formulation turns a gap in prior work into an object of inquiry and differs from both a topic and a method.The paper uses this distinction to build RQ-Bench and test LLMs and LLM judges at the pre-ideation stage.
3 Formulation
The formulation models research-question generation as inferring grounded gaps from background papers and then writing a question from those gaps. This keeps RQs grounded in prior work while allowing broader scientific knowledge to connect papers and transform gaps into questions.
- 3 Formulation: Research-question generation proceeds in two steps: infer grounded gaps from background papers, then write an RQ from those gaps.Given background papers P = {p1, . . . , pm}, the model produces a question q through an intermediate gap set G = {g1, . . . , gk}.
- 3 Formulation: The formulation treats an RQ as grounded in prior work rather than as a free-form idea.Grounding is imposed by deriving the question from gaps inferred from the supplied papers.
- 3 Formulation: The model may also use broader scientific knowledge to connect papers and turn inferred gaps into questions.This broader knowledge supplements, rather than replaces, grounding in the background papers.
4 Benchmark Dataset Creation
RQ-Bench reconstructs author-anchored research questions by linking influential citations, grounded literature gaps, and source-paper ideas and contributions. The resulting benchmark contains 1,434 research questions from 746 arXiv computer science papers, with verification indicating strong faithfulness to their source papers.
- RQ reconstruction: The pipeline extracts influential citations, source-paper ideas and contributions, and grounded gaps before synthesizing author-anchored research questions.Grounded gaps constrain each question to how the target paper uses prior work to motivate its contributions.
- Dataset scale: 1,434 research questions were derived from 746 source papers spanning 13 arXiv computer science sub-fields.The dataset also includes 2,464 RQ-citation links, 1,375 unique cited papers, and 3,151 distinct gap statements.
- Paper sourcing and filtering: The benchmark samples recent arXiv computer science papers, retaining the top 10% by citation counts within each subfield when LATEX sources are available and publication venues are indexed.The source pool covers about ∼200K papers published between 01-01-2025 and 10-4-2026, while author-anchor overlap is reported as a memorization diagnostic.
- Citation identification: Influential citations are classified as inspiration or competitor/baseline, while irrelevant, minimally influential, and canonical references are excluded from the influential set.Classification uses citation titles, frequencies, and the sections in which citations appear.
- Verification: Gemini approved 98% of extracted questions against full source papers, and human judges approved 100% of 50 randomly sampled questions.These checks were used to assess whether the reconstructed questions were plausibly addressed by their source papers.
5 Evaluation Setup
The evaluation scores research questions separately for originality, gap addressing, and non-obviousness, using standalone and comparative judging against author-anchored references. It additionally evaluates scope through source-boundedness and diagnostic framing, and reports overlap with the source paper’s direction.
- Novelty metrics: Three novelty metrics—Orig, Gap, and NonObs—receive separate scores from 0 to 3 rather than being summed into a single novelty score.They measure originality, gap addressing, and non-obviousness.
- Scoring settings: Standalone scoring evaluates each RQ independently, whereas comparative scoring places the author-anchored reference and five generated RQs in the same context.Standalone scores measure absolute rubric values; comparative scores measure whether a judge prefers at least one generated RQ over the reference.
- Comparative evaluation: Metric-level Pass@5 compares the best of five generated RQs with the author-anchored reference, while combined wins require both judges to score the generated RQ higher.For comparative scoring, the evaluation reports win, tie, and lose rates rather than average comparative scores.
- Scope metrics: Scope is evaluated separately with SrcBound and DiagFrame, scored from 0 to 3, with lower scores indicating broader research questions.Source-boundedness captures closeness to one source element, while diagnostic framing captures analysis- or failure-diagnosis-oriented questions.
- Author-anchor overlap: Author-anchor overlap measures whether a generated RQ follows the source paper’s direction on a 0–4 scale, but it is not an absolute quality score.Low overlap can indicate that the model generated a different question from the same background.
6 Experiments
The experiments show that LLM judges often favor generated research questions over author-anchored references, especially under comparative scoring, while human experts prefer the references and identify narrower, source-bound generation. This disagreement challenges the reliability of LLM-as-a-judge novelty assessment.
- Task Formulation: Models receive source papers, extract methodological limitations or open challenges, and formulate exactly five specific, answerable research questions.For multi-paper scenarios, they are instructed to cross-reference the literature.
- Author-Anchor Overlap: Most models score close to 1.0 on Overlap@5m, while gpt-5.5 reaches 1.33, indicating broad topical alignment but frequent divergence from the author-pursued question.The metric is a diagnostic of author-anchor overlap, not an absolute quality score.
- Standalone Scoring: Under standalone judging, almost all model-generated RQs receive higher best-of-five novelty scores than the author-anchored reference across non-obviousness, originality, and gap addressing, although strict win rates remain low.gpt-5.5 consistently achieves the highest mean scores across the three metrics.
- Comparative Scoring: Under two-judge comparative scoring, gpt-5.5’s non-obviousness win rate rises from 27.2% to 49.1%, while its tie rate falls from 59.1% to 36.8%.The shared-context comparison increases strict wins for generated RQs and breaks ties in their favor.
- Human Expert Evaluation: Human–LLM agreement on non-obviousness falls as low as 22%, while experts prefer the author-anchored GT with win rates of 78% and 56%.Within-type agreement is higher: 60% for Expert–Expert and 52% for LLM–LLM evaluations.
- Human Expert Evaluation: Experts report that generated questions often imitate research-gap structure but remain less original, less non-obvious, narrow, and tied to the background papers.The findings suggest LLM judges may reward surface gap language while missing deeper novelty.
7 Scope and Narrowness: What LLM Judges Miss in Novelty Scoring
LLM-generated research questions are narrower than author-anchored references, chiefly because of source-boundedness, yet LLM novelty judgments often miss this limitation. Explicit narrowness assessments expose the mismatch, while comparative scoring amplifies it.
- Semantic Similarity with Background Papers: Across both embedding models and evaluation settings, author-anchored research questions show higher semantic similarity to background papers than generated questions.This indicates semantic similarity does not capture the narrowness identified by human experts.
- Narrowness Results and Discussions: gpt-5.5’s source-boundedness lose rate rises from 71.7% standalone to 85.4% comparative, while diagnostic framing loss increases from 2.9% to 53.4%.Comparative presentation amplifies judges’ baseline preference for broader author-anchored questions.
- Narrowness Results and Discussions: LLM judges select author-anchored references as less source-bound than gpt-5.5 in 82–90% of cases, with human non-obviousness agreement reaching 72–76%.These direct narrowness assessments support experts’ observations that generated questions are restricted in scope and lack non-obviousness.
- Prompt Adjustment: Removing constraining prompt instructions lowers gemini-3.1-pro’s source-boundedness from 1.00 to 0.47 but degrades novelty scores across originality dimensions.Broader generated questions still receive lower non-obviousness preference from human experts than author-anchored references.
8 Conclusion
The paper uses research-question generation as an inspectable testbed and introduces RQ-Bench with author-anchored questions from real papers. It finds a novelty mirage in LLM judging that conflicts with domain-expert preferences, showing current pipelines do not yet provide reliable scientific RQ novelty evaluation.
- Benchmark and setup: RQ generation serves as an upstream, easier-to-inspect testbed than complete methods or research plans.The study uses it before full idea generation because research questions are easier to inspect than complete methods or research plans.
- Benchmark and setup: RQ-Bench provides author-anchored research questions from real papers as reference points for comparing model-generated and human-anchored questions.The author-anchored questions are reference points rather than the only valid questions, enabling tests of novelty-judgment stability.
- Main findings: LLM judges produce a novelty mirage, rating model-generated research questions highly novel across non-obviousness, originality, and gap-addressing, while comparative judging amplifies preference for model outputs.The result holds under standalone scoring in every listed novelty dimension, and comparative settings heavily favor model outputs over author-authored baselines.
- Main findings: Domain experts sharply disagree with LLM judges, preferring author-anchored research questions for broader scope and non-obviousness.This disagreement is central to the paper’s concern about the reliability of LLM-as-judge pipelines for scientific novelty.
- Implications and limitations: Current LLM-as-judge pipelines do not yet meet the standard of stable scientific RQ novelty evaluation that avoids strong conflict with expert judgment.The paper notes that novelty remains partly subjective and that human reviewers can also disagree.
- Implications and limitations: Future evaluations should use multiple judges, check agreement, and treat scope or narrowness as a first-class dimension.These recommendations follow the finding that useful evaluation should be stable across settings and include dimensions that LLM judges may miss.
Limitations
The benchmark is limited to recent computer science papers from arXiv, while novelty evaluation remains difficult because it depends centrally on LLM-as-judge. Broader and more diverse human studies are still needed.
- Benchmark scope: The benchmark covers recent computer science papers from arXiv and may not capture research practices in other scientific domains.This limits how broadly the findings can be generalized across scientific fields.
- Evaluation: Novelty evaluation remains challenging because open-ended research question generation depends centrally on LLM-as-judge.Expert evaluations are included to investigate this dependence, but it remains a central limitation.
- Human validation: Broader and more diverse human studies are needed to further assess novelty evaluation.The current expert evaluations do not fully replace wider human validation.
A Appendix
The appendix reports the distribution of source papers and research questions across arXiv computer-science subfields.
- Table 13 summarizes source-paper and research-question distributions across arXiv CS subfields.
A.1 Novelty Metric
The novelty metric evaluates candidate research questions along three 0–3 dimensions: originality, gap addressing, and non-obviousness. Higher scores indicate greater departure from the background, engagement with substantive gaps, and unexpected or non-trivial formulation.
- Novelty dimensions: Originality (0–3) measures whether the candidate’s framing or object of inquiry goes beyond what is directly stated or asked in the background.Scores range from 0 for a background paraphrase to 3 for a framing absent from the background.
- Novelty dimensions: Gap addressing (0–3) measures whether the candidate engages an explicit or implicit gap in the background.Scores range from 0 for reiterating established content to 3 for targeting a substantive, central gap.
- Novelty dimensions: Non-obviousness (0–3) measures whether the candidate represents a non-trivial step beyond what is in the background.Scores range from 0 for a trivial follow-up to 3 for a striking or unexpected angle not hinted at by the background.
A.2 Narrowness Metric
The narrowness metric evaluates research questions along two dimensions: dependence on the source paper and diagnostic or test-list framing. Both dimensions use 0–3 scales, with higher scores indicating greater source-boundedness or more measurement-oriented framing.
- Source-boundedness (0–3) measures whether a candidate depends on the background paper’s method, dataset, benchmark, component, result, behavior, or setting.
- A score of 0 indicates a broader problem, new mechanism, setting, or direction unrelated to the source paper’s contribution.
- Scores of 1–2 range from reusing some source concepts while going beyond the paper to mainly asking about its method, result, behavior, setting, or a close variant.
- A score of 3 directly studies, diagnoses, tunes, evaluates, or modifies the source paper’s own contribution.
- Diagnostic / test-list framing (0–3) measures whether a candidate is posed as an analysis, ablation, correlation, causal test, predictor study, or measurable-factor list.
- Its scale runs from 0 for methods, mechanisms, frameworks, or broader directions to 3 for measurement checklists, ablations, causal or correlation tests, failure analyses, or predictor studies.
A.3 Author-Anchor Overlap Metric
The author-anchor overlap metric independently scores each generated research question against its author-anchored reference across four dimensions. A 0–4 pairwise scale ranges from unrelated questions to questions that subsume the reference.
- Metric dimensions: The metric evaluates topic overlap, object of inquiry, expected answer type, and scope relative to the author-anchored reference.Scope captures whether the generated question is broader, narrower, or equal to the reference.
- Pairwise overlap scale: A score of 0 means the questions are unrelated, with no topic overlap and no informative relationship between their answers.
- Pairwise overlap scale: A score of 1 indicates the same topic but a different question, while 2 indicates substantial overlap with a distinct scope or angle.At level 1, differences in object, answer type, or scope prevent answer overlap; at level 2, one answer only partially covers the other.
- Pairwise overlap scale: A score of 3 denotes semantic equivalence, whereas 4 means the generated question strictly subsumes the reference through fully containing scope.Level 4 requires that any complete answer also answers the reference; vague wording alone does not qualify.
A.4 Prompts … A.8 Alternate RQ Generation Prompt
The appendix specifies standalone and comparative novelty prompts that score originality, gap addressing, non-obviousness, and relevance against background papers. It also defines narrowness and author-anchor overlap evaluations, with example outputs showing model RQs can receive strong novelty scores while expert judgments may prefer generated or author-anchored questions.
- A.4 Prompts: Novelty is evaluated against background papers using originality, gap_addressing, non_obviousness, and hierarchical relevance scoring.Off-topic candidates receive relevance = 0 and all three novelty dimensions are set to 0.
- A.4 Prompts: Comparative judging scores candidates independently, ranks them by total novelty score, and breaks ties using additional novelty dimensions.The ranking is a permutation of candidate indices in most-to-least novel order.
- System: Narrowness is decomposed into independent source_boundedness and diagnostic_framing scores rather than overall question quality, novelty, or feasibility.The prompt explicitly prevents increasing one score merely because the question has the other property.
- System: Comparative narrowness evaluates each candidate against the same background independently and returns one JSON score object per candidate in presentation order.The scores array length must equal the number of candidates.
- System: Author-anchor overlap uses four dimensions—topic, object of inquiry, expected answer type, and scope—to assign each candidate a 0–4 pairwise score.A score of 4 means the candidate is strictly more general and fully contains the author-anchored RQ.
- System: The minimal covering subset contains the smallest candidate-index set whose union is semantically equivalent to the author-anchored RQ, or [] when no subset covers it.Any candidate scoring 3 or 4 alone determines the covering subset.
- AUTHOR-ANCHORED RQ Score: 8: For the DeepSeek-R1 example, the author-anchored compression question receives originality = 3, gap_addressing = 3, non_obviousness = 2, and relevance = 1.The best standalone model RQ is attributed to gemma-4-31b-it with Score: 8, while other listed model scores range from 5 to 7.
- A.6 Expert Evaluation Outputs: Expert verdicts differ across examples: gpt-5.5 wins all listed comparisons in one output, whereas another has GT winning twice and tying once.The second output records Expert 1 and Gemini-3.1-pro choosing GT, Expert 2 choosing GT, and DeepSeek-v4-pro producing a tie.