Source-linked AI summary
Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications
Ethan Lin, Zhiyuan Peng, Yi Fang
TL;DR
LLM novelty evaluation has focused mainly on semantic creativity, while scholarly novelty remains largely unexamined. The paper introduces SchNovel, a 15,000-pair benchmark, and RAG-Novelty, which retrieves similar papers to emulate review practices. RAG-Novelty outperforms recent baseline models, but the evaluation relies on titles, abstracts, and metadata rather than complete papers.
Problem
Scholarly novelty assessment remains largely unexamined, although scholarly novelty depends on new ideas, methods, or insights relative to existing research.
Method
The paper constructs SchNovel from 15,000 arXiv paper pairs across six fields and proposes RAG-Novelty, which retrieves similar papers to support novelty assessment.
Results
RAG-Novelty outperforms recent baseline models in assessing novelty in scholarly papers.
Takeaways & Limitations
SchNovel enables systematic evaluation of LLMs’ scholarly novelty assessment, while RAG-Novelty improves that assessment within the benchmark.
Takeaways & Limitations
The evaluation uses titles, abstracts, and metadata rather than complete papers, so abstracts may omit technical details needed for deeper novelty assessment.
Abstract
from arXiv · showhide
Recent studies have evaluated the creativity/novelty of large language models (LLMs) primarily from a semantic perspective, using benchmarks from cognitive science. However, accessing the novelty in scholarly publications is a largely unexplored area in evaluating LLMs. In this paper, we introduce a scholarly novelty benchmark (SchNovel) to evaluate LLMs' ability to assess novelty in scholarly papers. SchNovel consists of 15000 pairs of papers across six fields sampled from the arXiv dataset with publication dates spanning 2 to 10 years apart. In each pair, the more recently published paper is assumed to be more novel. Additionally, we propose RAG-Novelty, which simulates the review process taken by human reviewers by leveraging the retrieval of similar papers to assess novelty. Extensive experiments provide insights into the capabilities of different LLMs to assess novelty and demonstrate that RAG-Novelty outperforms recent baseline models.
1 Introduction
Existing creativity benchmarks mainly measure semantic novelty, leaving scholarly novelty assessment largely unexamined. The paper introduces SchNovel and RAG-Novelty to evaluate and improve LLM-based assessment of novelty in scholarly papers.
- Motivation: Scholarly novelty concerns new ideas, methods, or insights relative to past and current research, unlike cognitive-science benchmarks focused on semantic creativity.Its assessment depends on understanding research trends and relationships across time.
- SchNovel benchmark: SchNovel provides 15,000 paper pairs across six fields, assuming the more recently published paper in each pair is more novel.Publication gaps range from 2 to 10 years, and LLM accuracy is evaluated on the resulting comparisons.
- RAG-Novelty: RAG-Novelty retrieves similar papers to emulate a human reviewer’s literature-based novelty assessment and outperforms recent baseline models.The method is proposed specifically to enhance LLM performance on scholarly novelty assessment.
- Experimental analysis: The benchmark examines how categories, starting years, and publication-year gaps affect LLMs’ ability to assess paper novelty.These experiments broaden evaluation beyond a single benchmark configuration.
2 Related Work
Prior work evaluates LLM creativity with cognitive-science benchmarks, human or LLM-based scoring, and increasingly elaborate prompting strategies. RAG-Novelty instead uses retrieved scholarly papers to support novelty assessment.
- 2.1 Existing Benchmarks: Cognitive-science benchmarks assess creativity through tasks such as picture responses and generating unconventional uses, using metrics including fluency, flexibility, originality, and elaboration.Examples include TTCT, AUT, and rCAB.
- 2.2 Prompting Methods: Prompt-based methods improve creativity-task performance through basic prompting, associative thinking, multi-step reasoning, role-play, and discussion frameworks.RAG-Novelty differs by retrieving similar papers as additional evidence for novelty assessment.
- 2.2 Evaluation Methods: Existing studies evaluate LLM outputs using human assessments or stronger LLM judges, rather than directly evaluating scholarly novelty.These approaches are applied to benchmarks such as TTCT and other creativity tasks.
3 Scholarly Novelty Benchmark
The paper defines scholarly novelty as introducing ideas, methods, or discoveries not previously established in the literature and operationalizes it through temporal pairwise comparisons. SchNovel samples 15,000 arXiv paper pairs across six fields, using titles, abstracts, and metadata rather than full text.
- 3 Scholarly Novelty: Scholarly novelty differs from semantic novelty because it concerns contributions that have not previously been explored or established in the literature.Assessing it requires understanding relationships between ideas across time.
- 3 Scholarly Novelty: SchNovel establishes ground truth by assuming that the later paper in a same-field pair is more novel than the earlier paper.This temporal assumption is applied to the benchmark’s labels.
- 3.1 Dataset Construction: The benchmark contains 15,000 arXiv paper pairs sampled from six fields, with each pair encoding field, publication year, year gap, paper identifiers, and a novelty label.The dataset is sampled from an arXiv corpus containing approximately 2.5 million articles.
- 3.2 Tasks and Evaluation Metrics: The task asks an LLM to choose the more novel paper using each paper’s title, abstract, and optional metadata, without access to full text.This limits context while reducing token consumption and cost, though important details may be missed.
4 RAG-Novelty
RAG-Novelty retrieves temporally appropriate similar papers and supplies their information to an LLM, emulating how reviewers use related literature. The method incorporates retrieved-paper dates into novelty evaluation while avoiding future-paper leakage.
- 4 RAG-Novelty: RAG-Novelty addresses LLMs’ limited access to the current research climate by retrieving similar scholarly papers for novelty assessment.The approach is inspired by retrieval-augmented generation and supplements paper information such as abstracts.
- 4.1 Indexing and Retriever: The retriever encodes abstracts into embeddings and uses cosine-similarity search to select relevant papers, with FAISS available for large candidate corpora.The indexing procedure samples papers from 2000 to 2023 for each field.
- 4.1 Indexing and Retriever: For pairwise comparisons, retrieval can extend through the more recent paper’s publication date, matching the knowledge available when judging novelty.The method also filters out papers published after the query paper in the realistic review setting.
- 4.2 Prompt Design: RAG-Novelty builds on self-reflection prompting by adding retrieved-paper information and an instruction that later average publication dates generally indicate greater novelty.The prompt averages retrieved documents’ publication dates and uses that average as evaluation context.
5 Experimental Setup
The experiments compare prompting and retrieval-based approaches for LLM novelty assessment, examining model, paper, publication, and metadata factors. RAG-Novelty is evaluated against recent baselines, including on SciEval with GPT-4o-mini.
- The experiments compare pointwise scoring with pairwise comparison and test zero-shot, two-shot, chain-of-thought, and self-reflection prompting strategies.Pointwise comparison scores each paper before comparing them, while the prompting variants alter demonstrations or reasoning instructions.
- The evaluation examines how LLM performance varies across models, research categories, publication start years, publication-year gaps, and other metadata attributes.These factors correspond to the study’s research questions about model capability and the effects of paper and metadata characteristics.
- The study evaluates whether RAG-Novelty outperforms recent baseline methods for assessing scholarly novelty.This is one of the study’s stated research questions, and Table 1 reports averaged accuracy for RAG-Novelty versus baselines on SciEval with GPT-4o-mini.
6 Experimental Results
Experiments show that comparison context, field, model choice, metadata, and affiliation affect scholarly-novelty assessment. RAG-Novelty achieves the strongest overall results, while mathematics and position or affiliation biases remain important limitations.
- 6.1 RAG-Novelty vs. Baseline Models (R7): RAG-Novelty achieves the highest overall results, significantly outperforming the second-best method except in mathematics.Self-Consistency is the strongest baseline; mathematics shows limited improvement, possibly because of slower field progression, complex symbols, or insufficient training data.
- 6.1 RAG-Novelty vs. Baseline Models (R7): Pairwise comparison is consistently much better than pointwise comparison across different year gaps.The result supports providing both papers as context when assessing relative novelty.
- 6.3 The Impact of Different Fields (R3): Computer science has the highest accuracy across most year gaps, whereas mathematics and physics have lower accuracy amid domain-specific challenges.The reported challenges include complex mathematical notation and theoretical physics frameworks; limited domain knowledge in training data is another proposed explanation.
- 6.4 The Impact of Start Years (R4): Results are relatively consistent across five start years, suggesting that year gap matters more than the specific publication years.For example, a ten-year gap from 2009 to 2019 is described as comparable in difficulty to one from 2013 to 2023.
- 6.5 The Impact of Different LLMs (R2): GPT-4o-mini outperforms the other evaluated models, while smaller models show stronger position biases tied to paper order.Mistral 7b favors the second paper, whereas LLaMA 3.1-8b favors the first paper appearing toward the prompt’s middle.
- 6.6 The Impact of Metadata (R6): Adding TLDR metadata slightly decreases accuracy but makes ascending- and descending-year results nearly equal, mitigating position bias.The metadata experiment uses GPT-4o-mini with Self-Reflection in computer science.
- 6.6.3 Adding Affiliation: Affiliation bias appears as models increasingly favor papers associated with a top research university as the year gap grows.The top-university condition is nevertheless outperformed by the no-affiliation condition, leaving the processing of affiliation information unresolved.
7 Conclusion and Future Work
The paper introduces SchNovel for evaluating LLM novelty assessment in scholarly publications and proposes RAG-Novelty to improve that capability.
- SchNovel contains 15,000 paper pairs across six fields for evaluating LLMs’ ability to assess novelty in scholarly publications.
- RAG-Novelty significantly outperforms strong baseline models in comprehensive experiments.
- Future work includes expanding the benchmark, identifying which paper sections best represent whole-paper novelty, and studying affiliation and position biases.
8 Limitations
The study assesses novelty using paper titles, abstracts, and metadata, but abstracts may omit technical details needed for comprehensive evaluation.
- Novelty assessment uses a paper’s title, abstract, and metadata rather than its complete text.
- Because abstracts can omit important technical details, this streamlined input may limit the depth of novelty assessment.
A.1 Statistics of arXiv
Figure 6 reports the number of papers for each field in the arXiv dataset through 2023.
- Figure 6 presents the number of papers for each field in the arXiv dataset up to 2023.
A.2 Zero-shot
The zero-shot evaluations ask LLMs to compare two papers’ novelty from titles and abstracts, using structured criteria and scoring; RAG-Novelty adds retrieved-paper context and publication dates.
- Zero-shot evaluation: Zero-shot prompts ask models to identify each paper’s problem and solution, assess uniqueness and creativity, and select the more novel paper.
- Zero-shot evaluation: The evaluation instructs models to judge novelty from titles and abstracts while ignoring the random presentation order.
- SchNovel construction: SchNovel samples papers across six fields, start years, and year gaps, pairing a paper from year s with one from year s-g.
- RAG-Novelty: RAG-Novelty retrieves similar abstracts, averages their publication dates, and uses the resulting temporal context alongside title-abstract content for novelty assessment.