PaperlayerView exact revision on arXiv

Source-linked AI summary

Expert Preference-based Evaluation of Automated Related Work Generation

Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych

arXiv:2508.07955v3cs.CL

TL;DR

Scientific-writing evaluation must account for domain-specific criteria and expert preferences, which conventional metrics and LLM judges may miss. GREP evaluates related-work drafts through fine-grained criteria and contextual examples in an iterative feedback process. Its judgments align more closely with experts than standard LLM judges, while current LLMs still struggle with related-work constraints.

  • Problem

    Conventional metrics and LLM-as-a-judge systems can miss the domain-specific criteria and expert preferences needed to evaluate scientific writing.

    Source: §1 · p. 1

  • Method

    GREP evaluates related-work drafts using hard and soft criteria, localized judgments, contrastive examples, and iterative natural-language feedback.

    Source: §1 · p. 2

  • Results

    GREP matched expert judgments on citation faithfulness at 78% with PreciseGREP and 66% with OpenGREP, versus 53% for specialized SoTA LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GREP’s module outputs can serve as proxies for human judgment when assessing LLM performance, reducing the cost of human-in-the-loop experimentation.

    Source: §6 · p. 9

    The dataset prioritizes quality and evaluation diversity over scale because large datasets incur LLM-call costs and manual-curation overhead.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Expert domain writing, such as scientific writing, typically demands extensive domain knowledge. Although large language models (LLMs) show promising potential in this task, evaluating the quality of automatically generated scientific writing is a crucial open issue, as it requires knowledge of domain-specific criteria and the ability to discern expert preferences. Conventional automatic evaluation metrics and LLM-as-a-judge systems, primarily designed for mainstream NLP tasks, are insufficient to grasp expert preferences and domain-specific quality standards. To address this gap and support realistic human-AI collaborative writing, we focus on related work generation, one of the most challenging scientific tasks, as an exemplar. We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences. GREP decomposes the evaluation into smaller fine-grained dimensions. This localized evaluation is further augmented with contrastive examples to provide detailed contextual guidance for the evaluation dimensions. Empirical investigation reveals that GREP is able to assess the quality of related work sections in a much more robust manner compared to standard LLM judges, reflects natural scenarios of scientific writing, and bears a strong correlation with the assessment of human experts. We also observe that generations from state-of-the-art LLMs struggle to satisfy validation constraints of a suitable related work section.

1 Introduction

Scientific writing requires domain knowledge and expert judgment, but conventional evaluation can miss domain-grounded reasoning and expert preferences. GREP evaluates related-work drafts against fine-grained criteria and uses feedback in an iterative generation process.

1 Introduction

  • Conventional LLM judges face pretraining bias, weak domain-grounded reasoning, misalignment with expert preferences, and opaque judgments.The paper frames the missing guidance as both what to judge and how to judge it.Source: §1 · p. 1
  • GREP combines hard validity constraints with soft constraints reflecting expert preferences, using gold related-work sections as preference examples.Its PreciseGREP variant uses proprietary judges, while OpenGREP uses open-weight models.Source: §1 · p. 2
  • GREP assessments matched expert judgments more closely than specialized SoTA LLMs on citation faithfulness: 78% for PreciseGREP and 66% for OpenGREP versus 53%.The expert study involved 16 domain experts evaluating generated related-work sections pairwise.Source: §1 · p. 2
  • Frontier LLMs struggle to cite prior work coherently; o3-mini did so in only 20% of cases, and preference changes can sharply degrade quality.Reported failure modes include inconsistent edits and difficulty following simple length instructions; changing preferences can prompt new papers or section reorganization.Source: §1 · pp. 2–3

2 Related Work

Prior related-work generation research varies its inputs and increasingly uses agentic pipelines, but often lacks cited-paper context and expert-preference-aware evaluation. This paper positions GREP as an expert-domain evaluation approach beyond LLM-as-a-judge systems.

2 Related Work

  • Earlier related-work generation methods used summarization-oriented inputs, while LLM-era work expanded to citation intent, citation spans, and agentic retrieval or outlining pipelines.The cited works include Yasunaga et al. (2019), Xing et al. (2020), Arita et al. (2022), and Shi et al. (2023).Source: §2 · p. 3
  • Prior frameworks often omit sufficient cited-paper context and expert preferences such as a paper’s position in the literature or the emphasis given to each cited work.This paper uses cited-paper introductions and considers expert preferences in generation and evaluation.Source: §2 · p. 3
  • ROUGE and BERTScore miss expert domain requirements, while LLM judges and general checklists can lack robustness or instance-specific criteria for iterative collaboration.The paper describes GREP as instance-specific evaluation grounded in expert preferences.Source: §2 · p. 3
  • The authors characterize their work as the first to frame automated related-work evaluation as an expert-domain task and develop evaluation techniques beyond LLM-as-a-judge.They state that these techniques address limitations of judge systems.Source: §2 · p. 3

3 Methodology

GREP combines an extended-context related-work dataset with hard validity checks and soft preference criteria. Its iterative framework evaluates drafts and turns evaluation reports into feedback for the next generation.

3.1 Dataset

  • The dataset extends abstract-, metadata-, citation-intent-, and example-sentence context with paper context; manual curation aggregates complete cited-paper sets from heterogeneous sources, yielding 50 main papers, 725 cited papers, and 13.50 per main paper.Source: §3.1 · pp. 3–4

3.2 Evaluation criteria

  • Hard constraints define essential validity requirements, while soft constraints model preferences among valid drafts using gold sections as examples, not uniquely optimal targets.Criteria include citation faithfulness and positioning, plus length and the relative content allocated to each citation; the preferences focus on aspects that can be specified, measured, and controlled.Source: §3.2 · p. 4

3.3 Evaluation framework

  • GREP decomposes evaluation into criteria-specific judgments with contextual examples, then interleaves evaluation and generation to produce report-based feedback for the next draft.The examples include manually checked synthetic failure cases; the framework is designed to simulate multi-turn human–AI interaction.Source: §3.3 · p. 5

4 Experiments

The experiments select evaluators through meta-evaluation, validate GREP with domain experts, and assess a range of language models as related-work generators.

Evaluator selection

  • Evaluator selection used three manually verified 50-sample benchmarks for faithfulness, positioning type, and positioning ratio, with three repeated runs and majority voting.The authors report a clear preliminary gap between proprietary and open models; PreciseGREP uses GPT-4o and o3-mini, while OpenGREP uses Gemma 3 and Llama 3.3.Source: §4 · p. 5

Domain expert evaluation

  • Sixteen senior PhD students and postdoctoral researchers from NLP and computer vision evaluated paired model drafts and provided independent feedback across iterations.Experts assessed faithfulness, positioning, and instruction following without gold sections, shaping their own drafts independently; the study also compared evaluator judgments with expert judgments.Source: §4 · pp. 5–6

RW Generator evaluation

  • The generator evaluation ran five iterations, testing ten language models with OpenGREP and four selected models with PreciseGREP.The models span multiple families and scales; the four PreciseGREP models were selected by systematic sampling from models ranked by average scores.Source: §4 · p. 6

5 Results

GREP outperforms judge-model baselines and aligns with expert judgments across NLP and CV evaluations, while generated related-work sections still struggle with citation and preference constraints.

5 Results

  • Across NLP and CV evaluations, GREP variants outperform judge-model baselines; GREP closely matches ordinal expert judgments in NLP and remains robust across domains.Table 2 reports match rates with expert judgments.Source: §5 · p. 6, Table 2 · p. 6
  • Few papers satisfy all hard constraints in the first iteration; faithfulness is hardest, and feedback’s gains vary by model.Figure 2 reports criterion scores averaged across related-work sections; o3-mini is generally best at passing hard constraints within the dataset.Source: §5 · pp. 6–7, Figure 2 · p. 7
  • Feedback quickly narrows the faithfulness advantage; most models do not improve, while o3-mini starts failing more often.Across models, failing to cite provided papers is more common than hallucinating nonexistent ones.Source: §5 · p. 8
  • For soft constraints, o3-mini adjusts length best, Gemma 3 steadily improves citation emphasis, but all models struggle to reflect author preferences in citation allocation.Models almost perfectly reflect the expected positioning type, with Llama 3.3 as an exception at the paragraph level.Source: §5 · p. 8
  • Introducing papers at iteration three raises missing-paper failures, but feedback helps models recover; dynamic introduction also improves citation emphasis.The citation-emphasis improvement is especially apparent for Gemma 3.Source: Figure 3 · p. 7, §5 · p. 8
  • After style changes at iteration three, positioning-type and positioning-ratio scores decline, and two feedback iterations do not restore initial performance.The evaluation detects failures to adapt to changing user preferences in this simulated interaction.Source: Figure 4 · p. 7, §5 · p. 8
  • Incorrect citation formats can trigger parsing errors that harm missing-paper and faithfulness measurements; splitting feedback into smaller steps may help, but adds cost.The passage notes that correcting one feature can sometimes deteriorate another.Source: §5 · p. 8

6 Conclusion

GREP evaluates related-work generation through specialized modules grounded in expert preferences. Its outputs are presented as proxies for human judgment, with potential to reduce human-in-the-loop experimentation costs.

6 Conclusion

  • GREP uses specialized modules for different aspects of related-work evaluation, providing granular interpretations based on expert preferences.The framework is designed to simulate scientific-writing collaboration with evolving human preferences.Source: §6 · p. 8
  • OpenGREP and PreciseGREP are described as variants supporting robust generalization across domains and diverse research communities.The paper characterizes one variant as high-precision and proprietary, and the other as cost-efficient and open-source.Source: §6 · p. 8
  • Module outputs can serve as proxies for human judgment when assessing LLM performance, reducing human-in-the-loop experimentation costs.Source: §6 · p. 9

Limitations

GREP’s LLM-driven evaluations remain susceptible to errors from insufficient domain-grounded reasoning. The dataset prioritizes quality and diversity over scale, while future work could add search agents and investigate nuanced stylistic preferences.

Limitations

  • GREP’s LLM-driven evaluations remain susceptible to errors because of limited domain-grounded reasoning.Specialized modules and contrastive few-shot examples are used to mitigate this known challenge.Source: Limitations · p. 9
  • The dataset prioritizes quality and evaluation diversity over scale because very large datasets increase LLM-call costs and manual-curation overhead.Source: Limitations · p. 9
  • Future work could incorporate search agents and examine nuanced stylistic author preferences, such as active versus passive voice.Search agents could be used to find relevant papers and evaluate their combined performance.Source: Limitations · p. 9

A Dataset

The dataset is constructed to provide cited-paper information for related-work generation, combining existing resources with paper PDFs. It excludes non-open-license cited papers and drops citing papers when removals leave critical gaps or too little text.

A Dataset

  • S2ORC and OARelatedWork provide content for 57% of cited papers; content for the rest is collected from PDFs retrieved through metadata URLs.Source: A Dataset · p. 13
  • PDF parsing problems are corrected manually, and cited papers without open licenses are excluded along with their associated text segments.Source: A Dataset · p. 13
  • A citing paper is dropped if removed citations are critical to its related-work section or the remaining content is too short.Source: A Dataset · p. 13

B Evaluation Methodology

The evaluation methodology operationalizes citation emphasis, faithfulness, and paper positioning through distinct checks. It uses tolerance-based citation-emphasis scoring, paper-specific faithfulness comparisons, and contrastive examples without enforcing a positioning template.

B Evaluation Methodology

  • Citation-emphasis evaluation compares generated and gold citation emphasis, returning the mean share of cited papers whose emphasis falls within a tolerance threshold.The procedure allocates sentence tokens to the currently cited papers and checks each gold citation against upper and lower bounds.Source: B Evaluation Methodology · p. 13

B.1 Faithfulness

  • Faithfulness is evaluated per cited paper by comparing citation sentences with the paper’s abstract and introduction, then averaging passing sentence-paper pairs.A valid related-work section should attain a perfect score of 1.0.Source: §B.1 · p. 14

B.2 Positioning

  • Positioning examples convey common ways to express a paper’s differences, but the method avoids explicit templates to minimize potential bias.Source: §B.2 · p. 14

B.3 Length

The appendix describes the length constraint and tolerance choice, then lists supporting evaluation, prompt, and pipeline configuration details. It also specifies how open evaluators were run and sampled.

B.3 Length

  • A tolerance ratio of 0.25 lets the length test detect meaningful deviations across iterations and models, and is used in the main experiments.Small tolerances produce oscillations between too-long and too-short sections; large tolerances pass most sections and make deviations harder to assess.Source: §B.3 · p. 14

B.4 Citation Emphasis

  • Citation emphasis uses the same tolerance ratio as the length criterion: 0.25.The appendix says it provides an algorithmic representation of citation-emphasis evaluation.Source: §B.4 · p. 14

B.5 Contrastive Few-shot Examples

  • Contrastive few-shot examples are provided for faithfulness, positioning type, and contribution-positioning ratio evaluation.The examples appear in Tables 5–11, grouped by evaluation dimension.Source: §B.5 · p. 14

B.6 Evaluation Framework

  • The appendix provides an algorithmic representation of GREP.The passage points to the framework algorithm without describing its steps.Source: §B.6 · p. 14

C.1 Evaluation Prompts

  • Evaluation-stage prompts are provided for faithfulness, positioning type, and positioning-ratio evaluation.The appendix points to Tables 15–18 for these prompts.Source: §C.1 · p. 15

C.2 Generation Prompts

  • Generation-stage prompts cover draft generation and feedback generation.The appendix points to Tables 12–14.Source: §C.2 · p. 15

D.1 Pipeline Configurations

  • Open-weight LLMs run locally through vLLM with 4-bit quantization on one 80GB NVIDIA A100; OpenAI models use API version 2025-03-01-preview.Temperature is 0.8 except for o3-mini, which does not support temperature adjustment; other parameters remain at defaults.Source: §D.1 · p. 15

D.2 Open Evaluators

  • OpenGREP results are reported separately for hard and soft constraints, while PreciseGREP uses four sampled models to span model families and performance levels within cost constraints.OpenGREP scores average across papers and cover the full pipeline without new-paper introduction or style changes.Source: §D.2 · p. 15

D.3 Full Results

The full-results discussion reports imperfect hard-constraint passing and greater variability for some generator checks, while LLM-based evaluations have lower standard deviations. It also describes evaluation costs and points to figures tracking changes across iterations.

D.3 Full Results

  • GPT-4o and o3-mini are selected for LLM-supported evaluation dimensions because they outperform other models.The passage does not specify the comparison metrics.Source: §D.3 · p. 15
  • Repeated evaluation can reach five-digit costs, so multiple-run experiments use a smaller subset of 10 papers.A paper requires five iterations, and evaluation cost scales with the number of citations and repeated faithfulness checks.Source: §D.3 · p. 15
  • Hard constraints for hallucinated and missing papers pass at high but not consistently perfect rates; length and citation-emphasis checks are poor for nearly all models except o3-mini.These fluctuations lead to higher standard deviations, while LLM-based evaluation results have lower standard deviations.Source: §D.3 · p. 15

D.4 Performance Changes Over Iterations

  • Figures 6–8 show performance changes across iterations.The passage does not state the direction or size of those changes.Source: §D.4 · p. 15

E Expert Evaluation

This section presents an expert evaluation of generated drafts across iterations, covering citation faithfulness, positioning, instruction following, and feedback-following. It documents the evaluation prompts and contrastive examples, and reports pairwise expert judgments, model scores, and pipeline results across four models and three settings.

Expert Evaluation

  • Experts supplied 48 judgments per criterion across three iterations, covering citation faithfulness, positioning, and instruction following.The study reports 16 experts × 3 iterations for each criterion.Source: E Expert Evaluation · pp. 15, 17
  • To reduce participant burden, the study uses pairwise comparisons and TrueSkill™ to dynamically rank generators and select informative model pairs.A complete comparison set is impractical because the number of comparisons grows quadratically with the number of models.Source: E Expert Evaluation · p. 17
  • For GREP systems, the higher-scoring draft at each iteration is selected; feedback-following is measured by improvement between consecutive iterations.Baselines use higher-scoring drafts or pairwise selection, depending on their structure.Source: E Expert Evaluation · p. 17
  • Participants received aspect-specific instructions and examples, interface guidance, and missing or hallucinated paper information plus length evaluations to reduce cognitive load.An instructional video demonstrated use of the evaluation interface.Source: E Expert Evaluation · p. 17
  • Expert alignment uses faithfulness and positioning scores, while feedback-following is assessed through relative score improvements across evaluation modules.A model with greater improvement across modules is considered to have followed feedback more effectively; the Self-taught Evaluator directly selects among drafts using evaluation-perspective explanations.Source: E Expert Evaluation · pp. 17–18
  • Draft-generation prompts for the first and later iterations, plus a feedback-generation prompt, are included.Source: Table 12 · p. 27, Table 13 · p. 28, Table 14 · p. 28
  • The evaluation materials cover faithfulness, positioning existence and type, and positioning-ratio checks at paragraph and final-paragraph comparison levels.They include contrastive examples for comparing a context related-work paragraph with the final paragraph.Source: Table 11 · p. 26, Table 16 · p. 29, Table 15 · p. 29
  • Full Pipeline results report means and standard deviations across iterations for GPT-4o, o3-mini, Llama 3.3, and Gemma3.Source: Table 19 · p. 31, Table 20 · p. 31, Table 21 · p. 31
  • New Paper Pipeline results report means and standard deviations across iterations for GPT-4o, o3-mini, Llama 3.3, and Gemma3.Source: Table 23 · p. 32, Table 24 · p. 32, Table 25 · p. 32
  • Style Change Pipeline results report means and standard deviations across iterations for GPT-4o, o3-mini, Llama 3.3, and Gemma 3.Source: Table 27 · p. 33, Table 28 · p. 33, Table 29 · p. 33

Related papers

Loading 2508.07955v3…