Source-linked AI summary
RWGBench: Evaluating Scholarly Positioning in Related Work Generation
Anzhe Xie, Weihang Su, Jiaxin Mao, Yiqun Liu, Min Zhang, Shaoping Ma, Qingyao Ai
TL;DR
Related work generation is a scholarly positioning task, yet common evaluations rely on text similarity and can miss citation errors. RWGBench evaluates citation decisions in a curated benchmark, and experiments report that retrieval-based systems struggle with citation selection while placement, organization, and framing challenges persist even when canonical references are supplied.
Problem
Existing RWG evaluations largely rely on lexical overlap, semantic similarity, or fluency, although these metrics can miss academically critical citation errors.
Source: §1 · p. 1Method
RWGBench evaluates RWG as citation decision-making using a curated computer science benchmark and a framework covering citation selection, placement, organization, and rhetorical framing.
Source: §1 · p. 2Results
Current retrieval-based systems struggle with citation selection, while GPT-5.5 improves citation recovery with canonical references but still faces placement, organization, and framing challenges.
Source: §7 · p. 9Takeaways & Limitations
RWGBench provides reproducible infrastructure and interpretable evaluation signals to support systematic, targeted progress in automated scholarly writing.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Large language models have shown strong fluency in scientific writing, yet the evaluation of related work generation (RWG) remains limited. Existing RWG evaluations largely inherit summarization-oriented metrics, using lexical or semantic similarity to reference sections as proxies for quality. However, related work writing is fundamentally a citation-level scholarly positioning task: it requires selecting, organizing, and framing prior work to clarify how a target paper relates to, differs from, and contributes beyond existing research.As a result, models may generate coherent and semantically-relevant text while exhibiting academically critical failures, such as inappropriate citation selection or misplaced references, that conventional metrics do not capture.To this end, we introduce \textbf{RWGBench}, a benchmark that evaluates RWG from the perspective of citation decision-making rather than text similarity. RWGBench is constructed from a large-scale collection of 40,108 computer science papers and a retrieval corpus of 1.09 million documents, with a carefully curated test set comprising 100 papers and their corresponding published related work sections.We propose a multi-dimensional evaluation framework that assesses citation selection, contextual appropriateness, organization, and discourse structure.Experiments reveal systematic limitations in current systems that are obscured by standard evaluations, while Oracle studies further disentangle retrieval-level and generation-level bottlenecks. Human evaluation further shows that our citation-centric metrics align substantially better with expert judgment than surface-level text metrics. RWGBench offers a citation-centric testbed for developing and evaluating related work generation systems that are better aligned with scholarly writing practices.
1 Introduction
Related work generation is a scholarly positioning task, but evaluations centered on text similarity miss important citation decisions. RWGBench evaluates these decisions with a curated benchmark and a multidimensional framework.
1 Introduction
- RWG positions a specific target paper by selecting, organizing, and framing prior work to clarify its relevance, differentiation, and contribution.These interdependent citation decisions shape how the target paper is perceived in the literature.Source: §1 · p. 1
- Summarization-oriented metrics can reward text resembling references while missing poorly matched, narrow, omitted, or misplaced citations.Existing benchmarks also provide limited support for evaluating document-level citation selection, placement, organization, and discourse-level positioning.Source: §1 · pp. 1–2
- RWGBench pairs 40,108 computer science papers and 1.09 million retrieval documents with a 100-paper peer-reviewed test set averaging 97.9% citation match.The framework evaluates citation selection, placement, organization, and rhetorical framing alongside generated text.Source: §1 · p. 2
- The paper reformulates RWG as scholarly positioning centered on citation decision-making and builds a benchmark with curated evaluation targets and a large-scale retrieval corpus.Its empirical analysis aims to diagnose failure modes and reports critical limitations in representative systems, including a GPT-5.5 generator suite.Source: §1 · p. 2
2 Task Definition and Problem Formalization
RWG is a document-level task conditioned on a target paper, in which retrieving candidate works and generating a related work section serve the broader goal of scholarly positioning.
2.1 Task Formulation
- RWG takes a target paper and a corpus of candidate papers as input, then produces a related work section that positions the target within its research context.The target may include a title, abstract, and optionally an introduction.Source: §2.1 · p. 2
- The task is commonly implemented as retrieval of candidate papers followed by generation of related work text with citations.Retrieval identifies candidate papers to cite; generation synthesizes the related work text.Source: §2.1 · p. 2
- The core challenge is the quality of citation decisions embedded in retrieval and generation, not text synthesis alone.The section describes the challenge as scholarly positioning, where citation selection, placement, and framing jointly situate the target paper.Source: §2.1 · p. 2, §2 · p. 2
2.2 Scholarly Positioning
- Scholarly positioning requires reasoning over interdependent citation selection, strategic placement, and rhetorical framing.These dimensions distinguish RWG from generic text generation or summarization.Source: §2.2 · p. 2
- Citation selection chooses works according to their relevance to the target paper’s context, limitations, and contributions.Topically related works may still play different roles in positioning the target.Source: §2.2 · p. 2
- Strategic citation placement reflects discourse structure and guides readers through the paper’s logical development, shaping interpretation of evidence and claims.Source: §2.2 · p. 2
- Rhetorical framing presents cited works as building on or contrasting with prior approaches, clarifying relationships between studies.The generated narrative should clarify the target paper’s relevance and differentiation.Source: §2.2 · p. 2
3 Dataset Construction
RWGBench is built from extracted computer science related work sections, a large retrieval corpus, and a curated peer-reviewed test set with high citation coverage.
3.1 Related Work Collection
- RWGBench collects 40,108 computer science papers published from 2020 to 2025 with extracted metadata, citation lists, and related work sections.The collection uses arXiv metadata and publicly available HTML, including sections with titles such as Related Work and Literature Review.Source: §3.1 · p. 2
3.2 Corpus Preparation
- The final retrieval corpus contains 1,091,394 papers, combining SurGE documents with successfully retrieved and normalized cited papers missing from that corpus.This hybrid strategy is intended to combine broad coverage with citation alignment.Source: §3.2 · pp. 2–3
3.3 Test Set Construction
- The peer-reviewed test set contains 100 exact-title matches: 42 ICLR, 34 NeurIPS, and 24 ICML papers from 2023 to 2025.Papers were filtered for at least 95% citation extraction rates and 10–50 citations; the shared split was evaluated across 52 configurations, producing 5,200 sections.Source: §3.3 · p. 3
3.4 Statistics and Analysis
- Across the full dataset, related-work sections average 673 words and 14.7 matched citation identifiers per paper; test papers are more citation-intensive, with 25.3 extracted entries and 24.7 matched identifiers.Source: §3.4 · p. 3
- 58.2% of extracted related-work citation entries are covered by the corpus on average in the full dataset, compared with 97.9% for the test set.All matched identifiers used as test gold labels are present in the corpus, supporting Oracle and retrieval-based analyses over a defined reference set.Source: §3.4 · p. 3
3.5 Ethical Considerations and Licensing
- The release separates code from document-derived content, redistributing text only when the corresponding license permits it.It provides identifiers, metadata, and extraction scripts; no full-text PDFs are redistributed.Source: §3.5 · p. 3
4 Evaluation Framework
The framework evaluates scholarly positioning through citation quality, citation organization, citation framing, and text quality.
4 Evaluation Framework
- Citation Quality covers citation-set selection and the local appropriateness of citations.Source: §4 · p. 3
- Citation Organization and Citation Framing capture section-level placement and rhetorical use.Source: §4 · p. 3
- Text Quality captures surface-level fluency and coherence alongside measures of scholarly citation decisions.Source: §4 · p. 3
4.1 Citation Quality
Citation quality is assessed by comparing generated citations with the authors’ references, checking whether citations support their local claims, and measuring topical alignment with the target paper.
4.1 Citation Quality
- Citation quality asks whether selected papers support the argument, rather than merely resembling it topically.Source: §4.1 · p. 3
4.1.1 Citation Precision, Recall, and Soft Recall.
- Citation precision and recall compare generated citations with the author-written reference set, while soft recall gives partial credit for topically compatible alternatives.Generated inline citation markers are resolved through the ordered candidate list; the soft-recall compatibility score uses citation titles and abstracts and grades entailment, neutral, and contradiction differently.Source: §4.1.1 · pp. 3–4
4.1.2 Claim-Support Consistency (CSC).
- Claim-Support Consistency uses NLI to assess whether each cited paper’s abstract supports the sentence in which it is cited.Entailment, neutral, and contradiction receive scores of 1.0, 0.5, and 0.0, respectively.Source: §4.1.2 · p. 4
4.1.3 Topical Relevance to Target Work (TRT).
- Topical Relevance to Target Work measures document-level topical alignment between generated citations and the target paper using abstract embeddings.The authors describe topical proximity as necessary but insufficient for citation quality.Source: §4.1.3 · p. 4
4.2 Citation Organization
Citation organization measures how citations are allocated across a related-work section, including both citation rate and distribution across cited papers.
4.2 Citation Organization
- The organization dimension evaluates citation allocation and distribution as strategic placement and emphasis decisions in scholarly positioning.Source: §4.2 · p. 4
4.2.1 Citation Density (CD).
- Citation density measures whether citations occur at a rate comparable to human-authored related-work sections, with extremes indicating under- or over-citing.Such extremes may reflect insufficient grounding or citation clustering that obscures argumentative focus.Source: §4.2.1 · p. 4
4.2.2 Citation Distribution Balance (CDB).
- Citation Distribution Balance measures the share of cited papers appearing exactly once, reflecting a balance between citation diversity and depth.Very low values indicate repetitive reliance on few papers; very high values may signal shallow engagement.Source: §4.2.2 · p. 4
4.3 Citation Framing and Structural Coherence
Citation framing and structural coherence evaluate whether related-work sections follow scholarly positioning conventions and use citations in comparable rhetorical roles.
4.3 Citation Framing and Structural Coherence
- This dimension evaluates global rhetorical structures such as contextualization, gap identification, and contribution positioning.Source: §4.3 · p. 4
4.3.1 Positioning Structure Similarity (PSS).
- Positioning Structure Similarity aligns ordered generated and reference content blocks with a soft longest-common-subsequence score.It penalizes missing, extra, or reordered blocks and is a reference-relative diagnostic, not a judgment that the author’s order is uniquely correct.Source: §4.3.1 · p. 4
4.3.2 Citation Frame Alignment (CFA).
- Citation Frame Alignment compares generated and reference counts for seven citation functions, including motivation, background, uses, extends, similarities, differences, and future work.It is a comparative framing diagnostic, not a claim that the reference distribution is the only acceptable profile.Source: §4.3.2 · p. 5
4.4 Text Quality
Surface-level text metrics are used alongside positioning measures to distinguish linguistic deficiencies from citation decision-making failures.
4.4 Text Quality
- Text quality is assessed with ROUGE-L for lexical overlap, BERTScore for semantic similarity, and an LLM judge for publication readiness while excluding citation correctness and coverage.The judge rates coherence, organization, fluency, and academic writing style on a normalized 0–1 scale.Source: §4.4 · p. 5
5 Experiments and Analysis
The experiments show that RWGBench surfaces distinct failures in citation selection, scholarly positioning, and surface writing quality; retrieval gains alone do not resolve citation use, placement, or framing.
5.1 Implementation Details
- The evaluation uses Direct generation and retrieval-augmented baselines with DeepSeek-V3, Llama-3-8B, Qwen-2.5-7B, and a GPT-5.5 frontier check.Retrieval uses BM25 or Dense over 1.09M documents; RAG, ALCE, and TAG use top-50 papers, while Workflow combines multiple top-20 query lists.Source: §5.1 · p. 5
5.2 Baseline Systems
- The baselines span standard RAG, prompt-based ALCE and TAG, iterative Workflow generation, Direct prompting, and Oracle runs with ground-truth citations.TAG uses keyphrase-guided organization and explicit target relationships; Workflow plans themes, retrieves through multiple queries, and iteratively refines content.Source: §5.2 · p. 5
5.3 Main Results
- Under GPT-5.5 RAG, Dense raises hard citation recall from 0.140 to 0.175; in Workflow, Dense raises it from 0.161 to 0.178.BM25 and Dense cover 0.144 and 0.182 of matched gold citations in top-50 RAG candidates; GPT-5.5 Workflow multi-query retrieval covers 0.167 under BM25 and 0.186 under Dense.Source: §5.3 · pp. 5–6, Table 2 · p. 6
- Text-quality scores can remain competitive despite citation failure, while generated sections also deviate from human organization and emphasis.GPT-5.5 Direct receives an LLM-as-Judge score of 0.720 but has zero citation recall; stronger generation improves readability and some framing without solving section-level organization.Source: §5.3 · p. 7
- With canonical references supplied, Oracle-RAG reaches 0.971 hard recall and 0.993 soft recall, yet placement and framing metrics remain below the human reference.In Oracle runs, 57 generations have no resolvable citation markers; excluding these, precision is 1.000 over 1,543 generations.Source: §5.3 · p. 7
5.4 Metric Complementarity and Case Study
- Citation-set metrics are weakly correlated with PSS, CFA, and LLM-as-Judge, supporting their use as distinct diagnostic signals.Qualitative cases show that soft recall can identify plausible alternatives, Oracle recall isolates generation-level framing failures, and text metrics can remain high with ungrounded citations.Source: §5.4 · p. 7
5.5 Evaluation Consistency Analysis
- On non-tied pairs, GPT-5.5 agrees with Claude and Gemini on 95.3% and 93.4% of pairwise orderings, respectively.The audit covers 265 sections; saturated external scores on the GPT-only subset limit within-GPT ranking.Source: §5.5 · p. 7
5.6 Human Validation
- Human validation finds strongest metric alignment on each metric’s intended preference dimension, with Soft Recall aligning with overall usefulness and CSC matching all seven clear support-quality comparisons.Six annotators completed 60 tasks; the validation supplements rather than replaces the full automatic benchmark.Source: §5.6 · p. 8
6 Related Work
Prior work addresses scientific writing, citation generation, retrieval, and text-generation evaluation, but leaves document-level scholarly positioning insufficiently assessed. RWGBench focuses evaluation on citation selection, organization, and framing across a complete related-work section.
6.1 Automated Scientific Writing
- Survey generation synthesizes broad topic-level literature, whereas related-work generation positions a specific target paper within prior research and lacks standardized benchmarks [11] [38] [43] [6] [25].Discourse studies treat citation choice and organization as rhetorical functions, not merely content summarization.Source: §6.1 · p. 8
6.2 Citation Recommendation and Generation
6.3 Retrieval-Augmented Generation
- RAG expands grounding and knowledge access, but retrieval alone does not ensure that cited papers are integrated strategically into a related-work argument [24] [10] [16] [17].Source: §6.3 · pp. 8–9
6.4 Benchmarks for Scientific Text Generation
- OARelatedWork evaluates with ROUGE and BERTScore, while RWGBench targets citation decisions; DeepScholar-Bench reports aggregate synthesis scores with limited diagnosis of citation-level failures [9] [32] [36].RWGBench combines fine-grained diagnosis of selection, placement, and organization with Oracle experiments that separate retrieval- and generation-level limitations.Source: §6.4 · p. 9
7 Conclusion
RWGBench evaluates related work generation as scholarly positioning using a curated peer-reviewed corpus and a multidimensional framework. Results show citation-selection difficulties and persistent generation challenges even when canonical references are supplied.
7 Conclusion
- RWGBench combines 100 curated peer-reviewed computer science papers with author-written related work sections, a retrieval corpus of over 1.09 million papers, and multidimensional evaluation.Source: §7 · p. 9
- Current retrieval-based systems struggle with citation selection, while stronger GPT-5.5 generators improve writing quality and citation recovery under matched retrieval conditions.Source: §7 · p. 9
- Providing canonical references substantially improves GPT-5.5 citation recall, but citation placement, organization, and framing challenges remain.Source: §7 · p. 9
- RWGBench provides reproducible infrastructure and interpretable evaluation signals intended to support systematic, targeted progress in automated scholarly writing.Source: §7 · p. 9