CORWA: A Citation-Oriented Related Work Annotation Dataset

1 Introduction

• Because related work reviews shared prior work and can be similar in content and format, the authors motivate generating these sections automatically.In NLP, literature reviews are usually presented in a Related Work section.Source: §1 · p. 1
• Related work generation systems that treat sections as sentence-level summaries miss variable-length fragments from different sources and non-citation sentences with discourse roles.The authors argue that these heterogeneous fragments should be distinguished rather than treated as homogeneous sentences.Source: §1 · pp. 1–2
• CORWA labels NLP related work by discourse role and marks citation spans, distinguishing detailed explanations of cited papers from citations illustrating high-level concepts.The dataset is presented as a first step toward linguistically motivated related work generation.Source: §1 · p. 2
• The paper’s contributions include a baseline tagger for unlabeled related work text, evidence that citation spans outperform citation sentences on two tasks, and a human-in-the-loop iterative abstractive framework.The dataset also decomposes annotation into discourse tagging, citation span detection, and citation type recognition.Source: §1 · p. 2

2 Related Work

• Extractive approaches select sentences or cited-text spans, but often neglect concise, abstractive reference-type citations.Examples include topic-based selection and coverage methods, as well as matching cited-text spans.Source: §2 · p. 2
• Abstractive approaches generate citation sentences or paragraphs using context, cited-paper information, citation networks, or document representations (Luu et al., 2021).The described systems use different combinations of inputs and objectives.Source: §2 · p. 2
• Earlier work largely emphasized generation over dataset collection, and differing problem definitions made its datasets and models difficult to reuse and compare.CORWA focuses on a dataset intended for varied related work generation settings rather than another incomparable approach.Source: §2 · p. 2

3 CORWA Dataset

• The dataset draws on NLP papers in S2ORC, extracts related work sections by section-title matching, and prioritizes papers whose cited works are mostly available.The authors limit the scope to NLP publications.Source: §3 · pp. 2–3

3.1 Annotation Scheme

• CORWA decomposes related work annotation into discourse tagging, citation span detection, and citation type recognition.These tasks address sentence roles, the text derived from cited papers, and whether a citation is discussed in detail or illustrates a high-level concept.Source: §3.1 · p. 3, §3.1.2 · p. 3, §3.1.3 · p. 3

3.1.1 Discourse Tagging

• Discourse tagging assigns one of six labels: single_summ, multi_summ, narrative_cite, reflection, transition, or other.The labels distinguish detailed summaries of one or multiple works, high-level citation commentary, reflection on the current work, transitions, and sentences outside the related work section.Source: §3.1.1 · p. 3
• Sentences outside the related work section caused by section-boundary detection errors receive the other label.The section boundaries are extracted automatically using heuristics based on section titles.Source: §3.1.1 · p. 3

3.1.2 Citation Span Detection

• Citation spans cover text whose information derives directly from a cited paper, from a summary across sentences to a citation mark alone.Spans are bounded by paragraph boundaries because a citation rarely crosses them without a new explicit mark.Source: §3.1.2 · p. 3

3.1.3 Citation Type Recognition

• Citation type recognition labels detailed discussion of a cited work as dominant and use of a work to illustrate a high-level concept as reference.Dominant citations are usually discussed through summarization and are often longer than reference citations.Source: §3.1.3 · p. 3

3.2 Annotation Process and Agreement

• Inter-annotator agreement was κ = 0.824 for discourse tagging, 0.965 for citation type recognition, and 0.878 for citation span detection.Agreement was calculated on 50 re-annotated test sections; citation-type and span labels were converted to token-level labels for this calculation.Source: §3.2 · p. 4
• Model-assisted correction sped annotation and enabled a larger dataset, but annotators could inherit model-prediction errors they failed to notice.The authors suggest this bias may explain why model performance exceeds inter-annotator agreement.Source: §3.2 · p. 4

3.3.1 Relations among CORWA Subtasks

• Dominant citation spans average 34.5 tokens, compared with 8.2 tokens for reference spans.Dominant spans often summarize cited papers in detail, while reference spans are more abstract and occur in narrative sentences discussing high-level ideas.Source: §3.3.1 · p. 4, Figure 2 · p. 4

3.3.2 Related Work Writing Styles

• 34.6% of paragraphs contain only summarization sentences, 32.1% only narrative sentences, and 33.3% mix both styles.The mixed paragraphs are neither purely descriptive nor purely integrative.Source: §3.3.2 · p. 5
• Frequent label sequences include single_summ followed by reflection for comparisons, and single_summ followed by transition for more impersonal criticism.The reflection pattern usually compares cited work with the current work without directly criticizing it; transition avoids direct comparison.Source: §3.3.2 · p. 5

4 Joint Related Work Tagger

• The joint related work tagger is trained with multi-task learning across discourse tagging, citation span detection, and citation type recognition.The model is intended to propagate CORWA labels to massive unlabeled related-work sections.Source: §4 · p. 5

4.1 Model Design

• A transformer encoder processes related-work paragraphs, sharing representations across tasks; citation labels are decoded token by token and discourse labels at sentence level.The paragraph-wise encoding reflects the independence of paragraphs in CORWA citation-span annotations.Source: §4.1 · p. 5, Figure 4 · p. 5

4.1.1 Paragraph Encoder

• The selected encoder is SciBERT, a BERT variant trained on scientific text with domain-specific tokenization that includes NLP papers.The authors experimented with several pretrained transformer encoders before focusing on SciBERT.Source: §4.1.1 · p. 5

4.1.2 Task-specific Decoders

• Citation spans and citation types use BIO2 token tagging, while discourse tagging applies sentence-level attention and classification.Citation-type labels distinguish dominant and reference citations; discourse tagging decodes sentence encodings into discourse labels.Source: §4.1.2 · pp. 5–6

4.1.3 Multi-task Learning

• The model balances the three task losses with a weighted sum of cross-entropy losses and tuned task-specific weights.The loss terms correspond to discourse tagging, citation span detection, and citation type recognition.Source: §4.1.3 · p. 6

4.2 Experiments

• More than 0.9 test F1 was achieved on each of discourse tagging, citation type recognition, and citation span detection by the final distantly supervised tagger.The authors use token-level F1 for citation span detection and citation type recognition.Source: §4.2 · p. 6, Table 2 · p. 6

5.1 Queries for Relevant Sentence Retrieval

• Reference-span queries retrieve cited sentences with higher average ROUGE scores than reference-citation sentences, while dominant-span and sentence queries perform similarly.None of the reference spans exceed one sentence, while 27.7% of dominant spans are shorter than a sentence, 46.6% are equal to one sentence, and 25.7% are longer.Source: §5.1 · pp. 6–7, Figure 6 · p. 7

5.2.1 Experimental Setting

• The pilot span-generation task conditions generation on the target introduction, the partial related-work paragraph, and cited-paper information, targeting the gold citation span.The setup includes explicit citation marks, titles, and abstracts for cited papers.Source: §5.2 · p. 7, §5.2.1 · p. 7

5.2.2 Experimental Results

• Span-based generation scores are similar to prior sentence-level models, and pretraining improves span-generation performance, but the task scores are not directly comparable.Span-level generation scores are lower for dominant citations and higher for reference citations than sentence-level generation, with different target texts preventing direct comparison.Source: §5.2.2 · p. 8
• Human judges rate all citation texts highly for fluency, with moderate inter-annotator agreement.The reported Kendall’s τ values among the three annotators are 0.298, 0.205, and 0.172.Source: §5.2.2 · p. 8
• Gold spans receive significantly lower relevance scores than generated spans, plausibly because judges lack cited-paper body text that some gold spans draw on.The authors note that gold citation spans are not necessarily the best target for every task setting.Source: §5.2.2 · p. 8
• Generated spans are rated slightly higher than generated sentences overall, supporting span-level citation text generation.Dominant sentences and spans receive similar human scores, while reference sentences receive lower relevance scores than reference spans.Source: §5.2.2 · p. 8

6 Toward Full Related Work Generation

• Prior systems do not address sentence ordering, paragraph grouping, or rhetorical transitions, and none bridge individual citation-text generation to a full related-work section.The proposed pipeline is also presented as mitigating resource limits of long-text generation.Source: §6 · p. 8
• The proposed pipeline generates citation spans, aggregates and rewrites them into citation text blocks, then forms paragraphs with transition and reflection sentences.Human involvement is intended to let users prune the search space and reduce error propagation during iterative generation.Source: §6 · p. 8

6.1.1 Scientific Document Understanding

• CORWA’s discourse tagging distinguishes information sources in related-work sentences and complements discourse tagging schemes focused on rhetorical components.The latter study rhetorical components in clauses, sentences, or spans beyond citations.Source: §6.1.1 · p. 9

6.1.2 Cited Text Span

• Cited Text Span datasets and tasks identify cited-paper sentences underlying citations, supporting precise citation-text generation rather than relying mainly on abstracts or introductions.The passage describes both annotated CTS data and the CL-Scisumm shared task.Source: §6.1.2 · p. 9

6.1.3 Studies of Literature Reviews

• Descriptive literature reviews detail individual studies, whereas integrative reviews emphasize synthesized ideas and results across papers.The cited studies analyze properties of these two review styles.Source: §6.1.3 · p. 9

7 Conclusion

• CORWA comprises discourse tagging, citation span detection, and citation type recognition, with analyses of writing style and discourse patterns.Source: §7 · p. 9
• A baseline model propagates CORWA labels to massive unlabeled related-work sections.Source: §7 · p. 9
• The paper reports citation spans as a better alternative to citation sentences for retrieval and generation, and proposes human-in-the-loop iterative abstractive generation.Source: §7 · p. 9

A.1 Training Configurations

• Joint tagger training lasts 2.5 hours on one GPU with SciBERT, BERT-base, or RoBERTa-base encoders, and 6.5 hours with LED-base.Models are trained for 15 epochs; five-fold hyperparameter search across language models takes approximately one week using eight GPUs.Source: §A.1 · p. 13
• Citation-span generation training runs for two days on one Tesla V100 GPU, for up to three epochs with early stopping on validation loss.Source: §A.1 · p. 13

• The dataset is derived from S2ORC, released under CC BY-NC 2.0, while the developed Huggingface models are released under Apache License 2.0.Source: §A.2 · p. 13
• Annotators were paid at twice the local minimum wage.Source: §A.2 · p. 13