PaperlayerView exact revision on arXiv

Source-linked AI summary

CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding

Dustin Wright, Isabelle Augenstein

arXiv:2105.10912v2cs.CLcs.DLcs.LG

TL;DR

Scientific NLP lacks abundant labels across diverse fields. The paper introduces contextualized CITEWORTH and reports that paragraph-level modeling improves cite-worthiness detection by 5.31 F1 points over the existing state of the art.

  • Problem

    Scientific NLP faces domain-specific, diverse language and a shortage of labeled data; scientific-task datasets are often costly to annotate, small, and limited to a few fields.

    Source: §1 · p. 1

  • Method

    The paper curates CITEWORTH, an English cite-worthiness dataset that labels every sentence in its scientific-document paragraphs to preserve context.

    Source: §1 · p. 2

  • Results

    The paragraph-level contextualized model improves cite-worthiness detection by 5.31 F1 points over the existing state of the art, and transfer learning achieves state-of-the-art citation-intent results.

    Source: §7 · p. 9

  • Takeaways & Limitations

    CITEWORTH provides a testbed for studying domain adaptation and is suitable for downstream natural-language understanding tasks.

    Source: §7 · p. 9

    CITEWORTH's 30.7M training and development tokens are small compared with SciBERT's 3.1B-token training corpus, and improvements on other tasks tend to be minimal.

    Source: §6 · p. 8

Abstract

from arXiv · show

Scientific document understanding is challenging as the data is highly domain specific and diverse. However, datasets for tasks with scientific text require expensive manual annotation and tend to be small and limited to only one or a few fields. At the same time, scientific documents contain many potential training signals, such as citations, which can be used to build large labelled datasets. Given this, we present an in-depth study of cite-worthiness detection in English, where a sentence is labelled for whether or not it cites an external source. To accomplish this, we introduce CiteWorth, a large, contextualized, rigorously cleaned labelled dataset for cite-worthiness detection built from a massive corpus of extracted plain-text scientific documents. We show that CiteWorth is high-quality, challenging, and suitable for studying problems such as domain adaptation. Our best performing cite-worthiness detection model is a paragraph-level contextualized sentence labelling model based on Longformer, exhibiting a 5 F1 point improvement over SciBERT which considers only individual sentences. Finally, we demonstrate that language model fine-tuning with cite-worthiness as a secondary task leads to improved performance on downstream scientific document understanding tasks.

1 Introduction

Scientific NLP has limited labeled data despite the domain-specific diversity of scientific language, and document structure offers signals for creating larger datasets. The paper introduces CITEWORTH and reports gains from contextual cite-worthiness modeling and transfer to downstream tasks.

1 Introduction

  • Scientific NLP lacks abundant labels, and existing datasets are often small and domain-limited, despite document structure and citations offering potential training signals.Source: §1 · p. 1
  • Cite-worthiness detection identifies sentences containing an external-source reference and has applications in document editing and citation recommendation.Source: §1 · p. 1
  • Manual evaluation supports CITEWORTH's quality, while findings show large cross-field generalization differences and significant gains from sentence context.Source: §1 · p. 2
  • Transfer learning with cite-worthiness improves citation-intent performance over SciBERT and achieves state-of-the-art citation-intent results.Source: §1 · p. 2
  • Paragraph-level contextual modeling improves cite-worthiness detection by 5 F1 points over SciBERT.The model uses the paragraph surrounding each sentence.Source: §1 · p. 2
  • CITEWORTH contains 1.2M rigorously cleaned sentences labeled for cite-worthiness and balanced across 10 scientific fields.Every sentence in each paragraph is labeled in order, retaining paragraph context.Source: §1 · p. 2

2 Related Work

Prior cite-worthiness datasets and studies were limited in domain coverage and analysis, and had not examined sentence context. The paper situates its work as extending cite-worthiness to transfer learning across scientific text tasks.

2.1 Cite-Worthiness Detection

  • Earlier cite-worthiness datasets had class imbalance, narrow domain coverage, limited data-quality analysis, and no study of sentence-context effects.Source: §2.1 · p. 2
  • Cite-worthiness had improved citation-intent classification as an auxiliary multitask objective, but its transfer-learning use across scientific text problems remained unstudied.Source: §2.1 · p. 2
  • Scientific document datasets are often small and restricted to a few fields, limiting cross-domain study; CITEWORTH spans 10 fields for domain adaptation and downstream tasks.Source: §2.1 · p. 3

3 RQ1: CITEWORTH Dataset Construction

CITEWORTH is built by filtering and cleaning scientific-paper paragraphs so cite-worthy sentences can be labelled without obvious citation artifacts. Manual evaluation finds higher extraction and marker-removal quality than a baseline that removes citation spans alone.

3 RQ1: CITEWORTH Dataset Construction

  • The construction starts from S2ORC and removes citation markers to avoid models learning trivial prediction signals.S2ORC provides structured article text and metadata, including linked citations and MAG categories.Source: §3 · p. 3

3.1 Data Filtering

  • Filtering restricts positive examples to sentence-final parenthetical author-year or bracketed-numerical citations and excludes inline citations and unmatched formats.Sentences must also meet extraction checks, including capitalization, terminal punctuation, and a minimum length of 20 characters.Source: §3.1 · p. 3

3.2 Extracting Cite-Worthy Sentences in Context

  • Paragraph-level extraction retains sentence context, checks for missed or non-final citations and hanging markers, and samples evenly from 10 MAG fields.Paragraphs are drawn from sections with permitted titles, and every sentence must pass the construction checks.Source: §3.2 · p. 4

3.3 Manual Evaluation

  • 6.83% absolute improvement in extraction quality and 5.32% in citation-marker removal; 1.1% of the sample had cleaning issues and 1.9% retained trivial citation markers.The comparison is against removing citation markers based only on provided citation spans.Source: §3.3 · p. 5, Table 3 · p. 5

4 RQ2: System Evaluation5

The system evaluation finds that contextualized Longformer sentence labeling performs best on CITEWORTH, while model explanations and errors reveal difficult lexical cues. The section also examines how model and training choices affect detection.

4 RQ2: System Evaluation5

  • 67.45 F1 for Longformer-Ctx, 5.31 points above the next-best model, shows the strongest result for paragraph-contextualized prediction.Each sentence is represented by its [SEP] token output after the paragraph is processed by Longformer.Source: Table 4 · p. 6, §4 · p. 6
  • Longformer-Ctx achieves the best F1 (67.45), 5.31 points above the next-best model; class weighting adds over 4 F1 points, while PU learning beats unweighted training and yields the highest recall.Source: §4 · p. 6
  • Explanations show that models use lexical cues such as “previous work,” but difficult examples expose these cues as unreliable when they conflict with the label.The model misclassified “briefly discussed” as non-cite-worthy and “described earlier” or “previous work” as cite-worthy in examples with the opposite labels.Source: §4 · p. 6

5 RQ3: Domain Evaluation

Domain evaluation finds that scientific fields differ in their citation data and in cite-worthiness model performance. Performance is strongest in-domain, and embedding-space similarity is associated with better out-of-domain results.

5 RQ3: Domain Evaluation

  • Embedding clusters pair Engineering with Computer Science and Biology with Chemistry; their cluster purity is 57.61.The analysis interprets the separated domain clusters as evidence that fields draw from different data distributions.Source: §5 · p. 7
  • The five-field evaluation tests all 25 train/test combinations, with in-domain splits using 80% training, 10% validation, and 10% test data.The selected fields are Chemistry, Engineering, Computer Science, Psychology, and Biology.Source: §5 · p. 7, Table 5 · p. 7
  • Same-field training gives the best performance for each split, while performance varies across train domains and is inversely correlated with embedding-space distance.The standard deviation in maximum performance across fields is 3.32.Source: §5 · p. 7, Table 5 · p. 7

6 RQ4: Cite-Worthiness for Transfer Learning

The section tests whether CITEWORTH fine-tuning transfers to downstream scientific document tasks, with the strongest influence on citation intent classification. Fine-tuning generally improved performance, though results varied by task.

6 RQ4: Cite-Worthiness for Transfer Learning

  • SciBERT was fine-tuned on CITEWORTH for cite-worthiness detection, masked language modeling, or both before downstream task fine-tuning.The joint model trains both objectives on each batch, summing their losses.Source: §6 · pp. 7–8
  • CITEWORTH fine-tuning generally improved downstream task performance, except for NER on the NCBI-Disease corpus.The reported SciBERT results were rerun locally for fair comparison.Source: §6 · p. 8
  • 1.8 F1 points on ACL-ARC and 0.5 F1 points on SciCite were the average improvements, with the best average performance from CITEBERT.CITEBERT combines masked language modeling and cite-worthiness as objectives.Source: §6 · p. 8
  • CITEWORTH's train and development splits contain 30.7M tokens, compared with 3.1B tokens in SciBERT's original training corpus.The authors note that incorporating more data or using cite-worthiness during language-model pre-training could potentially yield further improvements, but this was outside the study's scope.Source: §6 · p. 8

7 Conclusion

The conclusion presents CITEWORTH and a paragraph-level model as contributions to cite-worthiness detection, and reports transfer-learning results for citation intent classification. The supplied appendix passages also identify section-title conventions and model-resource tables.

7 Conclusion

  • The conclusion presents CITEWORTH as a rigorously curated dataset for cite-worthiness detection.The conclusion also notes potential uses in downstream language-understanding tasks and future work on citation recommendation and author writing styles.Source: §7 · pp. 8–9
  • The paragraph-level contextualized model improves cite-worthiness detection by 5.31 F1 points over the existing state of the art, and transfer learning achieves state-of-the-art citation intent results.The conclusion also describes CITEWORTH as a testbed for studying domain adaptation in scientific text.Source: §7 · p. 9

A List of Permissible Section Titles

  • Appendix A lists permissible section titles, including introduction, methods, results, discussion, and conclusion variants.The list also includes titles such as literature review, methodology, and experimental design.Source: A List of Permissible Section Titles · p. 11

B List of Regular Expressions

  • Table 7 reports average runtimes for each model, while Table 8 gives the number of parameters per model.The supplied captions do not include the table values.Source: Table 7 · p. 11, Table 8 · p. 11

C Reproducibility

The reproducibility passages describe the shared-cluster computing setup and point to tables reporting model runtimes and parameter counts.

C.1 Computing Infrastructure

  • All experiments ran on a shared cluster; requested jobs consisted of 16GB RAM and 4 Intel Xeon Silver 4110 CPUs, and experiments used one NVIDIA Titan X GPU with 12GB RAM.Source: §C.1 · p. 11

C.2 Average Runtimes

  • Table 7 reports average runtimes for each model over the entire experiment run.Runs may have been placed on different nodes in the shared cluster.Source: §C.2 · p. 11

C.3 Number of Parameters per Model

  • Table 8 reports the number of parameters in each model.Source: §C.3 · p. 11

C.4 Validation Performance

The validation-performance section points readers to average validation results for each tested model.

C.4 Validation Performance

C.5 Evaluation Metrics

The evaluation uses F1 score, and the appendix reports model-specific hyperparameter searches and final settings.

C.5 Evaluation Metrics

  • F1 score was the primary evaluation metric.The authors used sklearn's precision_recall_fscore_support implementation.Source: §C.5 · p. 12

C.6 Hyperparameters

  • Basic Transformer used batch size 64, 33 epochs, and learning rate 0.0001406 after Bayesian grid search optimizing validation F1.Source: §C.6 · p. 12
  • BERT, SciBERT, and Longformer-Ctx each used 3 epochs and 300 warmup steps, with batch sizes 8, 4, and 4 respectively.Source: §C.6 · p. 12

C.7 Data

CiteWorth is built from S2ORC, specifically its 20200705v1 release.

C.7 Data

  • CiteWorth is constructed from the S2ORC dataset.Source: §C.7 · p. 12
  • The dataset uses the 20200705v1 release of S2ORC.Source: §C.7 · p. 12
Loading 2105.10912v2…