Source-linked AI summary
Structural Scaffolds for Citation Intent Classification in Scientific Publications
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, Field Cady
TL;DR
Citation-intent classification matters for analyzing scientific literature, but existing models rely on hand-engineered features and datasets are often small, fine-grained, or domain-specific. The paper uses neural multitask learning with section-title and citation-worthiness scaffolds, achieving state-of-the-art ACL-ARC performance and introducing the larger, multi-domain SciCite dataset.
Problem
Citation-intent classification is important for scientific-literature analysis, while existing models and datasets rely on hand-engineered features or remain small, fine-grained, and domain-specific.
Method
A neural multitask framework uses section-title prediction and citation-worthiness prediction as structural scaffolds for citation-intent classification.
Results
67.9% F1 on ACL-ARC is a new state-of-the-art, a 13.3% absolute increase over previous results, while SciCite is at least five times larger and spans multiple scientific domains.
Takeaways & Limitations
Structural properties of scientific discourse can inform citation-intent classification, and carefully chosen auxiliary tasks can improve performance on a related main task.
Takeaways & Limitations
Adding extended surrounding context caused a large decline in overall performance, likely because of noise introduced by the additional context.
Abstract
from arXiv · showhide
Identifying the intent of a citation in scientific papers (e.g., background information, use of methods, comparing results) is critical for machine reading of individual publications and automated analysis of the scientific literature. We propose structural scaffolds, a multitask model to incorporate structural information of scientific papers into citations for effective classification of citation intents. Our model achieves a new state-of-the-art on an existing ACL anthology dataset (ACL-ARC) with a 13.3% absolute increase in F1 score, without relying on external linguistic resources or hand-engineered features as done in existing methods. In addition, we introduce a new dataset of citation intents (SciCite) which is more than five times larger and covers multiple scientific domains compared with existing datasets. Our code and data are available at: https://github.com/allenai/scicite.
1 Introduction
Citation intent classification supports automated analysis of scientific literature because citations can signal different relationships to prior work. The paper proposes structural scaffolds to improve classification without relying on predefined external features.
- Citation intents distinguish uses such as applying a method from acknowledging prior work, supporting literature analysis and impact measurement.
- Existing models rely on hand-engineered features and overlook other signals that could improve citation-intent prediction.
- The proposed neural multitask framework predicts citation intent alongside section titles and citation worthiness as structural scaffolds.
- Scaffold-task labels occur naturally during paper writing, enabling large amounts of training data without manual annotation.
- 67.9% F1 on ACL-ARC represents a 13.3% absolute increase over the previous state of the art.
- SciCite is introduced as a citation-intent dataset at least five times larger than existing datasets and spanning multiple scientific domains.
2 Model
The model represents citation contexts with a BiLSTM and attention, then jointly trains citation-intent prediction with structural scaffold tasks. Shared lower-layer parameters transfer information from predicting citation worthiness and section titles.
- Citation representation: Citation contexts are encoded by concatenated GloVe and ELMo vectors, followed by a BiLSTM that produces token representations.The BiLSTM processes each token relative to the full sequence.
- Citation representation: Attention combines the token representations into a single citation vector z used by the task-specific prediction layers.The attention query is a learned parameter vector.
- Structural scaffolds: The two structural scaffolds predict whether a sentence needs a citation and the section title where a citation appears.Their labels occur naturally in scientific writing, avoiding additional manual annotation.
- Multitask formulation: Multitask learning shares lower-layer parameters across the citation-intent task and auxiliary scaffold tasks while giving each task task-specific parameters.Each task uses an MLP and softmax prediction layer.
- Training and inference: Only the primary output is used at inference, while scaffold outputs are used during training to inform the model about document structure.The model is trained jointly across tasks in an end-to-end fashion.
- Training objective: The final objective combines the main loss with weighted auxiliary-task losses, with two scaffold tasks making n=3.The weights λ_i control each task’s sensitivity and can be tuned on a validation set.
3 Data
This section introduces SciCite alongside ACL-ARC, addressing the limited size and domain coverage of existing citation-intent datasets through broader data collection and annotation.
- Dataset overview: SciCite addresses existing dataset limitations by covering multiple scientific domains and exceeding ACL-ARC's size by more than five times.The paper identifies existing datasets as relatively small and commonly domain-specific.
- ACL-ARC citations dataset: ACL-ARC contains 1,941 citation instances from 186 NLP papers, with expert annotations across six intent categories.Its data are divided into stratified training, validation, and test sets.
- Data collection and annotation: SciCite annotations were collected from computer science and medicine papers through crowdsourcing and aggregated with confidence scores.Citation contexts were extracted using science-parse, while workers were screened with domain-expert test questions.
- Data collection and annotation: SciCite labels citation contexts as METHOD, RESULTCOMPARISON, or BACKGROUND, excluding fewer than 1% assigned to OTHER.OTHER was used as a quality-control option for incomplete or too-short contexts and later removed.
- Data collection and annotation: Instances with confidence scores of 0.7 or lower were discarded, and expert re-annotation yielded an 86% agreement rate with crowd workers.Each sentence received an average of 3.74 annotations from 850 workers.
- Scaffold data: The structural scaffolds use naturally occurring labels for citation worthiness and citation section titles, avoiding additional manual annotation for these auxiliary tasks.Citation markers were removed from citation-worthiness examples to prevent artificial task shortcuts, and section titles were normalized with regular-expression mappings.
4 Experiments
Experiments show that structural scaffolds consistently improve citation-intent classification across ACL-ARC and SciCite, with the strongest ACL-ARC result reaching 67.9% F1. Attention analyses and error studies suggest improved focus on intent-relevant words, while ambiguous cases remain sensitive to context and category frequency.
- ACL-ARC results: 63.1 macro F1 results from adding both scaffolds to the BiLSTM-Attn baseline, an 11.3-point improvement that suggests complementary signal.The section-title and citation-worthiness scaffolds individually improve F1 to 56.9 and 56.3, respectively.
- ACL-ARC results: 67.9% F1 is achieved on ACL-ARC by combining ELMo representations with both structural scaffolds, exceeding Jurgens et al. (2018) by 13.3%.The result uses no hand-curated features or additional linguistic resources.
- SciCite results: SciCite exhibits the same pattern: each scaffold improves performance, both scaffolds improve it further, and the best result also adds ELMo representations.SciCite is more than five times larger than ACL-ARC, so its F1 gains are generally smaller while performance numbers are generally higher.
- Per-category results: Per-category results are generally higher for categories with more instances; BACKGROUND performs best on ACL-ARC, while FUTUREWORK performs worst.The authors attribute the weaker FUTUREWORK performance to its having the fewest data points.
- Attention analysis: Attention examples show the scaffold model emphasizing words associated with FUTUREWORK and RESULTCOMPARISON, where the baseline attends to misleading words and misclassifies both examples.The comparison differs only by inclusion of structural scaffolds, supporting their role in the observed attention patterns.
- Error analysis: Some ACL-ARC errors confuse USE with BACKGROUND because limited MOTIVATION examples hinder learning, and certain cases remain ambiguous without additional context.Encoding extended surrounding context caused a large overall performance decline, likely because of added noise.
5 Related Work
Prior citation-intent research moved from fine-grained categorization and rules toward machine-learning models using citation-context patterns, while this work introduces a structural neural scaffold approach.
- Earlier citation-intent systems often used categories too fine-grained for practical automated analysis because some rarely occur in papers.
- Jurgens et al. proposed a six-category citation-intent scheme, while this paper adds a coarser scheme designed for multiple scientific domains.
- Automated classification progressed from rule-based systems to machine-learning models using linguistic patterns and hand-engineered citation-context features.
- This paper proposes a scaffold neural model that incorporates structural information from scientific discourse into citation-intent classification.
6 Conclusions and future work
The paper uses structural scaffolds to inform citation-intent classification and reports strong ACL-ARC performance alongside the broader SciCite dataset. Future work targets auxiliary-task design, related domains, and contextualized representations beyond ELMo.
- 6 Conclusions and future work: The model uses section-title and citation-worthiness prediction as auxiliary scaffolds for citation-intent classification.
- 6 Conclusions and future work: 67.9% F1 on ACL-ARC is a new state-of-the-art, a 13.3% absolute increase over the best previous results.
- 6 Conclusions and future work: SciCite is a large citation-intent dataset designed to fit scientific discourse across multiple scientific domains.
- 6 Conclusions and future work: Future work includes designing auxiliary tasks, adapting the model to similarly structured domains, and replacing ELMo with contextualized representations such as BERT.