Source-linked AI summary
Pretrained Language Models for Sequential Sentence Classification
Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, Daniel S. Weld
TL;DR
The paper addresses sentence-role classification when sentence meaning depends on document context. It jointly encodes all sentences with BERT and uses sentence-boundary representations for prediction, achieving state-of-the-art results across scientific-text tasks while facing a long-sequence limitation.
Problem
Sequential Sentence Classification requires assigning sentence roles while accounting for contextual dependencies across the document.
Method
The approach jointly represents all document sentences in one BERT input and classifies each sentence from its [SEP] representation.
Results
The approach achieves state-of-the-art results across multiple datasets for scientific abstract sentence classification and extractive summarization.
Takeaways & Limitations
Jointly encoding sentences improves performance across multiple scientific-domain datasets and tasks without additional complex architectures.
Takeaways & Limitations
Representing all sentences in one sequence is limited by BERT's 512-wordpiece capacity, and split boundaries can lose context.
Abstract
from arXiv · showhide
As a step toward better document-level understanding, we explore classification of a sequence of sentences into their corresponding categories, a task that requires understanding sentences in context of the document. Recent successful models for this task have used hierarchical models to contextualize sentence representations, and Conditional Random Fields (CRFs) to incorporate dependencies between subsequent labels. In this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF. Specifically, we construct a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences. Our approach achieves state-of-the-art results on four datasets, including a new dataset of structured scientific abstracts.
1 Introduction
The paper studies Sequential Sentence Classification, where sentence roles depend on document context. It proposes using BERT to jointly encode sentences and reports state-of-the-art results across multiple scientific-text datasets and tasks.
- The paper explores Sequential Sentence Classification, assigning roles or functions to sentences whose meanings depend on neighboring context.
- BERT's pretrained context modeling motivates replacing hierarchical sentence encoders and CRFs with direct document-level contextualization.The proposed representation lets BERT's Transformer layers use contextualized information from words across all sentences.
- The model applies SSC to scientific abstract sentence classification and extractive summarization of scientific documents.
- The BERT-based approach jointly encodes all sentences, introduces CSABSTRUCT, and achieves state-of-the-art results on multiple datasets for both tasks.
2 Model
The model concatenates all document sentences into one BERT input and classifies each sentence from its [SEP] representation. This enables direct cross-sentence attention but is constrained by BERT's sequence-length limit.
- Each sentence is concatenated with a [SEP] token, and a feedforward classifier predicts its label from that token's BERT representation.The input begins with [CLS], while [SEP] encodings rather than [CLS] are used for sentence classification.
- Joint self-attention lets BERT use contextual information from words in all sentences while retaining pretrained weights.This contrasts with hierarchical models that encode and contextualize sentences in two consecutive steps.
- Handling long sequences: BERT's 512-wordpiece limit constrains documents represented as one sequence.The authors split long documents recursively, but sentences near split boundaries may lose neighboring context.
- The approach avoids adding a CRF or another contextualizing layer because such additions were not empirically helpful.The authors suggest BERT's self-attention already captures necessary document context.
3 Tasks and Datasets
The paper evaluates sentence classification on three scientific-abstract datasets and treats extractive summarization as sentence classification. It introduces CSABSTRUCT, a manually annotated and stylistically varied computer-science abstract dataset.
- Scientific abstract sentence classification: Scientific abstract sentence classification assigns rhetorical roles such as introduction, method, and results across three datasets.The datasets include PUBMED-RCT, CSABSTRUCT, and NICTA.
- Scientific abstract sentence classification: CSABSTRUCT contains 2,189 manually annotated computer-science abstracts with sentences labeled by rhetorical role.
- Extractive summarization of scientific documents: Extractive summarization can be framed as SSC by labeling each sentence as a good or bad summary sentence using contextual information.Training uses CSPUBSUMEXT, while CSPUBSUM provides a separate test set; the model directly predicts ROUGE scores with mean squared error.
- CSABSTRUCT construction details: Unlike structurally templated biomedical abstracts, CSABSTRUCT contains free-form computer-science abstracts with greater writing-style variety.
- CSABSTRUCT construction details: CSABSTRUCT annotations use five rhetorical categories and are aggregated from multiple workers with confidence-based dataset splitting.
4 Experiments
Experiments show that joint BERT sentence encoding outperforms strong baselines across abstract classification and scientific summarization, while attention analyses indicate that contextual sentence interactions support improvements.
- Results: State-of-the-art results on all three abstract classification datasets outperform Jin and Szolovits (2018) and both BERT-based baselines.The performance gap is larger on the smaller CSABSTRUCT and NICTA datasets than on PUBMED-RCT.
- Results: Our approach outperforms BERT+TRANSFORMER on CSPUBSUM, and adding ABSTRACT-ROUGE slightly surpasses Collins et al. (2017)’s complex ensemble.The proposed model remains a single model with only one added feature.
- Analysis: 34/134 examples where the model beats BERT+Transformer require context for correct classification.One example is ambiguous in isolation but is clearly METHOD when interpreted with the preceding sentence.
- Analysis: After finetuning, attention forms sentence-level diagonal blocks, except for mutual attention between sentences 2 and 3.This pattern lets sentence 3 incorporate information from sentence 2 when determining its label.
5 Related Work
Prior scientific sentence-classification systems commonly use hierarchical encoders followed by CRFs, while related summarization systems often use hierarchical sequence-to-sequence models evaluated on newswire data.
- Sequential Sentence Classification: Scientific Sequential Sentence Classification work commonly encodes sentences hierarchically with LSTMs and applies a CRF over the sentence representations.
- Summarization: Related summarization work also uses hierarchical models, is often trained sequence-to-sequence, and commonly evaluates on CNN/Daily Mail newswire data.
6 Conclusion and Future Work
The paper concludes that pretrained BERT can support SSC without complex additional architectures, and that jointly encoding sentences improves performance across scientific datasets and tasks.
- Conclusion: BERT enables SSC without additional complex architectures.
- Conclusion and Future Work: Jointly encoding sentences improves performance across multiple scientific-domain datasets and tasks.Future work will explore better methods for encoding long sequences with pretrained language models.
A Additional analysis
Figures 3 and 4 compare BERT attention before and after fine-tuning in layers 8 and 12. Fine-tuning produces attention patterns that differ across sentences, likely reflecting their distinct roles and contextual needs.
- A Additional analysis: After fine-tuning, BERT attends to sentences differently, likely according to each sentence’s role and required contextual information.The comparison covers attention in layer 8 and the final layer, layer 12.
- A Additional analysis: Before fine-tuning, attention to [SEP] tokens and periods is almost identical between sentences.