Source-linked AI summary
Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts
Di Jin, Peter Szolovits
TL;DR
Unstructured scientific abstracts and sentence classifiers that ignore context make rhetorical information harder to retrieve and classify. The paper presents a hierarchical sequential labeling network that contextualizes sentence representations, and reports 2%-3% higher results than state of the art on two medical-abstract datasets.
Problem
Many scientific abstracts remain unstructured, and sentence labels in sequential classification are associated with surrounding sentences.
Source: §1 · p. 1Method
The hierarchical sequential labeling network encodes individual sentences, contextualizes their representations with a bi-LSTM, and predicts sentence-label probabilities.
Source: §1 · p. 1, §3.3 · p. 4Results
Takeaways & Limitations
The authors conclude that incorporating surrounding-sentence context can improve the overall quality of sequential sentence predictions.
Source: §6 · p. 8Only a small fraction of PubMed abstracts are structured and contain the label information used in this work.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Prevalent models based on artificial neural network (ANN) for sentence classification often classify sentences in isolation without considering the context in which sentences appear. This hampers the traditional sentence classification approaches to the problem of sequential sentence classification, where structured prediction is needed for better overall classification performance. In this work, we present a hierarchical sequential labeling network to make use of the contextual information within surrounding sentences to help classify the current sentence. Our model outperforms the state-of-the-art results by 2%-3% on two benchmarking datasets for sequential sentence classification in medical scientific abstracts.
1 Introduction
Scientific abstracts are often unstructured, making information retrieval difficult, while sentence labels in rhetorical structures depend on their surrounding sentences. The paper introduces a hierarchical sequential labeling network that uses sentence context and reports improved F1 scores on two medical-abstract datasets.
1 Introduction
- The large and growing research literature makes timely, efficient extraction of useful information difficult, motivating tools that help locate information of interest.The introduction cites over 50 million articles published since 1665 and approximately 2.5 million new papers each year.Source: §1 · p. 1
- Because many scientific abstracts remain unstructured, the paper develops machine learning to categorize their sentences into rhetorical sections for information retrieval.The desired categories include rhetorical elements such as purpose, methods, results, and conclusions.Source: §1 · p. 1
- Rhetorical sentence labels are associated with surrounding labels, unlike sentence classification that treats each sentence individually without context.The paper frames this as sequential sentence classification requiring contextual information.Source: §1 · p. 1
- HSLN combines individual sentence encoding, a bi-LSTM context layer, a feed-forward probability predictor, and joint CRF label-sequence optimization.Sentence encoding can use an RNN or CNN.Source: §1 · pp. 1–2
- Adding a context layer to the previous best architecture lets CRF-based inference use adjacent-sentence content and semantics alongside preceding labels.The stated aim is more accurate prediction of the current sentence.Source: §1 · p. 2
- The proposed contributions include removing character-based embeddings without sacrificing performance, using CNN sentence encoding for small datasets, and attention-based pooling for RNN and CNN models.The authors say the CNN option suffers less from over-fitting on small datasets and that attention pooling further improves performance.Source: §1 · p. 2
- Dropout with expectation-linear regularization replaces standard dropout to reduce the performance gap between training and test phases.Source: §1 · p. 2
- At least 2% higher F1 scores on two medical-abstract datasets than previous best models are reported for sequential sentence classification.The comparison is stated as an absolute F1 improvement.Source: §1 · p. 2
2 Related Work
Prior sequential sentence classification methods include feature-engineered statistical models and neural approaches. The related work distinguishes models that classify sentences without sentence-semantic context from approaches that use preceding labels or utterances.
2 Related Work
- Earlier systems used naive Bayes, SVM, HMM, and CRF methods with hand-engineered lexical, semantic, structural, statistical, and sequential features.Source: §2 · p. 2
- Neural short-text classifiers learn features from token or character embeddings, but the cited work says they omit sentence-semantic context needed for sequential classification.CNNs extract local n-gram features, while RNNs are suited to sequence modeling.Source: §2 · p. 2
- Prior neural sequential-classification examples used preceding dialogue utterances or, for biomedical abstracts, CRF optimization based on preceding labels.The biomedical CRF model outperformed state-of-the-art results on PubMed RCT and NICTA-PIBOSO.Source: §2 · p. 2
3 Proposed Model
The proposed model combines word embeddings, sentence encoding, contextualization across an abstract, and sequence-level label optimization.
3.1 Word Embedding Layer
- Word embeddings map each word to a lexical-semantic vector, and can be pretrained on large unlabeled datasets.The embedding matrix stores one vector per vocabulary item.Source: §3.1 · p. 3
3.2 Sentence Encoding Layer
- A bi-RNN or CNN encodes token embeddings into hidden states, which attention-based pooling combines into a sentence representation.Multiple context vectors can focus on different parts of sentence meaning.Source: §3.2 · p. 3
3.3 Context Enriching Layer
- A bi-LSTM contextualizes sentence representations across the abstract, and a feed-forward network converts each contextualized state into label probabilities.Each output probability vector covers the available labels.Source: §3.3 · pp. 3–4
3.4 Label Sequence Optimization Layer
- A CRF models dependencies between successive labels and selects the predicted label sequence by maximizing its score with the Viterbi algorithm.The sequence score combines individual label and transition probabilities.Source: §3.4 · p. 4
4 Experiments
The experiments use two medical-abstract benchmarks and select model configurations using validation data.
4.1 Datasets
- The evaluation uses PubMed RCT and NICTA-PIBOSO, datasets of medical abstracts whose sentences are annotated with rhetorical-structure labels.PubMed RCT is described as the largest dataset for sequential sentence classification; NICTA-PIBOSO targets Evidence-Based Medicine categories.Source: §4.1 · p. 4
4.2 Training Settings
- Test performance is reported at the epoch with the best validation performance, using F1 weighted by the number of true instances per label.Table 1 summarizes dataset statistics, including split counts and label and vocabulary sizes.Source: §4.2 · p. 4, Table 1 · p. 5
- Training uses Adam, a learning rate initially set to 0.003 and decayed by 0.9 each epoch, dropout regularization, and validation-based grid search.Word embeddings were fixed during training; sentence encoders varied by dataset, with CNN window sizes 2, 3, 4, and 5.Source: §4.2 · pp. 4–5
5 Results and Discussion
The model improves on prior published F1 results across the evaluated datasets, while the better sentence encoder varies by dataset. Ablations and error analyses highlight context enrichment’s role and reveal persistent classification difficulties.
5 Results and Discussion
- HSLN-RNN performs better on PubMed 20k and 200k, while HSLN-CNN performs better on NICTA; the RNN advantage grows with PubMed dataset size.The authors attribute CNN’s NICTA performance to fewer parameters and lower overfitting risk on smaller datasets, while the RNN’s greater capacity benefits from more data.Source: §5 · p. 6
- 2%-3% absolute F1 improvement over the previous best published results is reported across the datasets.Table 4 compares the models using support-weighted F1 scores.Source: §5 · p. 5, Table 4 · p. 6
- Removing context enrichment causes the largest F1 drop, bringing both models to prior-state-of-the-art parity; without sequence optimization, they still outperform published methods using it.HSLN-RNN is more robust to component removal; dropout regularization and attention pooling provide more limited improvements.Source: Table 5 · p. 6, §5 · p. 6
- Methods, Results, and Conclusions are predicted well; Background/Objectives are hardest, with about one fifth and around one fourth crossing labels, plus some Methods/Results confusion.Tables 6 and 7 report per-label scores and the confusion matrix for HSLN-RNN on PubMed 20k.Source: §5 · pp. 6–7, Table 6 · p. 7, Table 7 · p. 7
- The learned transition matrix favors Result-to-Conclusion over Result-to-Background transitions (2.48 vs -5.46), and yields the expected Background-to-Conclusion label sequence.The matrix encodes transition probabilities between successive labels.Source: Figure 2 · p. 7, §5 · p. 7
- Some PubMed 20k prediction errors may reflect debatable gold labels, including sentences that could plausibly fit Background rather than Objective, or Method rather than Result.Table 8 presents examples of prediction errors.Source: §5 · p. 7, Table 8 · p. 8
- Combining Wikipedia with PubMed abstracts for embedding training yields the best result; PubMed/MIMIC embeddings without Wikipedia do not guarantee better results.The authors suggest the smaller PubMed-plus-MIMIC corpus may explain the outcome compared with a corpus such as Wikipedia containing at least a billion tokens.Source: §5 · pp. 7–8, Table 9 · p. 8
6 Conclusion
The hierarchical sequential labeling network uses an LSTM to incorporate surrounding-sentence context, improving prediction quality. It outperforms state-of-the-art results by 2%-3% on two medical-abstract datasets.
6 Conclusion
- The model classifies sentences that appear sequentially in text using a hierarchical neural network.Source: §6 · p. 8
- 2%-3%: the model outperforms state-of-the-art results on two datasets for sequential sentence classification in medical abstracts.The conclusion attributes improved overall prediction quality to an LSTM that sequentially processes encoded sentence representations.Source: §6 · p. 8
- An LSTM processes encoded sentence representations sequentially to incorporate surrounding-sentence context when classifying the current sentence.The authors state that this can improve overall prediction quality.Source: §6 · p. 8
7 Future Work
Only a small fraction of PubMed abstracts contain the structured labels used in this work. The authors plan to pre-train on unannotated abstracts or full texts and fine-tune on the target annotated datasets.
7 Future Work
- Only a small fraction of PubMed abstracts contain the structured labels used in this work.The PubMed database contains over 2 million abstracts, with some accompanied by full-text articles.Source: §7 · p. 8
- The authors plan to pre-train on the remaining unannotated abstracts or full texts, then fine-tune on target annotated datasets.They expect this approach to further boost performance.Source: §7 · p. 8