Source-linked AI summary
Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction
Patrick Verga, Emma Strubell, Andrew McCallum
TL;DR
Existing biological relation extraction often misses cross-sentence and cross-mention relations, repeats computation for mention pairs, and must work with document-level rather than mention-level annotation. The paper proposes BRANs, which simultaneously score all mention pairs across abstracts and aggregate them for entity-level prediction, with joint entity prediction and weak supervision. BRANs achieve state-of-the-art performance without external knowledge-base resources on Biocreative V CDR, outperform comparable baselines on Biocreative VI ChemProt, and accompany the work with a much larger, higher-quality dataset.
Problem
Biological relation extraction models often miss cross-sentence and cross-mention relations, redundantly process mention pairs, and lack mention-level annotation.
Method
BRANs use efficient self-attention to score all mention pairs across abstracts, aggregate them with multi-instance learning, and jointly predict named entities.
Results
BRANs achieve state-of-the-art performance without external knowledge-base resources on Biocreative V CDR and outperform comparable baselines on Biocreative VI ChemProt.
Takeaways & Limitations
The paper provides a full-abstract extraction model and introduces a larger, higher-quality dataset for biological relation extraction benchmarking.
Takeaways & Limitations
The model predicts only relations in a fixed schema supplied by the data.
Abstract
from arXiv · showhide
Most work in relation extraction forms a prediction by looking at a short span of text within a single sentence containing a single entity pair mention. This approach often does not consider interactions across mentions, requires redundant computation for each mention pair, and ignores relationships expressed across sentence boundaries. These problems are exacerbated by the document- (rather than sentence-) level annotation common in biological text. In response, we propose a model which simultaneously predicts relationships between all mention pairs in a document. We form pairwise predictions over entire paper abstracts using an efficient self-attention encoder. All-pairs mention scores allow us to perform multi-instance learning by aggregating over mentions to form entity pair representations. We further adapt to settings without mention-level annotation by jointly training to predict named entities and adding a corpus of weakly labeled data. In experiments on two Biocreative benchmark datasets, we achieve state of the art performance on the Biocreative V Chemical Disease Relation dataset for models without external KB resources. We also introduce a new dataset an order of magnitude larger than existing human-annotated biological information extraction datasets and more accurate than distantly supervised alternatives.
1 Introduction
Biological relation extraction must handle cross-sentence context, cross-mention interactions, and document-level annotation while avoiding redundant mention-pair computation. BRANs address these challenges with full-abstract prediction and multi-instance and multi-task learning, achieving strong benchmark results and introducing a larger, higher-quality dataset.
- 30% of relations in the Biocreative V CDR dataset are expressed across sentence boundaries.
- Within-sentence mention-pair models cannot resolve relations requiring cross-sentence coreference, such as azathioprine causing fibrosis.
- 5,318 candidate mention pairs in the CDR training set require separate encoding and classification, versus encoding 500 abstracts once.
- BRANs simultaneously score mention pairs across full abstracts and aggregate them with multi-instance learning, while jointly predicting entities to avoid explicit mention-level annotation.
- BRANs achieve state-of-the-art performance without external knowledge-base resources on Biocreative V CDR and outperform comparable baselines on Biocreative VI ChemProt.
- The paper introduces a dataset an order of magnitude larger than existing gold-annotated datasets, covering more entity and relation types with higher accuracy than same-sized distantly supervised datasets.
2 Model
The model combines self-attention and convolutions to encode long contexts, then scores all mention pairs with bi-affine representations and aggregates them for entity-level relations.
- BRANs encode long, multi-sentence contexts while forming pairwise predictions without mention-pair-specific features.
- 2.1 Inputs: The input sequence combines token embeddings with learned positional embeddings, using randomly initialized vectors when positions exceed the embedding limit.
- 2.1 Inputs: Byte pair encoding builds sub-word pieces by iteratively merging frequent cooccurring tokens, supporting rare and unknown biological entities.
- 2.3 Relation Scoring: Head and tail representations form a pairwise relation-affinity tensor through bi-affine scoring, after which mention scores for each entity pair are pooled with LogSumExp.
- 2.2 Transformer: The Transformer contains B blocks that combine multi-head attention and convolutional subcomponents with residual connections and layer normalization.
- 2.2.2 Convolutions: Convolutional layers include a width-5 middle layer to model local n-gram context that often expresses relations concisely.
2.3 Bi-affine Pairwise Scores
The model projects contextually encoded tokens into head and tail representations, then scores every mention-pair relation using learned relation embeddings. It aggregates mention-pair scores for each entity pair with LogSumExp, supporting weakly supervised multi-instance learning.
- Contextually encoded tokens are projected through separate MLPs into head and tail representations for relation arguments.
- The model forms a pairwise affinity tensor that scores each head–relation–tail triple using learned relation embeddings.The learned embedding tensor has dimensions d × L × d, where L is the number of relations.
- Entity-level labels are weakly supervised because entity-level relations lack mention-level annotations and may be applied noisily to text.This setting motivates aggregating predictions across mentions rather than treating each mention pair independently.
- For each entity pair, the model aggregates scores across all combinations of head and tail mentions using LogSumExp.The operation produces per-relation scores for each entity pair from the pairwise affinity tensor.
- LogSumExp smoothly approximates maximum selection while combining multiple predictions and propagating denser gradients than max pooling.
2.5 Named Entity Recognition
The named entity recognition component predicts entity types from Transformer token representations and jointly trains with relation extraction. BIO labels encode entity spans over byte-pair tokenization.
- The NER component feeds Transformer token outputs into a linear classifier that assigns per-token entity labels.The classifier produces per-class scores for each token.
- BIO encoding augments entity-type labels to represent the boundaries of entity spans.
- BIO tags are applied to byte-pair tokens by assigning B- or I- labels to each subword within a mention span.
- NER and relation extraction are trained jointly with maximum-likelihood objectives using shared model features.
- A hyperparameter λ penalizes named-entity updates to trade off the NER and relation-extraction objectives.
3 Results
The experiments evaluate BRANs on CDR, ChemProt, and a new CTD dataset, covering document-level and within-sentence relation extraction. BRANs outperform comparable baselines without linguistic features and achieve effective performance across relation types and entity distances.
- Datasets: The evaluation covers CDR, ChemProt, and a new CTD dataset spanning document-level and within-sentence biological relation extraction.CDR requires cross-sentence reasoning, ChemProt primarily evaluates within-sentence relations, and the CTD dataset uses document-level annotations.
- CDR results: BRANs outperform prior CDR baselines without linguistic features, with further performance gains from adding weakly labeled data.The comparisons exclude external knowledge-base resources, and the added data comes from CTD abstracts without entity annotations.
- CDR results: Ablation experiments remove multi-head attention, wider convolution, or the named-entity recognition objective to assess their contributions.The ablations are reported as CNN-only, no-width-5, and no-NER variants.
- ChemProt results: BRANs outperform comparable ChemProt models while predicting all entity-pair relations simultaneously rather than classifying each mention pair independently.The model shows higher recall and lower precision than the precision-biased baseline models.
- CTD dataset: The new CTD dataset is designed to address the limited size and noisy labels of existing biological relation extraction datasets.Its construction uses curated CTD relations, PubTator entity annotations, and document-level abstract examples.
- CTD results: BRANs predict CTD relation types effectively, with higher performance for relations having more support, and use BPE to outperform Genia tokenization with one-fifth the vocabulary size.BPE also provides a 1.7 F1-point boost for predicting PubTator NER labels.
- CTD results: Accuracy increases as the maximum entity distance considered expands, indicating that the model uses information from distant entity pairs.The distance analysis considers maximum distances of 11, 25, 50, 100, and 500 tokens, with an average sentence length of 22 tokens.
4 Related work
Related work spans general and biological relation extraction, including CNN-, LSTM-, end-to-end, and cross-sentence approaches. BRAN differs by using raw tokens, efficient feed-forward encoding, and simultaneous scoring of all mention pairs.
- General relation extraction research has largely focused on news and web data, while biological work covers protein-protein, drug-drug, chemical-disease, and event relations.
- Previous neural relation extraction models use CNNs, LSTMs, or end-to-end architectures, but these models consider mention pairs separately.
- Cross-sentence methods have used feature-based classifiers over syntactic parses or document-level graph LSTMs.
- BRAN operates over raw tokens without part-of-speech or syntactic parse features, uses a feed-forward architecture, and considers all mention pairs simultaneously.The paper contrasts this with cascading errors from linguistic preprocessing and the efficiency limits of graph LSTMs.
- Bi-affine scoring connects BRAN to prior bilinear models for knowledge-graph link prediction and graph-based dependency parsing.
5 Conclusion
The paper concludes that BRAN simultaneously scores document-level mention pairs and performs well across benchmark and newly introduced biological datasets. Its current relation inventory remains limited to fixed schemas defined by the data.
- BRAN simultaneously scores all mention pairs within a document and performs well on three biological relation extraction datasets.These include two standard benchmark datasets and a new large, high-quality dataset introduced in the work.
- The model outperforms previous state of the art on Biocreative V CDR without additional linguistic resources or mention-level annotation.
- The current model predicts only relations in a fixed schema specified by the data.The authors suggest integrating it with open relation extraction architectures as a possible future direction.
A Implementation Details
The implementation uses a compact Transformer-style configuration with Adam optimization, regularization, and relation-specific threshold tuning. Training includes gradient clipping, noise, dropout, and early stopping.
- The model uses B = 2 transformer block repeats and is trained on a single TitanX GPU.
- Training clips gradients to norm 10, adds gradient noise, and tunes the decision threshold separately for each relation type.
- The implementation applies dropout to embeddings, internal layers, and final states, and samples positive or NULL-class minibatches with probability 0.5.
A.1 Chemical Disease Relations Dataset
For the Chemical Disease Relations experiments, token embeddings are pretrained on a subset of PubMed abstracts, and the data are split for training and early stopping. Results are averaged over multiple runs and splits.
- The merged training and development data are divided into 850 abstracts for training and 150 for early stopping.
- Reported results are averaged over 10 runs using different splits, while all baselines train on both the training and development sets.
- For standard CDR experiments, dropout keep probabilities are 0.85 for word embeddings, 0.95 for internal layers, and 0.35 for the final bilinear projection.
A.2 Chemical Protein Relations Dataset
The dataset setup uses byte-pair encoding and trains only on relation types included in evaluation. The implementation specifies embedding initialization and optimization settings.
- The byte-pair encoding vocabulary is constructed with a budget of 7,500.
- Only relation types in the evaluation set are trained, while remaining annotated types are assigned the Null relation.
- Embeddings use dimension 200 and are randomly initialized.
- The implementation sets ϵ to 1e-8, β1 to .1, β2 to 0.9, and gradient noise η to 1.0.
- Dropout uses keep probability 0.5 for word embeddings and 1.0 for internal layers.
A.3 Full CTD Dataset
The full CTD dataset setup uses relation-specific decision thresholds and a 50,000-unit byte-pair encoding vocabulary. Training and regularization settings are also specified.
- Separate decision boundaries are tuned for each relation type on the development set.
- Each prediction receives the relation type with maximum probability, unless it falls below that relation’s threshold and is set to NULL.
- The model uses embedding dimension 128 with randomly initialized embeddings.
- The byte-pair encoding vocabulary is constructed with a budget of 50,000.
- Models take 1 to 2 days to train.
- The settings use ϵ = 1e-4, β1 = .1, β2 = 0.9, gradient noise η = .1, and dropout keep probabilities of 0.95, 0.95, and 0.5.