Source-linked AI summary
Fine-tune Bert for DocRED with Two-step Process
Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, William Wang
TL;DR
DocRED requires predicting relations for entity pairs across documents, but prior BiLSTM baselines may not adequately model complex entity interactions. The paper fine-tunes BERT for document encoding and uses two steps: detect whether a relation exists, then classify its specific relation. BERT increases F1 by about 2%, and two-step training improves performance further; accuracy for the second step is above 90%.
Problem
DocRED requires predicting relations for entity pairs across documents, but prior BiLSTM baselines may not adequately model complex entity interactions.
Method
The paper fine-tunes BERT for document encoding and uses two steps: detect whether a relation exists, then classify its specific relation.
Results
BERT increases F1 by about 2%, and two-step training improves performance further; accuracy for the second step is above 90%.
Takeaways & Limitations
Relation-existence detection is the main bottleneck, while current models still fail to capture complex interactions among entities.
Abstract
from arXiv · showhide
Modelling relations between multiple entities has attracted increasing attention recently, and a new dataset called DocRED has been collected in order to accelerate the research on the document-level relation extraction. Current baselines for this task uses BiLSTM to encode the whole document and are trained from scratch. We argue that such simple baselines are not strong enough to model to complex interaction between entities. In this paper, we further apply a pre-trained language model (BERT) to provide a stronger baseline for this task. We also find that solving this task in phases can further improve the performance. The first step is to predict whether or not two entities have a relation, the second step is to predict the specific relation.
1 Introduction
DocRED requires predicting relations for entity pairs across documents, making complex entity interactions central. The paper argues that BERT and a two-step process provide stronger modeling than earlier BiLSTM-based baselines.
- DocRED requires predicting a relation for every entity pair in a document, requiring models to capture complex interactions between entities.
- Earlier DocRED baselines encoded the whole document with BiLSTM, averaged entity word embeddings, and applied a BiLinear relation classifier.
- BERT is proposed as a stronger encoder because pre-training captures important language features and may capture common-sense knowledge.
- Fine-tuning BERT increases F1 by about 2%, while two-step training further improves performance.
- The two steps first detect whether an entity pair has any relation, then classify its specific relation.
2 Model
The model encodes documents with BERT, extracts entity embeddings, and predicts pairwise relations with a BiLinear layer. To address abundant N/A pairs, training separates relation detection from specific relation classification.
- 2.1 BERT Model: BERT encodes the document, entity embeddings are extracted, and a BiLinear layer predicts each entity pair's relation.
- 2.1 BERT Model: The model uses BERT-base, with a whole-model structure illustrated in Figure 1.
- 2.2 Two-step Training Process: DocRED has severe label imbalance because most entity pairs have no relation and belong to N/A.
- 2.2 Two-step Training Process: Step one performs binary relation detection using all annotated data, with sub-sampling to balance relational and N/A pairs within batches.
- 2.2 Two-step Training Process: Step two predicts the specific relation using the same BERT structure but training only on entity pairs with annotated relations.
- 2.2 Two-step Training Process: At test time, the first model detects whether a relation exists, and the second model classifies it when present.
3 Experiments
Experiments use annotated DocRED data, compare BERT with CNN, LSTM, BiLSTM, and Context-Aware baselines, and examine both two-step training and document-level interaction modeling. BERT improves F1 by 2%, while sentence-by-sentence encoding performs similarly to BiLSTM.
- 3.1 DocRED Dataset: DocRED covers diverse topics and entity and relation types, requiring pattern, logical, coreference, and common-sense reasoning.Experiments use only the annotated training data.
- 3.2 Implementation Details: The BERT implementation uses BERT-base, a 10^-5 learning rate, 768-dimensional embeddings, and a 128-dimensional projected space.
- 3.3 Results: The evaluated baselines include CNN, LSTM, BiLSTM, and Context-Aware models, differing primarily in document encoding.
- 3.3 Results: BERT improves F1 by 2% over the compared baselines, and two-step training improves performance further.The main comparison reports F1 on the Dev and Test sets.
- 3.3 Results: The second step exceeds 90% accuracy, indicating that relation existence prediction is the bottleneck.
- 3.4 Complex interaction modeling: A sentence-by-sentence SentModel achieves performance similar to BiLSTM, suggesting current models use local entity information rather than complex cross-sentence interactions.
4 Conclusion & Discussion
The paper finds that BERT and two-step training improve document-level relation extraction, but identifies relation-existence detection and complex entity interaction modeling as central challenges.
- BERT significantly improves document-level relation extraction, and two-step training improves performance further.
- The dataset's main difficulty is determining whether an entity pair has any relation; identifying the specific relation is less challenging.
- Current models fail to capture complex interactions between entities and instead rely on local information around each entity.