Source-linked AI summary
DocRED: A Large-Scale Document-Level Relation Extraction Dataset
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, Maosong Sun
TL;DR
Document-level relation extraction requires synthesizing information across multiple sentences, beyond the focus of sentence-level methods. The paper introduces DocRED, a large-scale dataset with human-annotated and distantly supervised data, and finds that existing methods perform substantially worse than human performance, leaving substantial room for improvement.
Problem
Sentence-level relation extraction cannot adequately handle relational facts expressed across multiple sentences, motivating document-level methods and suitable large-scale evaluation data.
Method
The paper presents DocRED, a large-scale document-level relation extraction dataset constructed from Wikipedia and Wikidata with human annotations and distantly supervised data.
Results
Existing relation extraction methods decline significantly on DocRED, while human performance is significantly higher than baseline models.
Takeaways & Limitations
DocRED establishes a challenging benchmark for document-level relation extraction and identifies reasoning, inter-sentence information synthesis, and distant supervision as research directions.
Takeaways & Limitations
Document-level relation extraction remains more challenging than sentence-level extraction, with a performance gap between existing models and humans.
Abstract
from arXiv · showhide
Multiple entities in a document generally exhibit complex inter-sentence relations, and cannot be well handled by existing relation extraction (RE) methods that typically focus on extracting intra-sentence relations for single entity pairs. In order to accelerate the research on document-level RE, we introduce DocRED, a new dataset constructed from Wikipedia and Wikidata with three features: (1) DocRED annotates both named entities and relations, and is the largest human-annotated dataset for document-level RE from plain text; (2) DocRED requires reading multiple sentences in a document to extract entities and infer their relations by synthesizing all information of the document; (3) along with the human-annotated data, we also offer large-scale distantly supervised data, which enables DocRED to be adopted for both supervised and weakly supervised scenarios. In order to verify the challenges of document-level RE, we implement recent state-of-the-art methods for RE and conduct a thorough evaluation of these methods on DocRED. Empirical results show that DocRED is challenging for existing RE methods, which indicates that document-level RE remains an open problem and requires further efforts. Based on the detailed analysis on the experiments, we discuss multiple promising directions for future research.
1 Introduction
DocRED addresses the limitations of sentence-level relation extraction by providing a large-scale dataset requiring document-wide reasoning. Experiments show that existing methods perform poorly on this challenging task.
- Motivation: At least 40.7% of relational facts require information from multiple sentences, making sentence-level relation extraction insufficient.The example requires chaining facts across sentences to infer Riddarhuset’s country.
- Dataset gap: Existing document-level RE datasets are limited by small manual annotations, noisy distant supervision, narrow domains, or approach-specific construction.These constraints motivate a general-purpose, large-scale benchmark.
- Contribution: DocRED is a large-scale human-annotated document-level RE dataset constructed from Wikipedia and Wikidata.It annotates entities, relations, and document-level information needed for relation extraction.
- Contribution: DocRED contains 132,375 entities and 56,354 relational facts across 5,053 Wikipedia documents.These statistics make it the largest human-annotated document-level RE dataset described here.
- Contribution: DocRED also provides large-scale distantly supervised data for weakly supervised RE research.The dataset therefore supports both human-annotated and weakly supervised settings.
- Evaluation: Existing RE methods decline significantly on DocRED, indicating that document-level RE is more challenging than sentence-level RE and remains open.The authors identify multiple promising directions from detailed experimental analysis.
2 Data Collection
DocRED combines Wikipedia and Wikidata with staged human annotation, model-assisted recommendations, and distant supervision. Its collection process targets entity, coreference, relation, and evidence annotations at document scale.
- Collection pipeline: DocRED’s human-annotated data is collected through distant-supervision generation, entity and coreference annotation, entity linking, and relation-evidence labeling.The four stages progressively construct the document-level annotations.
- Quality control: Annotation uses three iterative passes: recommendation generation, manual correction and supplementation, and review for accuracy and consistency.Training procedures and experienced annotators support the final review pass.
- Data sources: Wikipedia introductory sections are paired with Wikidata to provide strong alignment between text and knowledge-base information.The corpus uses introductory sections because they are usually high-quality and contain key information.
- Collection pipeline: Distant supervision first identifies and links named entities, merges mentions with identical knowledge-base IDs, and labels relations between merged entity pairs.This generates recommendations for subsequent human annotation.
- Entity annotation: Human annotators review, correct, and supplement named-entity recommendations, then merge mentions referring to the same entities.This stage provides entity and coreference information for relation extraction.
- Relation annotation: Because potential entity pairs grow quadratically and most pairs lack relations, annotators receive recommendations from entity linking and RE models.The process averages 19.9 entity-linking recommendations and 7.8 RE-model recommendations per document.
- Distant supervision: The additional distantly supervised corpus contains 101,873 documents after excluding the 5,053 human-annotated documents.Named entity mentions are reidentified to align entity distributions across the two datasets.
3 Data Analysis
DocRED spans diverse entity and relation types while requiring substantial cross-sentence reasoning. Its statistics show that many relations depend on multiple supporting sentences and document-wide synthesis.
- Data Size: DocRED is larger than representative sentence-level and document-level relation extraction datasets across multiple data-size measures.The comparison appears in Table 1.
- Named Entity Types: DocRED covers diverse entity types, including people, locations, organizations, time, numbers, and miscellaneous entities.The miscellaneous category includes events, artistic works, and laws.
- Relation Types: DocRED includes 96 relation types spanning science, art, time, and personal-life categories rather than a single domain.The relation types are organized in a hierarchy and taxonomy.
- Inter-Sentence Relation Instances: 40.7% of relational facts can only be extracted from multiple sentences, making reading, synthesis, and reasoning essential for document-level RE.Each relation instance has 1.6 supporting sentences on average, and 46.4% have more than one.
4 Benchmark Settings
DocRED defines supervised and weakly supervised benchmark settings evaluated on human-annotated data. The settings expose reasoning, computational, and distant-supervision labeling challenges.
- Benchmark Settings: DocRED provides supervised and weakly supervised benchmark settings, both evaluated on the human-annotated dataset.The supervised setting uses human-annotated training data, while the weakly supervised setting replaces training with distantly supervised data.
- Supervised Setting: About 61.1% of relation instances depend on complex reasoning beyond pattern recognition.These instances require reasoning over global and complex document information.
- Supervised Setting: Document-level RE faces high computational cost because potential entity pairs grow quadratically with entity number, averaging 19.5 entities per document.Context-aware systems with quadratic or higher complexity are described as inefficient for this setting.
- Weakly Supervised Setting: Distantly supervised document-level data has substantial wrong-labeling noise, especially for inter-sentence instances.The reported incorrect-label rates are 41.4% for intra-sentence instances and 61.8% for inter-sentence instances.
5 Experiments
Experiments evaluate adapted RE models on DocRED under supervised and weakly supervised settings, alongside human performance, evidence prediction, and feature ablations. Results show substantial difficulty for existing systems and motivate richer document-level reasoning.
- Evaluation: Evaluation covers supervised and weakly supervised settings, human performance, supporting-evidence types, and feature contributions.The study also reports F1 and AUC variants that exclude relational facts overlapping between training and development or test sets.
- Experimental Setup: Four adapted RE systems use CNN, LSTM, BiLSTM, or Context-Aware architectures to encode documents and predict relations for entity pairs.The CNN, LSTM, and BiLSTM models differ in their document encoders; Context-Aware leverages contextual relations.
- Experimental Setup: Each word representation combines GloVe, entity type, and coreference embeddings, while entity representations average mention representations.Entity mentions are represented by averaging encoder hidden states, and entities average their mention representations.
- Experimental Setup: Relation prediction treats each entity pair as a multi-label classification problem and computes each relation type’s probability with a bilinear function.Relative-distance embeddings are concatenated with entity representations before relation scoring.
- Results: Models using richer contextual information generally perform better, but current models do not effectively exploit inter-relation information.LSTM and BiLSTM outperform CNN, while Context-Aware remains competitive without significantly outperforming other neural models.
- Results: Human performance is competitive on both relation extraction and joint relation-plus-evidence prediction, while RE models perform significantly worse.This indicates high human ceiling performance and substantial room for improvement on document-level RE.
- Results: Entity types, coreference information, and relative entity distance all contribute to BiLSTM performance, with entity types contributing most.Coreference and relative distance support synthesis across multiple named entity mentions.
- Supporting Evidence Prediction: The paper introduces supporting-evidence prediction as a joint task because evidence identification and relation reasoning are naturally dual tasks with potential mutual enhancement.It evaluates heuristic and neural evidence predictors alongside relation prediction.
6 Related Work
Prior RE datasets either provide limited relation coverage, rely on distant supervision, or target narrow domains. DocRED addresses these limitations with crowd-worker annotation, richer information, and domain breadth for general-purpose document-level RE.
- Earlier human-annotated RE datasets contain relatively limited relation types and instances.
- Distantly supervised datasets can suffer from wrong labeling, making their evaluations less reliable.
- DocRED uses crowd-worker annotation with rich information and no specific-domain restriction, supporting general-purpose document-level RE.
7 Conclusion
DocRED is presented as a large-scale document-level RE dataset designed to advance systems beyond sentence-level extraction. Human performance substantially exceeds RE baselines, leaving substantial room for improvement.
- DocRED combines large scale, multi-sentence reading and reasoning, and distantly supervised data for document-level RE.
- The dataset supports both supervised and weakly supervised research through human-annotated and distantly supervised data.
- Human performance is significantly higher than RE baseline models, indicating ample opportunity for future improvement.
A.1 Experimental Details
The experiments use standardized optimization settings and represent document-level entity distances with trainable distance embeddings.
- All baselines are optimized with Adam using a learning rate of 0.001.
- The experiments use β1 = 0.9 and β1 = 0.999 as reported optimization settings.
- Entity distances are divided into bins {1, 2, .., 2k}, with each bin assigned a trainable distance embedding.
- Table 8 reports the remaining experimental hyper-parameter settings.
A.2 Types of Named Entities
DocRED adapts named-entity types to better serve its document-level RE setting. The taxonomy covers six categories, documented in Table 9.
- DocRED adapts the named-entity types of Tjong Kim Sang and De Meulder (2003).
- The taxonomy includes Person (PER), Organization (ORG), Location (LOC), Time (TIME), Number (NUM), and other types (MISC).
- Table 9 presents the named-entity types in DocRED and their covered contents.
A.3 List of Relations
DocRED provides a relation list organized across Tables 10 and 11, with Wikidata identifiers, relation names, and descriptions.
- DocRED’s relation list includes Wikidata IDs, relation names, and descriptions.
- Table 10 presents the first part of the DocRED relation list.
- Table 11 presents the second part of the DocRED relation list.