Source-linked AI summary
SciREX: A Challenge Dataset for Document-Level Information Extraction
Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, Iz Beltagy
TL;DR
Most IE methods focus on sentences or short paragraphs, leaving full-document extraction underdeveloped despite relations spanning sections. SciREX introduces a multi-task scientific-article dataset and an end-to-end neural baseline; document-level modeling improves recall, but salient entities and document-level relations remain challenging.
Problem
IE has largely focused on sentence- or paragraph-level relations, while extracting entities and relations across full scientific documents remains underdeveloped and difficult to annotate.
Method
SciREX combines automatic and expert annotations using scientific knowledge resources and develops an end-to-end neural model for document-level IE.
Results
Document-level modeling significantly improves recall over paragraph-level approaches, while salient-entity identification and document-level relation discovery remain challenging.
Takeaways & Limitations
SciREX provides a comprehensive testing ground for document-level IE models covering entities, coreference, saliency, and N-ary relations.
Takeaways & Limitations
Document-level IE still poses challenges involving long-document modeling, cross-document coreference aggregation, saliency identification, and N-ary relation extraction.
Abstract
from arXiv · showhide
Extracting information from full documents is an important problem in many domains, but most previous work focus on identifying relationships within a sentence or a paragraph. It is challenging to create a large-scale information extraction (IE) dataset at the document level since it requires an understanding of the whole document to annotate entities and their document-level relationships that usually span beyond sentences or even sections. In this paper, we introduce SciREX, a document level IE dataset that encompasses multiple IE tasks, including salient entity identification and document level $N$-ary relation identification from scientific articles. We annotate our dataset by integrating automatic and human annotations, leveraging existing scientific knowledge resources. We develop a neural model as a strong baseline that extends previous state-of-the-art IE models to document-level IE. Analyzing the model performance shows a significant gap between human performance and current baselines, inviting the community to use our dataset as a challenge to develop document-level IE models. Our data and code are publicly available at https://github.com/allenai/SciREX
1 Introduction
SciREX addresses the lack of comprehensive document-level IE datasets by annotating scientific articles for entities, saliency, coreference, and cross-section N-ary relations, with an end-to-end neural baseline.
- Prior IE datasets and methods mainly target sentence-level relations, short paragraphs, or relations among ground-truth entities in long documents.
- Document-level annotation is difficult because entities and relations can span sentences or sections and require domain expertise and substantial effort.
- SciREX identifies Dataset, Metric, Task, and Method entities, their document-level relations, and which entities are salient to article results.
- Automatic high-recall mention detection followed by expert correction using article text and an external knowledge base reduces annotation time.
- The neural baseline jointly identifies mentions, saliency, coreference links, salient entities, and document-level relations, while salient-entity identification remains the hardest subtask.
- SciREX evaluates comprehensive IE, including long-document N-ary relations, and positions the dataset as a challenge for better document-level models.
2 Related Work
Related work largely addresses IE in sentences, abstracts, or short paragraphs, whereas SciREX combines extraction, coreference, saliency, and whole-document N-ary relation identification from raw text.
- Scientific IE datasets commonly extract entities and relations from sentences or abstracts, including SCIERC’s annotated scientific abstracts with binary relations.
- Document-level relation datasets have used distant supervision but typically formulate relation extraction as binary classification over ground-truth entity triplets.
- SciREX instead performs comprehensive IE from raw documents, requiring entity extraction, coreference resolution, saliency detection, and relation extraction.
- General-domain IE remains predominantly sentence-level, while DocRED studies cross-sentence relations in Wikipedia paragraphs shorter than SciREX documents.
- DYGIE++ jointly extracts entities and binary relations but is designed for short paragraphs and does not scale well to long documents.
- SciREX extracts 4-ary relations between salient entity clusters, requiring a more global document view than binary mention-pair extraction.
3 Document-Level IE
SciREX defines document-level IE over scientific articles and constructs its dataset by combining knowledge-base supervision, noisy automatic labeling, and expert correction; most higher-order relations cross sentences or sections.
- 3.1 Task Definition: The task extracts Dataset, Metric, Task, and Method entities, their mentions and coreferences, saliency, and binary, 3-ary, and 4-ary relations.
- 3.1 Task Definition: A 4-ary relation cannot generally be decomposed into binary relations because datasets may have multiple tasks, each with its own metric.
- 3.2 Dataset Construction: The annotation pipeline uses automatic labeling and expert correction to reduce the effort of annotating documents averaging about 5K tokens.
- 3.2 Dataset Construction: Papers with Code supplies distant supervision for result tuples, but it does not specify where tuple entities are mentioned in the article.
- 3.2 Dataset Construction: A SCIERC-trained BERT+CRF model proposes noisy mention spans, while Jaccard matching to knowledge-base entities favors high recall over precision.
- 3.2 Dataset Construction: Human annotators correct span boundaries and types and add missed spans after receiving a deliberate list of knowledge-base entities.
- 3.3 Dataset and Annotation Statistics: 83% of automatic mention labels were correct, 15% were added, 2% changed type, and a negligible percentage was deleted.
- 3.3 Dataset and Annotation Statistics: 1.34 sec per entity with automatic labeling versus 2.48 sec without it yielded a 1.85x speedup, while including non-salient mentions improved salient-mention recall by 24%.
4 Model
The model performs document-level IE by jointly representing sections and tokens, identifying and classifying mentions, clustering coreferent mentions into entities, and classifying entity tuples as relations.
- Document Representation: SciBERT encodes sections separately, while a BiLSTM over concatenated section embeddings captures cross-section dependencies.Sections exceeding 512 tokens are divided into subsections before encoding.
- Mention Identification and Classification: A BIOUL CRF identifies mention spans and their types from BERT-BiLSTM token embeddings.
- Mention Representation: Mention embeddings concatenate boundary-token embeddings with an attention-weighted average of span embeddings for later saliency and relation classification.Additional features include relative document position, marker-word indicators, and mention type.
- Saliency and Coreference: Each mention receives a saliency score, and clusters containing at least one salient mention are retained as salient entities.Saliency is treated as an entity property, although the mention score informs cluster identification.
- Saliency and Coreference: A separate SciBERT surface-form model scores mention pairs, which are grouped with agglomerative clustering and silhouette-based cluster selection.The integrated span-embedding alternative performed worse than the separate surface-form model.
- Relation Extraction: For candidate binary and 4-tuples of entity clusters, section embeddings are aggregated into a document representation that an FFN converts into relation probability.Cluster representations use max-pooled mention embeddings per section, with a learned bias when a cluster is absent; the document representation averages section embeddings.
5 Evaluation
The evaluation compares SciREX with existing models on applicable subtasks and analyzes performance component-wise and end-to-end. Results show that document-level saliency identification and error accumulation are central challenges, while document-level modeling improves cross-section relation recall.
- 5.2 Evaluation: SciREX is evaluated against DYGIE++ and DocTAET on applicable subtasks because no existing model performs the complete end-to-end task.The evaluation also includes SCIERC comparisons where the models apply.
- 5.2.1 Evaluation on SCIREX: Document-level modeling generally outperforms DYGIE++ on binary relations by recovering cross-section relations that section-level training misses.55% of binary relations occur across sentence boundaries, while 80% of DYGIE++ training examples contain no sentence-level binary relations.
- 5.2.1 Evaluation on SCIREX: Our relation model achieves comparable positive-class F1 to DocTAET when evaluated on document-level relation classification.DocTAET receives gold salient entity clusters, making the comparison correspond to the relation-classification stage.
- 5.2.2 Evaluation on SCIERC: On SCIERC, the model is slightly worse for mention identification and significantly worse for coreference clustering than DYGIE++.The mention-identification gap is attributed to overlapping entities that the CRF-based model cannot handle.
- 5.3 Component-wise Evaluation: Salient-mention identification is the weakest component because it requires aggregating information across the document.Coreference performs relatively well on SciREX, likely because many coreferences can be resolved from surface forms such as abbreviations.
- 5.4 End-to-End Evaluation: More than 10x improvement in 4-ary relation performance with gold salient clustering identifies salient-cluster prediction as the end-to-end bottleneck.End-to-end relation performance drops because errors accumulate across earlier pipeline stages.
- 5.4 End-to-End Evaluation: The model overweights mention frequency when predicting saliency, correctly classifying clusters averaging 15 mentions versus six for misclassified clusters.The authors suggest replacing simple filtering with a trained model that aggregates evidence from multiple mentions and context.
- 5.4 End-to-End Evaluation: Overall, entity saliency remains challenging and crucial for end-to-end document-level IE, while incremental component improvements can benefit the full model.The results emphasize the need for better models that perform careful document-level analysis.
6 Conclusion
SCIREX provides a challenging full-document IE dataset and an end-to-end neural baseline covering the necessary subtasks. Document-level modeling improves recall over paragraph-level approaches, while long-document IE remains technically difficult.
- SCIREX is a comprehensive dataset for information extraction on full documents.
- The baseline is the first reported neural document-level IE model to perform all necessary subtasks end to end.
- Document-level modeling significantly improves recall compared with existing paragraph-level approaches.
- Long-document IE requires handling transformer memory limits, cross-document coreference aggregation, salient-entity identification, and document-level N-ary relation extraction.
- Model analysis emphasizes the need for better document-level models and positions SCIREX as a testing ground for this challenging task.
A Model Details
The model is trained on annotated document splits using SciBERT, BiLSTM representations, feedforward networks, and attention, with task-specific optimization and coreference scoring components.
- The base document representation combines SciBERT-base with a BiLSTM using a 128-dimensional hidden state.A dropout rate of 0.2 is applied after the BiLSTM embeddings.
- Feedforward networks use two 128-dimensional hidden layers with GELU activation and dropout between layers.
- Training uses Adam for 30 epochs with separate learning rates for non-BERT and BERT weights, plus early stopping based on validation relation-extraction F1.The learning rates are 1e-3 and 2e-5, with patience 7.
- The models are implemented in AllenNLP and trained on 48Gb Quadro RTX 8000 GPUs.The multitask model takes approximately 3 hours to train.
- The BERT coreference model encodes mention pairs with special tokens and computes a coreference score from the [CLS] representation.
B Annotation Guidelines
The annotation guidelines decompose composite method entities before identifying their mentions, while the current model maps subentity mentions back to the composite method entity.
- Annotators break complex Method entities into simpler entities before searching for mentions.
- For example, DLDL+VGG-Face is decomposed into DLDL and VGG-Face.
- The current model treats mentions of subentities as mentions of the corresponding composite Method entity.
- Explicitly extracting relations between subentities is left for future work.