Source-linked AI summary
AutoRE: Document-Level Relation Extraction with Large Language Models
Lilong Xue, Dan Zhang, Yuxiao Dong, Jie Tang
TL;DR
DocRE requires extracting multiple relations and triplet facts across documents, while candidate-relation prompts are difficult to use with large relation sets such as Re-DocRED's 96 types. AutoRE uses the Relation-Head-Facts paradigm and an instruction-tuning dataset based on Re-DocRED to fine-tune Mistral-7B with QLoRA. On the dev and test sets, Vicuna-7B surpassed TAG by 10.03% and 9.03%, respectively.
Problem
DocRE requires extracting multiple relations and triplet facts across documents, while candidate-relation prompts are difficult to use with large relation sets such as Re-DocRED's 96 types.
Source: §1 · pp. 1–2Method
AutoRE uses the Relation-Head-Facts paradigm and an instruction-tuning dataset based on Re-DocRED to fine-tune Mistral-7B with QLoRA.
Source: §1 · p. 2Results
On the dev and test sets, Vicuna-7B surpassed TAG by 10.03% and 9.03%, respectively.
Source: §4.3 · p. 7Takeaways & Limitations
The authors present AutoRE as an extensible DocRE framework, with separate modules that can be targeted for improvement without affecting other subtasks.
Source: §4.2 · p. 6AutoRE is not equipped to handle unseen relations, which the authors attribute primarily to the limited scope of its training data.
Source: Limitations · p. 7
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated exceptional abilities in comprehending and generating text, motivating numerous researchers to utilize them for Information Extraction (IE) purposes, including Relation Extraction (RE). Nonetheless, most existing methods are predominantly designed for Sentence-level Relation Extraction (SentRE) tasks, which typically encompass a restricted set of relations and triplet facts within a single sentence. Furthermore, certain approaches resort to treating relations as candidate choices integrated into prompt templates, leading to inefficient processing and suboptimal performance when tackling Document-Level Relation Extraction (DocRE) tasks, which entail handling multiple relations and triplet facts distributed across a given document, posing distinct challenges. To overcome these limitations, we introduce AutoRE, an end-to-end DocRE model that adopts a novel RE extraction paradigm named RHF (Relation-Head-Facts). Unlike existing approaches, AutoRE does not rely on the assumption of known relation options, making it more reflective of real-world scenarios. Additionally, we have developed an easily extensible RE framework using a Parameters Efficient Fine Tuning (PEFT) algorithm (QLoRA). Our experiments on the RE-DocRED dataset showcase AutoRE's best performance, achieving state-of-the-art results, surpassing TAG by 10.03\% and 9.03\% respectively on the dev and test set. The code is available at https://github.com/THUDM/AutoRE and the demonstration video is provided at https://www.youtube.com/watch?v=IhKRsZUAxKk.
1 Introduction
The paper presents AutoRE, a DocRE model built around the Relation-Head-Facts (RHF) paradigm, and reports state-of-the-art results across multiple pretrained language models. It motivates this approach by the difficulty of handling many relations and facts in documents with existing methods.
Inefficacy in Multi-Relations
- Candidate-relation prompting becomes impractical when relation sets exceed 100 types, while Re-DocRED contains 96 relation types.A single Re-DocRED instance can include up to 27 relations and 142 triplet facts.Source: §1 · p. 2
Limitations of Current RE Paradigms
- Existing generative approaches either produce triplets directly or identify relations before generating facts, while earlier approaches identify heads first; the paper says these approaches fall short on document-level tasks with many relations and facts.The paper tests a pipeline approach that extracts relations before identifying subjects, followed by triplet extraction.Source: §1 · p. 2
Various RE Paradigms
- RHF orders extraction as relations, then subjects, then triplet facts, and the paper reports that this pipeline improves the capacity to find facts efficiently and accurately.The authors also state that they redefined the 96 relation descriptions and developed simplified extraction templates.Source: §1 · p. 2
Efficient DocRE Model
- AutoRE achieved state-of-the-art results on Re-DocRED, with the framework also reaching state-of-the-art across multiple pretrained language models.The model was built by fine-tuning Mistral-7B with QLoRA on an instruction-tuning dataset based on Re-DocRED.Source: §1 · p. 2
Easy Enhancement of Capabilities
- AutoRE uses three specialized QLoRA modules for relation extraction, head-entity identification, and triplet extraction, with a minimal computational increase and no interference between subtasks.Source: §1 · p. 2
2 Related work
Prior LLM-based RE work includes prompt-driven and pipeline or joint paradigms, but the paper describes performance gaps on DocRE and difficulties handling many relations and triplet facts. It frames these limitations as motivation for further fine-tuning and a paradigm shift.
LLMs for DocRE
- LLM performance on DocRE remains below state-of-the-art, and most models are tested on sentence-level RE rather than document-level tasks.The authors report low performance in their tests of ChatGPT, Mistral-7B, Vicuna-7B, and ChatGLM3-6B.Source: §2 · pp. 2–3
RE Prompt Template
- Prompt templates that list relations as alternatives become impractical for DocRE datasets such as Re-DocRED, with 96 relations.The cited work notes that detailed instructions may not fit datasets with long texts or many targets.Source: §2 · p. 3
RE Paradigms
- Pipeline methods identify relations before facts or heads before relation-tail pairs, while joint methods output triplets directly; both face challenges on DocRE samples with many relations and facts.Conventional pipeline processing can also be time-consuming when many entities lack interrelations.Source: §2 · p. 3
Summary
- The paper concludes that current LLM DocRE performance, candidate-relation templates, and existing paradigms motivate fine-tuning and a paradigm shift.The table caption reports that all four tested RE paradigms perform poorly.Source: §2 · p. 3, Table 1 · p. 3
3 Methodology
The methodology introduces RHF, which extracts relations, then heads, then triplet facts, and applies this staged paradigm through Re-DocRED instruction tuning and QLoRA modules. It also rewrites all 96 relation descriptions and uses stage-specific modules in AutoRE.
3.1 RE Paradigms
- RHF identifies relations, then a head for each relation, and finally the corresponding triplet facts in a step-by-step extraction pipeline.The authors report that this finer-grained approach enhances recall; extending a thought chain can also introduce more erroneous facts.Source: §3.1 · pp. 3–4
- In the compared paradigms, RHF adds relation-specific head identification before extracting the corresponding facts.The alternatives include directly generating facts or first predicting relations and then generating facts.Source: §3.1 · p. 3
3.2 Dataset Processing
- The training templates divide each Re-DocRED sample into relation extraction, head-entity extraction, and triplet extraction stages.The authors systematically rewrote all 96 relation descriptions and report that this markedly improved model performance.Source: §3.2 · pp. 4–5
3.3 QLoRA Tuning
- AutoRE combines three distinct QLoRA modules, each fine-tuned for a specific RHF stage and its dataset.The table reports AutoRE results across different pretrained language models and says all outperform TAG.Source: §3.3 · p. 5, Table 4 · p. 5
4 Experiment
AutoRE is evaluated on Re-DocRED using strict Micro F1, with a correct prediction requiring the relation and both entities. The reported results compare AutoRE with TAG and examine paradigm choices, relation descriptions, and training configurations.
4.1 Experimental Setup
- The evaluation used refined Re-DocRED test and validation sets and strict Micro F1, counting a prediction correct when it matches any valid head-tail alias pair.The test set contains 499 articles and 17,448 triplet facts; the validation set contains 498 articles and 17,236 triplets.Source: §4.1 · p. 5
4.2 Overall Result
- 7.44% on dev and 5.12% on test: Mistral-7B AutoRE’s reported improvements over TAG establish the paper’s overall Re-DocRED result.The result follows training three distinct QLoRA modules and combining them for final dev and test performance.Source: §4.2 · pp. 5–6
4.3 Ablation
- Three distinct QLoRAs outperformed one QLoRA across RHF stages, and relation descriptions benefited performance.The authors attribute the multi-module advantage to uneven subtask data volumes: relations 2.8%, head entities 24.23%, and factual triples 72.97%; combined training tends to favor triple prediction while relation prediction is relatively insufficient.Source: §4.3 · p. 7
- 10.03% on dev and 9.03% on test: Vicuna-7B surpassed TAG by these amounts, while ChatGLM3-6B also surpassed current SOTA levels.The authors suggest ChatGLM3-6B’s greater proportion of Chinese pre-training may explain its lower performance on the English task.Source: §4.3 · p. 7
5 Conclusion
The paper presents RHF and AutoRE as a DocRE approach using LLMs with QLoRA. It reports state-of-the-art results on Re-DocRED and identifies a unified RE framework as future work.
5 Conclusion
- AutoRE combines LLMs with QLoRA and achieves state-of-the-art results on Re-DocRED for document-level extraction of multiple relations.The authors’ future goal is a comprehensive, unified RE framework.Source: §5 · p. 7
Limitations
The limitations concern relation coverage in Re-DocRED and AutoRE’s inability to handle unseen relations.
Limitations
- Real-world applications can involve thousands of relations, far exceeding Re-DocRED’s 96; broader datasets and relation coverage are needed for these scenarios.Source: Limitations · p. 7
- AutoRE cannot handle unseen relations, a generalizability limitation the authors attribute primarily to the limited scope of its training data.Source: Limitations · p. 7