Source-linked AI summary
InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction
Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang, Siyuan Li, Chunsai Du
TL;DR
Large language models still perform poorly on information extraction, motivating a unified approach that can generalize across IE tasks. InstructUIE uses instruction-tuned text-to-text generation and introduces a 32-dataset benchmark, achieving comparable supervised performance to Bert and significantly better zero-shot performance than state-of-the-art systems and GPT-3.5.
Problem
18.22 F1 on Ontonotes shows that gpt-3.5-turbo has a substantial performance gap on information-extraction tasks.
Method
InstructUIE reformulates IE as instruction-guided natural-language generation with task schemas, candidate options, and auxiliary extraction tasks.
Results
InstructUIE performs comparably to Bert in supervised settings and significantly outperforms state-of-the-art systems and GPT-3.5 in zero-shot settings.
Takeaways & Limitations
IE INSTRUCTIONS standardizes evaluation across 32 diverse datasets, enabling consistent comparison of models on information-extraction tasks.
Abstract
from arXiv · showhide
Large language models have unlocked strong multi-task capabilities from reading instructive prompts. However, recent studies have shown that existing large models still have difficulty with information extraction tasks. For example, gpt-3.5-turbo achieved an F1 score of 18.22 on the Ontonotes dataset, which is significantly lower than the state-of-the-art performance. In this paper, we propose InstructUIE, a unified information extraction framework based on instruction tuning, which can uniformly model various information extraction tasks and capture the inter-task dependency. To validate the proposed method, we introduce IE INSTRUCTIONS, a benchmark of 32 diverse information extraction datasets in a unified text-to-text format with expert-written instructions. Experimental results demonstrate that our method achieves comparable performance to Bert in supervised settings and significantly outperforms the state-of-the-art and gpt3.5 in zero-shot settings.
1 Introduction
InstructUIE addresses the performance gap of large language models on information extraction by unifying IE tasks through instruction tuning. The paper also introduces a 32-dataset benchmark and reports strong supervised and zero-shot results.
- 18.22 F1 on Ontonotes illustrates the unsatisfactory information-extraction performance of gpt-3.5-turbo.
- The framework reformulates IE as natural-language generation, using task instructions, candidate options, and generated target structures with types.
- InstructUIE is an end-to-end framework that uses natural-language instructions to guide large language models on IE tasks.
- IE INSTRUCTIONS contains 32 diverse information-extraction datasets unified in a text-to-text format with expert-written instructions.
- InstructUIE matches Bert comparably in supervised settings and significantly outperforms state-of-the-art systems and GPT-3.5 in zero-shot settings.
2 Methodology
The methodology represents diverse information-extraction tasks in a shared text-to-text schema and trains auxiliary subtasks alongside the main task. IE INSTRUCTIONS standardizes evaluation across 32 datasets and three IE task types.
- Instruction Tuning Background: Instruction tuning guides a language model to generate output structures and labels from source text and human-readable task instructions.
- Task Schema: Each task instance uses task instruction, options, text, and output to represent IE tasks in a sequence-to-sequence format.
- Task Schema: Options constrain the model’s output labels, while task instructions describe the extraction target, output format, and applicable rules.
- Task Schema: Outputs convert original tags into natural-language formats for NER, relation extraction, and event extraction, using None when no matching structure exists.
- Auxiliary Tasks: Auxiliary span, typing, entity-pair, relation-classification, trigger, and argument-extraction tasks provide complementary information for common structures and diverse semantics.
- IE INSTRUCTIONS: IE INSTRUCTIONS contains 32 public datasets spanning NER, RE, and EE across domains, with labels normalized and datasets converted to text-to-text format.
- IE INSTRUCTIONS: The benchmark provides a standardized platform for more accurate model comparisons and developing more effective and robust IE models.
- Experimental Results: For 20 NER datasets, InstructUIE outperforms Bert on 17 using Entity F1.
3 Experiments
The experiments evaluate InstructUIE across three information extraction tasks under supervised and zero-shot settings, using standardized metrics and comparisons with strong baselines.
- Experimental setup: Experiments use 11B FlanT5 as the InstructUIE backbone and evaluate supervised and zero-shot settings.The study describes the experimental setup, datasets, and comparison methods for these settings.
- Tasks: The evaluation covers named entity recognition, relation extraction, and event extraction tasks.The supervised experiments are conducted on these three tasks.
- Results: 67.98% average Relation Strict F1 is achieved across eight relation-extraction datasets, comparable to the baseline.The reported result summarizes InstructUIE’s performance on the RE task.
- Baselines: InstructUIE is compared with UIE, USM, and BERT as strong baseline models.UIE and USM are unified information extraction frameworks, while BERT is widely used as a text encoder.
- Evaluation: Span-based offset Micro-F1 is the primary evaluation metric, with task-specific correctness criteria for entities, relations, triggers, and arguments.NER requires correct entity boundaries and types; RE requires correct subject, object, and relation; EE reports trigger and argument metrics.
b. Event Argument F1
InstructUIE achieves strong event-extraction results across trigger and argument evaluation, including state-of-the-art performance on nearly all reported datasets.
- Event Argument F1: 72.94% Event Argument F1 on ACE2005 exceeds UIE by 18 points and USM by 17 points.
- Event Extraction: InstructUIE achieves state-of-the-art performance on all event-extraction datasets except CASIE’s Event Trigger F1 metric.
- Evaluation: The benchmark evaluates information extraction under supervised and zero-shot settings across diverse datasets and task types.
4 Related Work
Prior instruction-tuning work demonstrated generalization mainly on broad NLP tasks, while information extraction traditionally relied on separate task-specific models. InstructUIE addresses this gap by training a unified IE framework across diverse instructed tasks.
- 4.1 Instruction Tuning: Instruction tuning guides large language models with natural-language instructions for downstream tasks.
- 4.1 Instruction Tuning: Recent instruction-tuning research focused on question answering and text classification rather than information extraction.
- Research Gap: No prior work had examined training one model across a wide range of information-extraction tasks with varied instructions.
- Information Extraction: Traditional information-extraction methods typically design separate architectures and train separate models for different tasks.
5 Conclusion
InstructUIE is presented as an end-to-end framework for universal information extraction guided by natural-language instructions. Its 32-dataset benchmark uses a unified text-to-text format, and experiments report strong supervised and zero-shot performance from one multi-task model.
- InstructUIE is an end-to-end framework that uses natural-language instructions to guide large language models on information extraction tasks.
- The IE INSTRUCTIONS benchmark unifies 32 diverse information extraction datasets in a text-to-text format for consistent evaluation.
- InstructUIE achieves state-of-the-art results under supervised and zero-shot settings while solving many tasks with a single multi-task model.
6 Appendix
The appendix documents IE INSTRUCTIONS, task-specific instructions, and overall evaluation results. It covers NER, RE, and EE datasets, specifies output targets and formats, and compares InstructUIE with established baselines.
- 6.1 Data Details: IE INSTRUCTIONS contains 32 publicly available datasets covering named entity recognition, relation extraction, and event extraction.
- 6.1 Data Details: Dataset splits use official partitions when available, while datasets lacking them are divided or resampled into training, validation, and test sets.
- 6.2 Instruction Details: NER extracts entities and types, RE extracts relation triplets, and EE extracts event types, triggers, and arguments.
- 6.2 Instruction Details: Task instances are described through instructions that specify the extraction target, output structure, and constraints, alongside task-specific label options.
- InstructUIE performs better or comparably to BERT on several popular NER datasets, matches baselines on most RE datasets, and surpasses prior systems on some EE datasets.