PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Structure Generation for Universal Information Extraction

Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, Hua Wu

arXiv:2203.12277v1cs.CL

TL;DR

Information extraction must handle diverse targets, structures, and schemas, while existing task-specialized solutions limit sharing and adaptation. UIE unifies extraction through structure generation, schema-guided prompting, and large-scale pre-training. Across 4 tasks and 13 datasets, it improves performance in supervised, low-resource, and few-shot settings.

  • Problem

    Information extraction uses diverse targets, structures, and demand-specific schemas, while task-specialized solutions limit knowledge sharing and adaptation.

  • Method

    UIE encodes heterogeneous structures with a structural extraction language, controls generation with structural schema instructor, and learns general abilities through large-scale pre-training.

  • Results

    Across 4 IE tasks and 13 datasets, UIE improves performance in supervised, low-resource, and few-shot settings, including 1.42% F1 improvement over task-specialized architectures on supervised datasets.

  • Takeaways & Limitations

    The results support UIE’s effectiveness, universality, and transferability across different IE tasks, settings, and scenarios.

Abstract

from arXiv · show

Information extraction suffers from its varying targets, heterogeneous structures, and demand-specific schemas. In this paper, we propose a unified text-to-structure generation framework, namely UIE, which can universally model different IE tasks, adaptively generate targeted structures, and collaboratively learn general IE abilities from different knowledge sources. Specifically, UIE uniformly encodes different extraction structures via a structured extraction language, adaptively generates target extractions via a schema-based prompt mechanism - structural schema instructor, and captures the common IE abilities via a large-scale pre-trained text-to-structure model. Experiments show that UIE achieved the state-of-the-art performance on 4 IE tasks, 13 datasets, and on all supervised, low-resource, and few-shot settings for a wide range of entity, relation, event and sentiment extraction tasks and their unification. These results verified the effectiveness, universality, and transferability of UIE.

1 Introduction

UIE addresses the diversity and fragmentation of information extraction by unifying heterogeneous tasks as text-to-structure generation. It combines uniform structure encoding, schema-guided generation, and large-scale pre-training, achieving improvements across tasks and resource settings.

  • Motivation: Information extraction spans varying targets, heterogeneous structures, and demand-specific schemas, leading most approaches to use dedicated architectures and isolated models.These specialized solutions hinder architecture development, knowledge sharing, and cross-domain adaptation.
  • Motivation: Different IE tasks can be decomposed into shared spotting and associating operations over task-specific structures.For example, entity extraction spots typed mention spans, while relations associate spans through semantic roles.
  • Approach: UIE uses a structural extraction language to encode heterogeneous IE structures into a uniform representation.The framework also includes a unified structure generation network for this representation.
  • Results: 1.42% F1 scores improvements over state-of-the-art task-specialized architectures were achieved on all datasets in supervised settings.Experiments covered 4 main IE tasks, 13 datasets, and supervised, low-resource, and few-shot settings; few-shot and low-resource results also outperformed baselines by a large margin.
  • Approach: UIE is a unified text-to-structure generation architecture that models different IE tasks, adaptively generates targeted structures, and learns general IE abilities from different knowledge sources.Its structural schema instructor controls what to spot, associate, and generate.
  • Approach: UIE is pre-trained as a large-scale text-to-structure generation model to support knowledge sharing and adaptation across IE settings.The paper describes it as the first text-to-structure pre-trained extraction model.

2 Unified Structure Generation for Universal Information Extraction

UIE unifies diverse information-extraction tasks as text-to-structure generation, using a structured extraction language and schema-based prompting to encode and control heterogeneous outputs.

  • Motivation: IE tasks differ in target structures and demand-specific schemas, creating challenges for uniform modeling and adaptive extraction control.Examples include entities, relations, and events with spans, triplets, records, and task-specific schemas.
  • Structured Extraction Language: UIE encodes heterogeneous extraction structures with the structured extraction language (SEL), which represents spotting, associating, and information-span units hierarchically.SPOTNAME and ASSONAME identify types and associations, while INFOSPAN represents corresponding text spans; colons map spans to names and parentheses form hierarchy.
  • Structured Extraction Language: SEL represents entities as spotting structures and relations or events as association structures within one unified representation.Examples include person:Steve, organization:Apple, time:1997, and event arguments associated with the trigger became.
  • Structured Extraction Language: SEL allows different IE tasks to share one text-to-structure generation process, support joint extraction, and use compact output structures.Entity recognition and event detection share spotting grammar, while relation and event extraction share association grammar despite differing structures.
  • Structural Schema Instructor: The structural schema instructor (SSI) uses schema-based prompts to control which information UIE spots, associates, and generates.SSI prepends targeted SPOTNAME, ASSONAME, and special-symbol segments to the input text, such as person, company, and work for.
  • Structure Generation: UIE encodes the schema instructor and text, then autoregressively decodes a linearized SEL sequence into an extracted information record.A Transformer encoder produces hidden representations, and a Transformer decoder predicts SEL tokens until <eos>.
  • Transfer and Sharing: Schema-based prompting guides SEL generation, transfers general IE ability to new tasks, and supports sharing across labels and tasks.The text-to-structure paradigm also verbalizes labels as natural-language tokens, enabling transfer from pretrained language models and related tasks.

3 Pre-training and Fine-tuning for UIE

UIE is pre-trained on heterogeneous text, text-structure, and structure data to capture shared information-extraction abilities, then fine-tuned for specific tasks and settings. Its pre-training combines text-to-structure mapping, valid structure generation, and semantic representation objectives, while rejection noise supports selective fine-tuning.

  • Pre-training corpus: UIE pre-training uses web-derived text, structured records, and text-structure pairs to capture common IE abilities.Dpair aligns Wikipedia with Wikidata; Drecord draws from ConceptNet and Wikidata; Dtext uses English Wikipedia.
  • Text-to-structure pre-training: Text-to-structure pre-training samples positive and negative schemas so UIE learns general mappings rather than memorizing observed records.Negative schemas are combined with positive schemas into a meta-schema for each text-record pair.
  • Structure-generation pre-training: Structure-generation pre-training treats records as SEL expressions, enabling the decoder to learn SEL regularities and interactions between labels.Drecord supplies structured records for this decoder pre-training objective.
  • Semantic representation pre-training: Span-corruption pre-training on unstructured text retrofits semantic representations and alleviates catastrophic forgetting of SPOTNAME and ASSONAME token semantics.The corrupted source text is paired with corrupted target spans in the masked-language-modeling objective.
  • Final pre-training criteria: The final pre-training objective combines the text-to-structure, structure-generation, and text-representation tasks, with heterogeneous instances packed into shared batches.UIE-base and UIE-large are initialized from T5-v1.1-base and T5-v1.1-large.
  • On-demand fine-tuning: UIE is adapted to downstream IE tasks through fine-tuning, while rejection noise teaches it to output [NULL] for schema elements unsupported by the input.During learning, negative SPOTNAME and ASSONAME units are randomly inserted; [NULL]-valued spans are ignored during inference.

4 Experiments

UIE is evaluated across diverse IE tasks, datasets, and resource settings, with experiments examining supervised performance, adaptation, pre-training components, and rejection noise. Results show broad gains and support the framework’s cross-task transferability.

  • Experimental Settings: Experiments cover 13 IE benchmarks across entity, relation, event, and sentiment extraction, including combined tasks.The evaluation includes supervised, low-resource, and few-shot settings.
  • Supervised Settings: UIE improves average supervised performance by 1.42% F1 over state-of-the-art task-specialized architectures across datasets.The comparison excludes additional dataset-specific knowledge and larger-scale contexts.
  • Supervised Settings: UIE’s pre-trained text-to-structure model captures transferable extraction abilities, improving all tasks, especially event and sentiment extraction.The paper attributes this transfer to universal modeling and large-scale pre-training, even when those task types are absent from pre-training data.
  • Low-resource Settings: SSI-equipped UIE improves average performance by 4.16 points in n-shot experiments and 3.30 points in n-ratio experiments over UIE without SSI.The comparison evaluates six low-resource partitions across four IE tasks.
  • Ablations on Pre-training Tasks: Structure-generation pre-training raises CoNLL04 performance from 72.12 to 75.70 and 16res performance from 72.03 to 74.28.The ablation identifies SEL and sequence-to-structure mapping pre-training as crucial, particularly for small datasets.
  • Ablations on Pre-training Tasks: Removing LPair decreases relation, event, and sentiment performance by 0.90, 1.43/1.48, and 0.46 points, respectively.The result supports LPair’s role in learning extraction ability.
  • Effects of Rejection Noise: Rejection noise improves average precision by 13.16 points in the CoNLL03 10-shot development setting.The method addresses the negative influence of mis-generated labels on extraction precision.

5 Related Work

Prior IE research largely develops task-specialized architectures and pre-training techniques, whereas UIE targets universal text-to-structure modeling for knowledge sharing and adaptation.

  • Prior IE Architectures: Classical IE architectures include sequence tagging, span classification, and machine reading comprehension.Task-specific pre-training techniques have also been developed for these architectures.
  • Research Focus: Previous IE studies primarily focus on developing more effective task-specialized models.The paper contrasts this focus with universal modeling across IE tasks.
  • Universal IE: UIE frames diverse IE tasks within one unified text-to-structure framework to support knowledge sharing and quick adaptation.The paper presents this goal against the background of broader efforts to build universal NLP models.

6 Conclusion

The paper concludes that UIE unifies diverse IE tasks through text-to-structure generation and demonstrates effectiveness, universality, and transferability across settings. It also releases a pre-trained model and identifies extensions to knowledge-aware and document-aware IE as future work.

  • Conclusion: UIE universally models different IE tasks, adaptively generates targeted structures, and learns general IE abilities from different knowledge sources.This is the framework’s central contribution.
  • Conclusion: Experiments verify UIE’s universality, effectiveness, and transferability in supervised and low-resource settings.The conclusion summarizes the reported experimental scope without adding task-specific results.
  • Future Work: The paper releases a large-scale pre-trained text-to-structure model for future studies.Future work extends UIE toward knowledge-base-aware entity linking and document-aware co-reference.

A Experiment Details

The experiment-details section states that the paper describes pre-training and downstream-task fine-tuning procedures.

  • Experiment Details: The experimental details cover pre-training and fine-tuning on downstream tasks.

A.1 Pre-training Details

UIE pre-training combines structured records, text-record pairs, unstructured text, and ConceptNet tuples into a unified text-to-structure learning process. The model is initialized from T5 checkpoints and trained with pair, record, and text objectives.

  • Data construction: Wikipedia, Wikidata, and ConceptNet supply the pre-training data for UIE.Wikipedia and Wikidata provide structured and text-linked knowledge, while ConceptNet adds common-sense associations.
  • Pre-training process: UIE pre-training processes triplets of schema prompt, source text, and target structure using pair, record, and text losses.The combined objective is loss = LPair + LRecord + LText, with corrupted text samples used for the text objective.
  • Data construction: 65M instances are constructed for each of Dpair, Drecord, and Dtext, with 50K instances retained for development.The datasets are derived from Wikipedia and Wikidata resources.
  • Data construction: ConceptNet contributes 1M instances to augment the structured extraction language with common-sense knowledge.ConceptNet tuples omit context and entity types, so the corresponding fields are left blank.
  • Training details: UIE-base and UIE-large are initialized from T5-v1.1 checkpoints and trained for 500K steps with batch size 512 on eight NVIDIA A100 GPUs.Training uses Adam, a learning rate of 1e-4, and linear scheduling with 6% warmup.

A.2 Details of Downstream Tasks

Downstream evaluation covers entity, relation, event, and sentiment extraction across 13 datasets. Performance is measured primarily with span-based offset Micro-F1 under task-specific correctness criteria, while offset conversion uses a heuristic mapping.

  • Task coverage: UIE is evaluated on four IE tasks spanning entity, relation, event, and sentiment extraction across 13 datasets.The downstream benchmark includes multiple domains, languages, and sentiment aspect datasets.
  • Entity extraction: Entity experiments use ACE04, ACE05-Ent, ACE047, ACE05-Ent8, and CoNLL03 datasets with prior preprocessing and splits where applicable.Nested entity datasets receive the preprocessing used by previous work.
  • Relation extraction: Relation experiments cover ACE05-Rel, CoNLL04, NYT10, and SciERC across several languages and domains.The experiments use end-to-end relation extraction settings and follow prior preprocessing procedures.
  • Event and sentiment extraction: Event evaluation uses ACE05-Evt and CASIE, while sentiment evaluation uses SemEval 2014, 2015, and 2016 aspect-sentiment datasets.CASIE is split chronologically into 697 training, 100 validation, and 200 test documents.
  • Evaluation: Span-based offset Micro-F1 is the primary metric, with correctness defined by task-specific matching of offsets, types, boundaries, and relation or event labels.Relation Strict requires entity offsets and types, whereas Relation Triplet uses subject and object strings with boundary matching.
  • Evaluation: The reported offset evaluation has less than 0.5% error from heuristic span-to-offset mapping.More complicated mapping methods are left for future work.

A.3 Comparison of UIE-base

UIE-base shows strong transfer across sentiment extraction and overlapping entity-relation extraction. The results attribute this portability to unified pre-training, flexible generation, and universal SEL expressions.

  • Sentiment extraction: 3.24 average improvement over T5-v1.1-base is achieved across four aspect-based sentiment analysis datasets.UIE-base remains competitive with state-of-the-art task-specific architectures on sentiment triplet extraction.
  • Relation extraction: SEL and SSI achieve state-of-the-art performance on NYT without UIE pre-training.UIE pre-training is omitted because NYT substantially overlaps with the pre-training data.
  • Relation extraction: Flexible generation and universal SEL expressions allow UIE to naturally handle entity overlap problems on NYT.This result is reported for the joint entity and relation extraction setting.
Loading 2203.12277v1…