Source-linked AI summary
Large Language Models for Generative Information Extraction: A Survey
Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, Enhong Chen
TL;DR
Generative LLM methods for information extraction have expanded rapidly, but existing approaches and benchmarks remain limited in generalizability across tasks and domains. This survey categorizes the field by IE subtasks and techniques, reviews applications and evaluations, and finds stronger performance for universal models on complex relation extraction and generative methods on event extraction. It also identifies unresolved challenges and future directions for universal and low-resource IE.
Problem
Existing generative IE methods and benchmarks are often tailored to specific domains or tasks, while unified LLM methods still face long-context and structured-output alignment limitations.
Method
The survey organizes LLM-based generative IE research by IE subtasks and techniques, reviews domain applications and evaluations, compares representative methods, and analyzes future directions.
Results
Universal IE models generally perform better on complex strict relation extraction, while generative methods significantly outperform discriminative methods in event extraction, especially argument classification.
Takeaways & Limitations
The survey provides a resource for exploring more efficient use of LLMs for IE and highlights universal and low-resource learning as continuing research priorities.
Takeaways & Limitations
Universal IE remains constrained by long-context inputs and misalignment of structured outputs, while low-resource systems still face challenges in example selection and cross-domain learning.
Abstract
from arXiv · showhide
Information extraction (IE) aims to extract structural knowledge from plain natural language texts. Recently, generative Large Language Models (LLMs) have demonstrated remarkable capabilities in text understanding and generation. As a result, numerous works have been proposed to integrate LLMs for IE tasks based on a generative paradigm. To conduct a comprehensive systematic review and exploration of LLM efforts for IE tasks, in this study, we survey the most recent advancements in this field. We first present an extensive overview by categorizing these works in terms of various IE subtasks and techniques, and then we empirically analyze the most advanced methods and discover the emerging trend of IE tasks with LLMs. Based on a thorough review conducted, we identify several insights in technique and promising research directions that deserve further exploration in future studies. We maintain a public repository and consistently update related works and resources on GitHub (\href{https://github.com/quqxui/Awesome-LLM4IE-Papers}{LLM4IE repository})
1 Introduction
This survey examines how LLMs support generative information extraction, motivated by IE’s structural complexity and the limitations of existing surveys. It organizes methods by IE subtasks and techniques, reviews domain applications and evaluations, and discusses future directions.
- IE converts plain text into structured knowledge such as entities, relations, and events for downstream tasks including knowledge graphs, reasoning, and question answering.
- LLMs’ text understanding, generation, zero-shot, few-shot, and agentic capabilities have driven interest in generating structural information for IE.
- Existing surveys insufficiently cover universal multi-task frameworks and cutting-edge learning techniques for limited-training-data scenarios.
- Recent work includes specialized single-subtask frameworks and universal frameworks that model multiple IE tasks through instructive prompts and inter-task dependencies.
- Applying LLMs to IE remains challenging because of structured-output misalignment, hallucination, contextual dependence, computational cost, and knowledge-update difficulties.
- The survey categorizes IE subtasks and LLM techniques, reviews domain applications and evaluations, compares representative methods, and identifies future research directions.
2 Preliminaries of Generative IE
The survey formalizes discriminative and generative IE, focusing on NER, RE, and EE. Generative IE conditions autoregressive output sequences on input text and prompts, while task definitions specify distinct extraction targets.
- The survey focuses primarily on Named Entity Recognition, Relation Extraction, and Event Extraction because they receive the most attention in IE research.
- Discriminative IE: Discriminative IE maximizes likelihood over annotated sentences and potentially overlapping relation triples, or over sequential BIESO tag sequences.
- Generative IE: Generative IE maps input tokens X and prompt P to a target extraction sequence Y by maximizing its autoregressive conditional probability.
- Generative IE: The LLM parameters θ may be frozen or trainable, and prompts or instructions can be appended to the input to improve task comprehensibility.
- IE subtasks: NER identifies entity spans and assigns entity types, whereas RE settings range from relation classification to relation triplets and strict relation extraction.
- IE subtasks: EE comprises event detection, which identifies and classifies triggers, and event-argument extraction, which identifies and classifies arguments by role.
3 LLMs for Different Information Extraction Tasks
The survey reviews LLM-based methods across NER, RE, EE, and unified IE, combining task-specific analyses with experimental comparisons. Results indicate strong potential for generative and universal models, while performance depends on learning paradigm, task complexity, and dataset distribution.
- Named Entity Recognition: Few-shot and zero-shot NER remain substantially behind supervised fine-tuning and data augmentation.The survey reports a large performance gap between few-/zero-shot settings and SFT or DA.
- Named Entity Recognition: GPT-NER achieves at least a 6% F1 advantage over other ICL methods on every dataset, reaching about 19% higher.The comparison finds large method-level variation despite relatively small backbone differences.
- Named Entity Recognition: After SFT, model performance differs only slightly even when backbone parameter counts vary by a few hundred times.SFT reduces performance differences among backbones relative to the larger variation observed under ICL.
- Named Entity Recognition: Universal SFT models vary substantially across datasets, while EnTDA remains stable and outperforms other methods on all NER datasets.YAYI-UIE and KnowCoder perform better on CoNLL03 but worse on GENIA, which the survey associates with domain-distribution gaps; EnTDA supports the robustness of DA for specific tasks.
- Named Entity Recognition: NER experiments compare LLM methods across five datasets using Micro-F1, covering ICL, SFT, DA, universal models, and discriminative systems.Table 1 distinguishes universal models, discriminative models, and IE techniques including CDL, zero-shot prompting, ICL, SFT, and DA.
- Relation Extraction: Universal IE models generally handle harder relation-strict tasks better, whereas task-specific methods address simpler relation-classification settings.The survey attributes the universal-model advantage on strict RE to learning dependencies across multiple tasks, and reports larger model-performance disparities in RE than in NER.
- Event Extraction: Event-extraction studies mainly use SFT, and generative methods outperform discriminative methods especially on argument classification.Only a few methods use zero-shot or few-shot learning, while the reported generative advantage is particularly large for argument classification.
- Universal Information Extraction: Universal IE frameworks unify NER, RE, and EE through NL-LLM or Code-LLM paradigms, with each offering distinct advantages and limitations.NL-LLMs support concise, easy-to-design prompts but may produce outputs with unnatural syntax; code-based methods represent diverse schemas more formally but require lengthy class definitions that limit context size.
4 Techniques of LLMs for Generative IE
The survey organizes LLM techniques for generative IE around data augmentation, prompt design, zero-shot learning, and related strategies, while comparing their benefits and limitations.
- The survey categorizes LLM techniques for generative IE, including data augmentation, prompt design, zero-shot learning, and constrained decoding generation.
- Data Augmentation: Data augmentation uses LLMs to generate meaningful, diverse training data through annotation, knowledge retrieval, inverse generation, or synthetic datasets for fine-tuning.These strategies aim to expand training data while avoiding unrealistic or misleading patterns.
- Prompt Design: Prompt design guides LLM behavior without changing model parameters, using approaches such as question answering and chain-of-thought reasoning.QA reformulates IE as multi-turn question answering, while CoT decomposes tasks into step-wise reasoning.
- Zero-shot Learning: Zero-shot methods address generalization to unseen tasks and domains by aligning IE with instruction-tuned formats such as multi-turn question answering.BART-Gen improved portability to unseen event types, while QA4RE and ChatIE targeted zero-shot IE through QA reformulation.
- LLMs are designed for free-form text generation and may struggle with structured prediction, motivating fine-tuning and data augmentation techniques for IE.Supervised fine-tuning generally outperforms purely prompt-based methods, while augmentation can improve generalization but introduce noise.
5 Applications on Specific Domains
LLM-based IE has been applied across multimodal, multilingual, medical, scientific, astronomical, historical, and legal domains, with domain-specific methods and challenges.
- The survey covers LLM applications in multimodal, multilingual, medical, scientific, astronomical, historical, and legal information extraction.It also reports statistics for research in specific domains in Table 5.
- Conditional prompt distillation combines text-image pairs with chain-of-thought knowledge to improve multimodal NER and relation extraction.
- Prompting strategies target specialized astronomical entities, including celestial object identifiers and telescope-related information.
- Historical-text NER with ChatGPT faces challenges involving annotation guidelines, entity complexity, code-switching, and prompt specificity.
6 Evaluation & Analysis
Evaluation studies find that LLM performance remains uneven across IE subtasks and settings, with errors and benchmark design affecting conclusions.
- Few-shot GPT-3 prompting achieves near-SOTA relation extraction performance, while ChatGPT still struggles with event extraction requiring complex instructions and robustness.
- ChatGPT mostly underperforms BERT-based models in standard IE but performs excellently in OpenIE, while unannotated spans are the predominant identified error type.The evaluation also examined explainability, calibration, and faithfulness.
7 Future Directions
Future work centers on making generative IE more general, effective in low-resource settings, better prompted, and more reliable for difficult Open IE tasks.
- Universal IE: Universal IE frameworks should adapt flexibly across domains and tasks while addressing long-context inputs and structured-output misalignment.The survey suggests integrating task-specific insights into universal models.
- Low-Resource IE: Low-resource IE requires better in-context example selection, robust cross-domain learning, and efficient LLM-assisted annotation.Suggested techniques include domain adaptation and multi-task learning from resource-rich domains.
- Prompt Design for IE: Prompt design should align input-output formats with LLM pretraining and support reasoning through techniques such as Chain-of-Thought prompting.The survey identifies both format alignment and improved model understanding as prompt-design goals.
- Open IE: Open IE remains difficult because models receive no candidate label set and must rely on task comprehension, with poor performance persisting on challenging tasks.The survey identifies further exploration as necessary for these harder settings.
8 Benchmarks & Backbones
The survey compiles representative IE datasets and benchmarks across tasks and domains, provides repository access, and summarizes commonly used generative IE backbones.
- Benchmarks: Table 6 summarizes representative datasets for named entity recognition, relation extraction, and event extraction.The survey introduces these datasets to help readers understand the three principal IE subtasks.
- Benchmarks: CoNLL03 contains English and German news articles annotated with four entity types for named entity recognition.The English portion comes from a Reuters-curated shared-task dataset.
- Benchmarks: CoNLL04 contains news sentences with entity-relation triples, four entity types, and five relation types for relation extraction.Its relation labels include Kill, Work For, Live In, Org-Based In, and Located In.
- Benchmarks: Table 7 covers common datasets across domains and tasks, with download links collected in the open-source LLM4IE repository.The table marks multimodal datasets and uses # for category or sentence counts.
- Backbones: Table 8 lists common backbones for generative information extraction and distinguishes commonly used base and large versions.The table is intended as a reference for generative IE model selection.
9 Conclusion
The conclusion presents the survey as a synthesis of universal IE frameworks, LLM-based IE techniques, domain applications, empirical analyses, challenges, and future directions.
- Conclusion: The survey introduces IE subtasks and examines universal frameworks that aim to unify all IE tasks with LLMs.It also provides theoretical and experimental analysis of these methods.
- Conclusion: Table 7 and Table 8 provide reference resources on IE datasets and common generative IE backbones.These resources complement the survey’s synthesis and analysis.
- Conclusion: It reviews LLM-based IE techniques and discusses their potential for extracting information in specific domains.The survey connects technique-focused analysis with domain-specific applications.
- Conclusion: It analyzes current challenges and presents potential future directions for more efficient utilization of LLMs in IE.The stated goal is to provide a resource for researchers exploring LLMs for IE.