Source-linked AI summary
ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, Wenjuan Han
TL;DR
Zero-shot information extraction is difficult because structured outputs with dependent elements are hard to obtain without labeled data. ChatIE addresses this gap by decomposing IE into two-stage, multi-turn question answering with ChatGPT. Across three tasks and six datasets in two languages, it achieves strong results and surpasses some full-shot models, although entity-type ambiguity remains a limitation.
Problem
LLMs show promise without parameter tuning, but their potential for zero-shot information extraction has been little studied, especially for complex structured outputs.
Method
ChatIE decomposes zero-shot IE into two multi-turn QA stages: identifying present element types, then extracting information for each type and composing structured results.
Results
ChatIE succeeds across relation extraction, named entity recognition, and event extraction, achieving impressive performance and surpassing some full-shot models on several datasets.
Takeaways & Limitations
The results support a zero-shot IE paradigm in which experts decompose IE into simpler sub-tasks and execute the specifications through chat-like prompts without training.
Takeaways & Limitations
Entity-type ambiguity produces errors, including predictions whose boundaries are correct but whose types differ from the ground-truth labels.
Abstract
from arXiv · showhide
Zero-shot information extraction (IE) aims to build IE systems from the unannotated text. It is challenging due to involving little human intervention. Challenging but worthwhile, zero-shot IE reduces the time and effort that data labeling takes. Recent efforts on large language models (LLMs, e.g., GPT-3, ChatGPT) show promising performance on zero-shot settings, thus inspiring us to explore prompt-based methods. In this work, we ask whether strong IE models can be constructed by directly prompting LLMs. Specifically, we transform the zero-shot IE task into a multi-turn question-answering problem with a two-stage framework (ChatIE). With the power of ChatGPT, we extensively evaluate our framework on three IE tasks: entity-relation triple extract, named entity recognition, and event extraction. Empirical results on six datasets across two languages show that ChatIE achieves impressive performance and even surpasses some full-shot models on several datasets (e.g., NYT11-HRL). We believe that our work could shed light on building IE models with limited resources.
1 Beijing Jiaotong University, Beijing, China 2 DAMO Academy, Alibaba Group, China
ChatIE reframes zero-shot information extraction as a multi-turn question-answering problem, using two stages to identify relevant types and then extract structured information. It applies this framework to relation extraction, named entity recognition, and event extraction.
- Motivation: ChatIE addresses zero-shot information extraction, where extracting structured data from unstructured text normally requires substantial labeling effort.The framework targets relation extraction, named entity recognition, and event extraction.
- Framework: ChatIE transforms zero-shot IE into a multi-turn question-answering problem with a two-stage framework based on ChatGPT.The first stage identifies possible element types, and the second performs task-specific extraction.
- Stage I: Stage I identifies entity, relation, or event types present in a sentence, filtering absent types to reduce the search space and computational complexity.The stage uses task-specific templates and candidate type lists in a question to ChatGPT.
- Stage II: Stage II uses multiple question-answering turns to extract information for each identified type and compose the outputs into structured results.Chained templates support cases where extracting one element depends on previously extracted elements.
- Task applications: For relation extraction, the framework predicts entity-relation triples by first identifying relation types and then extracting subjects, objects, and complex objects through chained questions.Stage II questions use previously extracted relation types and ChatGPT’s accumulated dialogue information.
- Task applications: For NER and EE, Stage II extracts entities or event arguments after Stage I identifies entity or event types.EE separates event classification from argument extraction, while NER extracts entities for each detected type.
3 Experiment
The experiments evaluate ChatIE across six datasets in Chinese and English, using few-shot, full-shot, and zero-shot baselines with task-appropriate F1 metrics. Evaluation distinguishes relation matching criteria and uses boundary-and-type matching for NER.
- Datasets and baselines: Experiments cover six datasets in Chinese and English and compare ChatIE with few-shot, full-shot, and zero-shot baselines.The baseline models use the same model architecture within each task.
- Datasets and baselines: The study includes datasets for relation extraction, named entity recognition, and event extraction, with task-specific baseline systems for each dataset.The listed datasets are DuIE2.0, NYT11-HRL, MSRA, conllpp, DuEE1.0, and ACE05.
- Evaluation metrics: Relation extraction uses micro F1 with border evaluation or strict evaluation, selecting Rel for NYT11-HRL and Rel+ for DuIE2.0.The distinction reflects whether entity types are annotated in the dataset.
- Evaluation metrics: Named entity recognition uses micro F1 with complete matching, requiring both predicted entity boundaries and types to be correct.A prediction is counted as correct only when both conditions hold.
- Evaluation metrics: Event extraction uses different matching criteria for DuEE1.0 and ACE05, including word-level F-measure for DuEE1.0 and entity-level micro F1 for ACE05 arguments.ACE05 predicted arguments are matched against manually marked arguments at the entity level.
- Evaluation metrics: Table 1 reports F1 scores across six datasets and two languages, distinguishing Chinese datasets, reported scores, and supervised state-of-the-art approaches.The table caption identifies these conventions with #, *, and Sup.
4 Results
ChatIE’s two-stage ChatGPT framework substantially outperforms single-turn prompting and performs strongly across zero-shot, few-shot, full-shot, and alternative-backbone comparisons. Its results remain robust across API versions and multiple LLM backbones.
- 16.65% points average improvement: ChatIE outperforms the single-turn ChatGPT baseline across six IE datasets.The single-turn baseline performs poorly, whereas the two-stage framework succeeds.
- ChatIE is comparable to fs-20 on MSRA and outperforms fs-100 on NYT11-HRL, conllpp, and ACE05 despite using zero-shot prompting.
- ChatIE surpasses independently trained full-shot models on DuIE2.0 and DuEE1.0.These full-shot models use high-quality labeled data and are trained from scratch.
- ChatIE remains highly superior with gpt-3.5-turbo-0301 and remains valid across ChatGLM2, InstructGPT, and Llama2-7b-chat backbones.The multi-round framework is distinguished from the single-round QA baseline in the backbone comparison.
5 Analysis
The analysis examines prompt robustness, evaluation leakage, and NER error patterns. Results indicate low sensitivity to prompt wording, while newly constructed tests address possible contamination and reveal boundary and type errors.
- Robustness: The variance of F1 is very small across different textual prompt wordings and phrasings.The robustness experiment uses 100 randomly sampled samples.
- Data Leakage: Data leakage can bias test-set performance estimates because LLM training data may contain released test samples.
- Data Leakage: Three unreleased test datasets are constructed by replacing conllpp entities with entities of the same type and manually checking the modified sentences.
- Error Analysis: 9.52% / 17.79%: NER errors include correct boundaries paired with false entity types for MSRA and conllpp, respectively.The example labels China as LOC in the expected output but GPE in the model output.
- Error Analysis: 9.38% / 2.41%: NER errors include correct entity types paired with false boundaries for MSRA and conllpp, respectively.The example predicts western Canada instead of the expected Canada.
6 Case Study
The case study illustrates ChatIE’s behavior across relation extraction, named entity recognition, and event extraction. Its two-stage process handles multiple relation labels, repeated relation instances, ambiguous entity senses, and event arguments.
- Relation Extraction: ChatIE assigns two relation types to the same entity pair, extracting both directional triples for India and Delhi.The study notes that the relation-name conversion did not change results, indicating recognition of equivalent relation formulations.
- Relation Extraction: ChatIE extracts four person-company triples from one sentence containing multiple people linked to Google.It filters the person-company relation in the first stage, then outputs all four associated triples in the second stage.
- Named Entity Recognition: ChatIE identifies Tasmania and Victoria as ORGANIZATION entities in context, despite their potentially ambiguous word senses.Both words can receive other entity interpretations, but they function as team names in the sentence.
- Event Extraction: For event extraction, ChatIE detects the event type from “died” and then extracts the victim, agent, and temporal arguments.The example identifies 19 rangers as the victim, Clinton as the agent, and two date expressions as temporal information.
7 Error Analysis
The error analysis on MSRA and conllpp identifies four main error types: incorrect entity types, incorrect boundaries, unrecognized entities, and over-recognition. Several errors reflect annotation ambiguity, segmentation difficulty, insufficient context, or type ambiguity.
- I Correct Boundary but False Type: Correct boundaries can still receive false entity types when human-plausible labels differ from the dataset annotation.“China” is predicted as GPE although the ground-truth label is LOC, and the authors regard GPE as reasonable.
- II Correct Type but False Boundary: False boundaries are often acceptable under different granularity, and this error occurs more frequently on MSRA than on conllpp.The authors associate the higher MSRA rate with greater Chinese word-segmentation difficulty than in English.
- III Unrecognized: Unrecognized entities mainly result from sentence incomprehension, with insufficient context also remaining possible.The analysis does not exclude inadequate context as a contributing factor.
- IV Over-recognized: Over-recognition is common across both datasets and may arise from ambiguity in entity types.“361 Group A” is an organization but belongs to MISC, a type not predefined for MSRA and conllpp; the authors speculate training-data exposure may contribute.
8 Prompt of Vanilla Prompt vs. ChatIE
The paper compares vanilla prompts with ChatIE’s chat-based prompts. The comparison is presented across Tables 7, 9, and 8.
- Prompt Comparison: Tables 7, 9, and 8 compare vanilla prompts, labeled Row Single, with ChatIE’s chat-based prompts.The comparison covers the two prompting approaches using the table-row labels specified in the paper.
9 Related Work
The paper situates ChatIE within growing interest in using ChatGPT for downstream tasks and proposes it as a simple zero-shot IE framework. This addresses the labor and time costs of producing labeling data while exploring ChatGPT’s IE capabilities.
- Manual labeling is hectic, labor-intensive, and time-consuming, motivating zero-shot and few-shot IE research.
- The related-work discussion includes illustrations contrasting vanilla prompts with Chat-based prompts for relation extraction, event extraction, and NER.
- Recent work shows ChatGPT being applied across NLP, medicine, and online examination tasks.
- The paper explores ChatGPT’s information extraction capabilities through a simple, effective zero-shot IE framework.
10 Conclusion
The conclusion presents ChatIE as a multi-turn QA framework for zero-shot IE and reports broad evaluation across tasks, datasets, and languages. It emphasizes strong performance, including results exceeding some full-shot models.
- ChatIE is a multi-turn QA framework for zero-shot information extraction based on ChatGPT.
- The framework decomposes complex IE tasks into simpler sub-tasks, then composes turn-level outputs into structured results.
- ChatIE is evaluated on relation extraction, named entity recognition, and event extraction across six datasets in English and Chinese.
- ChatIE achieves impressive performance and surpasses some full-shot models on several datasets.
A Details of Data
The data details cover six datasets spanning relation extraction, named entity recognition, and event extraction in English and Chinese. The experiments include few-shot and full-shot baselines, with task-specific dataset structures and evaluation metrics.
- Experiments report few-shot and full-shot baselines, with three repeated random training selections for each few-shot experiment.
- The six datasets cover English and Chinese relation extraction, named entity recognition, and event extraction tasks.
- RE: NYT11-HRL contains 12 relation types, while DuIE2.0 contains 48 predefined relation types.
- NER: The conllpp and MSRA datasets contain four and three entity types, respectively.
- EE: DuEE1.0 contains 65 event types, while ACE05 provides document- and sentence-level annotations across domains including newswires and online forums.
- Evaluation: Relation extraction uses micro F1 with border evaluation on NYT11-HRL and strict evaluation on DuIE2.0.
- Evaluation: NER uses complete matching and micro F1, while event extraction uses word-level F1 for DuEE1.0 and entity-level micro F1 for ACE05.
C Details of Results
The results section compares ChatIE with few-shot, full-shot, and supervised systems across IE tasks. It also documents a major evaluation limitation: incomplete remote-supervision labels in NYT11-HRL can penalize correct extractions.
- NER: NER experiments use AdaSeq Bert-CRF for few-shot and full-shot baselines, alongside supervised approaches reported for reference.
- RE: Relation extraction compares PaddleNLP LIC2021 IE on DuIE2.0 and CasRel on NYT11-HRL across few-shot and full-shot settings.
- RE: The relation-extraction results include original-paper or reimplemented baselines and additional supervised approaches for reference.
- Limitation: NYT11-HRL’s remotely supervised gold labels are incomplete and omit some relationships, affecting precision, recall, and F1.
- EE: Event-extraction results use PaddleNLP LIC2021 EE and Text2Event-T5-base as task-specific baselines, with additional supervised approaches provided.
- EE: ChatIE trigger-classification results on ACE05 are reported separately.