PaperlayerView exact revision on arXiv

Source-linked AI summary

Empirical Study of Zero-Shot NER with ChatGPT

Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, Hongwei Wang

arXiv:2310.10035v1cs.CL

TL;DR

The paper addresses the limited evidence on LLM reasoning for information extraction, focusing on zero-shot ChatGPT NER. It adapts decomposition, syntactic augmentation, and self-consistency to NER, and reports improvements across seven benchmarks in Chinese and English and in domain-specific and general-domain settings.

  • Problem

    LLM reasoning on information extraction remained unexplored, motivating study of ChatGPT's zero-shot NER capabilities.

    Source: §1 · p. 1

  • Method

    The paper adapts reasoning to NER through label-by-label Decomposed-QA, syntactic prompting and tool augmentation, and two-stage majority voting.

    Source: §1 · pp. 1–2

  • Results

    The proposed strategies significantly facilitate zero-shot NER across seven benchmarks, including Chinese and English and domain-specific and general-domain datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study reports effectiveness across Chinese and English scenarios and domain-specific and general-domain datasets, and also verifies the methods in few-shot settings and on other LLMs.

    Source: §5 · p. 9

    For cost saving, the study does not explore combinations of different syntactic information or manual label orders on general-domain datasets.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) exhibited powerful capability in various natural language processing tasks. This work focuses on exploring LLM performance on zero-shot information extraction, with a focus on the ChatGPT and named entity recognition (NER) task. Inspired by the remarkable reasoning capability of LLM on symbolic and arithmetic reasoning, we adapt the prevalent reasoning methods to NER and propose reasoning strategies tailored for NER. First, we explore a decomposed question-answering paradigm by breaking down the NER task into simpler subproblems by labels. Second, we propose syntactic augmentation to stimulate the model's intermediate thinking in two ways: syntactic prompting, which encourages the model to analyze the syntactic structure itself, and tool augmentation, which provides the model with the syntactic information generated by a parsing tool. Besides, we adapt self-consistency to NER by proposing a two-stage majority voting strategy, which first votes for the most consistent mentions, then the most consistent types. The proposed methods achieve remarkable improvements for zero-shot NER across seven benchmarks, including Chinese and English datasets, and on both domain-specific and general-domain scenarios. In addition, we present a comprehensive analysis of the error types with suggestions for optimization directions. We also verify the effectiveness of the proposed methods on the few-shot setting and other LLMs.

1 Introduction

The paper investigates zero-shot ChatGPT NER to address the unexplored question of LLM reasoning in information extraction. It adapts reasoning strategies to NER and evaluates them across seven benchmarks.

  • LLM reasoning had been demonstrated for arithmetic and symbolic tasks, while its information-extraction capabilities remained unexplored.The paper motivates evaluating ChatGPT on IE because it tests structured prediction and language understanding.Source: §1 · p. 1
  • The study systematically investigates zero-shot NER with LLMs, emphasizing ChatGPT.Source: §1 · p. 2
  • Four NER-tailored strategies adapt reasoning methods: Decomposed-QA, syntactic prompting, tool augmentation, and two-stage majority voting.Source: §1 · p. 2
  • The proposed strategies significantly facilitate zero-shot NER across seven benchmarks, spanning Chinese and English and domain-specific and general-domain datasets.Source: §1 · p. 2

2 Related Work

The related work covers LLM reasoning methods and emerging ChatGPT applications to information extraction. This study focuses on adapting reasoning methods for zero-shot NER.

2.1 Reasoning with LLM

  • Prior LLM reasoning work elicited chain-of-thought or decomposed complex problems, while self-consistency sampled diverse answers; this study examines these approaches for zero-shot NER.Source: §2.1 · p. 2

2.2 LLM on IE

3 Method

The method adapts task decomposition, syntactic augmentation, and self-consistency to zero-shot NER. Its components separate label-specific extraction, syntax-based guidance, and voting over predictions.

3.1 Decomposed-QA

  • Decomposed-QA asks ChatGPT to identify one entity label at a time in a multi-turn dialogue, carrying earlier questions and answers into later turns.Source: §3.1 · p. 2

3 Method

  • Figure 1 contrasts vanilla NER, decomposition, syntactic prompting, tool augmentation, their combination, and two-stage voting for mentions and types.Source: Figure 1 · p. 3

3.2 Syntactic Augmentation

  • Syntactic augmentation prompts ChatGPT to analyze structure or supplies parser-generated syntax; the paper also combines tool-provided syntax with a reasoning hint.It considers word segmentation, noun phrases, POS tags, constituency trees, and dependency trees; word segmentation is used only for Chinese.Source: §3.2 · pp. 3–4

3.3 Self-Consistency with Two-Stage Majority Voting

  • For decomposed-QA, self-consistency can vote on each question before continuing the dialogue or vote across independently repeated full dialogues.Source: §3.3 · p. 4

4 Experiment

The experiments test ChatGPT’s zero-shot NER strategies across domain-specific and general-domain datasets and multiple settings. Decomposed-QA, syntactic augmentation, and self-consistency combinations improve performance, with additional analyses of error patterns, few-shot use, and other LLMs.

4.1 Setup

  • The experiments use GPT-3.5 on Chinese electric-power-domain and general-domain datasets, with SC and technique combinations tested on selected datasets and samples for cost saving.The study also evaluates GPT-3 and Llama2; Llama2 is tested on English datasets because of its then-poor Chinese support.Source: §4.1 · p. 4

4.2 Overall Performance

  • Across Chinese and English, domain-specific and general-domain scenarios, decomposed-QA and syntactic augmentation improve zero-shot NER, while SC and technique combinations bring further gains.The reported results cover F1 performance and combinations of reasoning techniques.Source: Table 1 · p. 5, §4.2.1 · p. 5, §4.2.2 · p. 5

4.2.1 Effect of Decomposed-QA

  • Decomposed-QA significantly improves all benchmarks, with average F1 gains of 9.22% on domain-specific and 3.82% on general-domain datasets.The authors suggest the larger domain-specific gains may reflect the greater challenge of out-of-distribution data.Source: §4.2.1 · p. 5

4.2.2 Effect of Syntactic Augmentation

  • Tool augmentation improves six datasets; word segmentation performs best on Chinese and POS tags boost English most, while prompting alone offers limited benefit.Complex dependency trees can be informative yet difficult for ChatGPT to understand, producing unstable performance.Source: §4.2.2 · pp. 5–6

4.3 Effect of Self-Consistency and Combinations of Reasoning Techniques

  • SC improves almost all methods; syntactic prompting hurts tool augmentation without SC but helps when paired with SC.Putting syntactic hints after rather than before the answer and sampling more responses generally improve SC performance.Source: Table 2 · p. 6, §4.3 · p. 7, Figure 3 · p. 7

4.4.1 Error Types

  • Complete-O and wrong types account for over 80% of OntoNotes 4 errors; TS-SC reduces total errors by 24.1%.The authors associate these error types respectively with annotation or prior-knowledge issues and inadequate understanding of entity types.Source: Table 3 · p. 7, §4.4.1 · p. 8

4.4.2 Case Study of Error Correction and Error Increase

  • TS-SC mainly reduces contain-gold and completely-O errors, but increases contained-by-gold and omitted-mention errors.The authors suggest more accurate, comprehensive syntactic information and added type information as possible optimization directions.Source: §4.4.2 · p. 8, Figure 5 · p. 8

4.5 More analysis

  • In few-shot evaluation, standard CoT brings no improvement and may hurt performance, whereas the proposed strategies achieve significant improvements; OntoNotes 4 results average three randomly sampled sets of 300 test samples.The methods also show consistent improvements across GPT-3 (text-davinci-003), Llama2 13B chat, and the close-sourced ChatGPT model series.Source: §4.5 · p. 9, Table 4 · p. 9, Table 5 · p. 9

5 Conclusion

The conclusion presents four strategies for zero-shot NER and reports evaluation across languages, domains, settings, and LLMs.

5 Conclusion

  • The study proposes task decomposition, syntactic augmentation, and tailored self-consistency strategies for zero-shot NER with ChatGPT.The strategies are framed as ways to simulate ChatGPT’s reasoning potential.Source: §5 · p. 9
  • The methods are evaluated in Chinese and English, on domain-specific and general-domain datasets, and in few-shot settings and other LLMs.Source: §5 · p. 9
  • The paper analyzes error types and suggests directions for optimization.Source: §5 · p. 9

6 Limitations

The limitations section identifies investigations omitted for cost-saving reasons.

6 Limitations

  • The study examines individual kinds of syntactic information but does not explore their combinations, citing cost saving.Source: §6 · p. 10
  • Manual label orders are not investigated on general-domain datasets, also for cost-saving reasons.Source: §6 · p. 10

A Statistics of PowerPlant Datasets

The appendix identifies tables reporting overall and classwise statistics for the PowerPlant datasets.

A Statistics of PowerPlant Datasets

B Statistics of Errors on PowerPlantFlat

The appendix describes error-type counts for three methods and percentages under the vanilla setting on PowerPlantFlat.

B Statistics of Errors on PowerPlantFlat

C Performance Under Vanilla Setting

Under the vanilla setting, reasoning techniques bring limited or negative benefits on general-domain data but improve out-of-distribution datasets. The section also describes syntactic-augmentation adaptations and evaluation under a 3-shot setting.

C Performance Under Vanilla Setting

  • Under the standard setting, syntactic information is not fully exploited, whereas decomposed QA effectively uses it.The paper points to Table 2 and Figure 3 for the decomposed-QA result.Source: C Performance Under Vanilla Setting · p. 13
  • Reasoning techniques improve out-of-distribution datasets but bring limited benefits, and sometimes hurt performance, on general-domain datasets under the standard setting.The authors suggest the out-of-distribution data is more challenging for ChatGPT and that reasoning helps it better understand that data.Source: C Performance Under Vanilla Setting · p. 13

D Syntactic Augmentation Under Few-shot Setting

  • For the 3-shot setting, syntactic prompting asks the model to analyze test-sample syntax before entity recognition, while tool augmentation supplies syntactic information for demonstrations and test samples.Table 10 reports the syntactic-augmentation experiment results under this setting.Source: D Syntactic Augmentation Under… · p. 13

E Evaluation on More Datasets

Evaluation on additional datasets finds mixed outcomes: reasoning techniques can hurt on two general-domain datasets, while biomedical datasets improve. Across the expanded evaluation, the methods improve results on ten of twelve benchmarks.

E Evaluation on More Datasets

  • The proposed methods are additionally evaluated on more datasets: general-domain CoNLL-2003 and WNUT-17, and biomedical-domain BC5CDR, BioNLP11, and CRAFT.Source: E Evaluation on More Datasets · p. 13
  • Reasoning techniques do not guarantee improvements on CoNLL-2003 and WNUT-17 and can hurt performance; the authors conjecture unsuitable label logic and noisy syntactic information contribute.They also conjecture that these datasets' shorter texts make the techniques harder to leverage.Source: E Evaluation on More Datasets · p. 13
  • The proposed methods achieve significant improvements on biomedical datasets BC5CDR, BioNLP11, and CRAFT, and improvements on ten of twelve evaluated benchmarks overall.The authors state that the results show improvements on challenging domains beyond the electric power domain.Source: E Evaluation on More Datasets · p. 13

F Evaluation on Other LLMs

The methods improve results across evaluated LLMs, including Llama2 13B. The paper also reports that expert-provided label orders perform better on power-plant datasets and supplies label-order instructions and prompt examples.

F Evaluation on Other LLMs

  • On Llama2 13B, the proposed strategies yield 19.72% F1 improvements on ACE05 and 17.51% F1 improvements on BC5CDR; the authors report consistent improvements across different LLMs.Table 12 gives results for GPT-3, GPT-3.5, and Llama2; its F1 results average three samples of 300 test examples.Source: F Evaluation on Other LLMs · p. 13, Table 12 · p. 16

G Label Order

  • On power-plant datasets, manual label orders from domain experts achieve significantly better results, and the paper suggests domain knowledge may boost performance.The paper also provides label orders for additional datasets and instructions for eliciting orders for PowerPlantFlat and ACE05.Source: G Label Order · p. 13, Table 13 · p. 17, Table 14 · p. 18

H Prompts

Loading 2310.10035v1…