Exploring the Feasibility of ChatGPT for Event Extraction

1 Introduction

• Limited annotated data makes event extraction challenging because expert annotation can be expensive, while LLMs have drawn interest for language tasks requiring few demonstrations.Source: §1 · p. 1
• Event extraction requires schemas specifying event types, triggers, and arguments, as well as handling complex special cases.Source: §1 · pp. 1–2
• ChatGPT averaged only 51.04% of EEQA’s performance in long-tail and complex scenarios.The paper compares ChatGPT with task-specific models in texts containing multiple events and long-tail cases.Source: §1 · p. 2
• Several prompt optimizations produced no consistent upward performance trend, and the authors found ChatGPT insufficiently robust for a good user experience.The paper also reports significant result variation among users using different prompt styles.Source: §1 · p. 2

2 Background

• The paper uses the ACE 2005 definition of an event as a specific occurrence involving participants and arguments with different roles.Source: §2 · p. 2
• An event mention is a sentence describing an event; its trigger is the clearest event-expressing word, and arguments are entities involved in it.In the example, “threw” is the ATTACK trigger, with demonstrators as attackers and stones and bottles as instruments.Source: §2 · p. 2
• Event detection identifies and classifies event types, while event argument extraction finds words, phrases, and entities filling event roles; this work focuses mainly on detection.Source: §2 · p. 2

3 ChatGPT for Event Extraction

• The proposed zero-shot approach frames event extraction as multi-turn question answering, without fine-tuning ChatGPT.It builds on prior work and asks a sequence of conversational questions to identify events.Source: §3 · p. 2
• For event detection, each event is serialized as a JSON record containing a trigger word and event type, using instructions with task descriptions, event definitions, and demonstrations.Source: §3 · p. 2
• The evaluation randomly selects 20 raw test samples, then uses the same samples to analyze how prompt elements affect performance.Source: §3 · pp. 2–3
• The study also compares ChatGPT with task-specific models on long-tail and multi-event texts and conducts usability testing.Source: §3 · p. 3

4.1 Setup

• Experiments used ACE05 with the prior-work data split and preprocessing, comparing ChatGPT against Text2Event and EEQA using trigger identification and classification criteria.Text2Event frames extraction as sequence-to-sequence generation; the EEQA baseline used its BERT-based trigger-detection model.Source: §4.1 · p. 3

4.2 Results

• EEQA achieved the highest F1; ChatGPT trailed both task-specific models, with recall comparable to Text2Event (T5-base) but significantly lower precision.The comparison used 20 randomly selected ACE05 test samples.Source: Table 1 · p. 3, §4.2 · p. 3
• 11.95 F1 points were lost without event definitions and 6.17 without a positive example; removing the negative example improved performance.The authors speculate that ChatGPT may have misinterpreted the negative sample as positive.Source: Table 2 · p. 3, §4.2 · p. 4
• ChatGPT reached only 49.2% of EEQA’s performance on low-frequency events and 56.7% on high-frequency events.The authors report no significant performance difference between ChatGPT’s high- and low-frequency subsets.Source: Table 3 · p. 4, §4.2 · p. 4
• ChatGPT performed significantly worse than EEQA and Text2Event on both simple and complex samples.Simple samples contained one event; complex samples contained more than two.Source: Table 3 · p. 4, §4.2 · p. 4
• ChatGPT produced structured events for all 20 samples with the full prompt or without positive examples, but only 19 of 20 without definitions and negative examples.In the remaining case, it explained that it could not recognize the event trigger or type rather than returning a structured event.Source: §4.2 · p. 4
• Four annotators tried five prompts on ten ACE05 samples; performance showed no consistent upward trend and varied with prompt style.Reported F1 scores ranged from 26.92 to 66.67.Source: Figure 2 · p. 5, §4.2 · p. 5

• ChatGPT performs well in simple scenarios but struggles to match task-specific models in complex and long-tail scenarios.Source: §5 · p. 5
• ChatGPT’s performance is highly dependent on prompt style, producing significant result variations across users.Source: §5 · p. 5
• The authors call for further research to refine ChatGPT’s event-extraction capabilities and overcome the challenges it presents.Source: §5 · p. 5