PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction

Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, Jing Shao

arXiv:2202.12109v2cs.CLcs.AI

TL;DR

EAE remains challenging, particularly when extraction must work across sentence and document levels with few training examples. PAIE uses extractive prompt tuning with role-specific span selectors, joint prompts, and bipartite matching, and reports average F1 gains across three benchmarks.

  • Problem

    Event Argument Extraction remains the main challenge after event detection, with practical settings requiring sentence- and document-level performance under limited training data.

  • Method

    PAIE jointly extracts role-specific argument spans through extractive prompts, span selectors, multi-role interactions, and bipartite matching for multiple arguments.

  • Results

    3.5% and 2.3% average absolute F1 gains were reported across three benchmarks for PAIE-base and PAIE-large, respectively.

  • Takeaways & Limitations

    PAIE is reported to be effective and efficient for sentence- and document-level EAE and to generalize to few-shot settings.

  • Takeaways & Limitations

    The paper identifies future work in using co-reference as an auxiliary EAE task and adding entity information to better determine argument boundaries.

Abstract

from arXiv · show

In this paper, we propose an effective yet efficient model PAIE for both sentence-level and document-level Event Argument Extraction (EAE), which also generalizes well when there is a lack of training data. On the one hand, PAIE utilizes prompt tuning for extractive objectives to take the best advantages of Pre-trained Language Models (PLMs). It introduces two span selectors based on the prompt to select start/end tokens among input texts for each role. On the other hand, it captures argument interactions via multi-role prompts and conducts joint optimization with optimal span assignments via a bipartite matching loss. Also, with a flexible prompt design, PAIE can extract multiple arguments with the same role instead of conventional heuristic threshold tuning. We have conducted extensive experiments on three benchmarks, including both sentence- and document-level EAE. The results present promising improvements from PAIE (3.5\% and 2.3\% F1 gains in average on three benchmarks, for PAIE-base and PAIE-large respectively). Further analysis demonstrates the efficiency, generalization to few-shot settings, and effectiveness of different extractive prompt tuning strategies. Our code is available at https://github.com/mayubo2333/PAIE.

1 Introduction

Event Argument Extraction remains the main challenge after event detection, especially when models must handle sentence- and document-level inputs with limited training data. PAIE addresses these demands through extractive prompt tuning, joint role interactions, and optimal span assignment, achieving reported gains across three benchmarks.

  • Event Argument Extraction is identified as the main challenge after recent progress in event detection.
  • Real-world EAE requires effective and efficient extraction at sentence and document levels, including few-shot settings without sufficient training data.
  • PAIE uses role prompts and two span selectors to jointly select argument spans while capturing interactions among argument roles.
  • PAIE supports multiple arguments with the same role through flexible prompt design and bipartite matching-based joint optimization.
  • 3.5% and 2.3% F1 gains were obtained on average across three benchmarks for PAIE-base and PAIE-large, respectively.

2 Related Works

Prior EAE research uses candidate-role classification, extractive QA, and generative formulations. PAIE instead prompts argument interactions and applies bipartite matching to improve multiple-argument detection and long-distance dependency modeling.

  • Traditional EAE methods identify candidate spans and then classify their roles, which can expose joint models to candidate-dependent error propagation.
  • Extractive QA reformulates EAE with question templates, leveraging pretrained language-model knowledge and enabling role-interaction constraints.
  • Generative approaches convert EAE into text-to-text or sequential generation and can extract arguments in a single pass.
  • PAIE differs by prompting argument interactions and optimizing multiple-argument detection with bipartite matching loss.
  • PAIE uses prompt tuning for extraction with a linear head, distinguishing it from prompt methods designed for classification or generation.

3 Methodology

PAIE formulates event argument extraction as prompt-based span selection, using joint role prompts to generate selectors that extract arguments simultaneously. It supports multiple arguments per role through flexible slots and optimizes their assignments with bipartite matching.

  • Prompted Span Selection: Two selectors score start and end tokens in the context to identify each role-specific argument span.The extractive formulation selects span boundaries directly rather than generating candidate text.
  • Prompt Creation: PAIE treats EAE as a prompt-based span extraction problem, creating joint prompts containing all event-specific roles.Each prompt is generated for an event type and its role set.
  • Prompt Creation: Role mentions act as prompt slots, enabling interactions among different roles and multiple slots for roles with multiple arguments.The number of slots for a role is heuristically based on its maximum argument count in the training data.
  • Role-specific Selector Generation: BART encodes the context and decodes the prompt separately, producing role-specific span selectors from prompt-derived role features.The prompt and context interact through decoder cross-attention; each role feature can yield a corresponding selector.
  • Prompt Creation: PAIE supports manual, soft, and concatenation prompt templates for representing event roles.Soft prompts connect roles with learnable role-specific pseudo tokens, while manual templates use natural language and concatenation joins role names.
  • Learning with Prompted Span Selector: Bipartite matching assigns predictions to ground-truth arguments with the least-cost global matching across multiple same-role slots.The model uses Hungarian matching to optimize assignments between slot predictions and ground-truth spans.

4 Experiments

PAIE is evaluated against established baselines on three EAE benchmarks using Arg-I and Arg-C metrics. It performs best across datasets, with gains for both sentence- and document-level extraction and improvements from key ablations.

  • Experimental Setup: Experiments cover RAMS, WIKIEVENTS, and ACE05, spanning document-level and sentence-level EAE.RAMS and WIKIEVENTS are document-level benchmarks, while ACE05 evaluates sentence-level EAE.
  • Overall Performance: PAIE performs best on all three datasets compared with the evaluated baselines.Baselines include classification, generation, and QA-based models.
  • Overall Performance: 2.1% absolute Arg-C improvement is achieved by the base model on ACE05 for sentence-level EAE.For document-level EAE, the base model gains 2.1% on RAMS and 6.3% on WIKIEVENTS.
  • Overall Performance: 3.5% and 2.9% Arg-C gains are achieved by the large model on RAMS and WIKIEVENTS, respectively.These results support generalization across varying context lengths.
  • Ablation Study: Role-specific selectors improve Arg-C on RAMS and WIKIEVENTS but slightly reduce it on ACE05.The authors relate this pattern to document length and the number of role types.
  • Ablation Study: Joint multi-argument prompts consistently improve all three datasets, while bipartite matching contributes an average 0.7% improvement.The authors associate joint prompts with implicit argument interaction and matching with permutation invariance.

5 Evaluation of Extractive Prompting

PAIE evaluates separate context-prompt encoding, alternative PLMs, and prompt designs for extractive EAE. Joint prompts and separate encoding are supported by the reported analyses, while performance is examined across argument-trigger distances.

  • 5.1 Architecture Variants: Concatenating context and prompt slightly impairs model performance compared with feeding them separately.The authors attribute this to over-interaction and reduced encoder input capacity for documents.
  • 5.1 Architecture Variants: PAIE compares Arg-C F1 across PLMs, including BERT and BART, with BERT also tested under the encoder-only PAIEE setting.In the table notation, BE and BA denote BERT and BART.
  • 5.2 Prompt Variants: Three joint prompts and one single-role template are compared: manual, concatenation, soft, and single templates.The comparison uses Arg-C F1 across three benchmarks.
  • 5.2 Prompt Variants: All three joint prompts outperform the single template, supporting joint prompting for argument extraction.Figure 3 reports Arg-C F1 for the four prompt types across the benchmarks.
  • 5.2 Prompt Variants: Manual templates are usually more stable and better than other prompt types, while soft prompts achieve comparable results.Soft prompts also reduce the need for laborious handcrafted template creation.
  • 5.2 Prompt Variants: The distance breakdown reports Arg-C F1 on the RAMS development set by argument-trigger distance and argument count.The analysis focuses on long-distance arguments, including arguments occurring after the trigger.
  • 5.2 Prompt Variants: Joint prompts are presented as leveraging implicit interaction among arguments in the distance analysis.The reported analysis connects joint prompting with long-range argument extraction.

6 Analysis on Real Scenario

PAIE is evaluated in realistic settings involving long-distance arguments, repeated same-role arguments, limited training data, and extraction efficiency. Across these settings, it improves performance while avoiding threshold tuning and accelerating inference.

  • 6.1 Long-range Dependencies: PAIE improves extraction of long-distance arguments, especially arguments occurring after the trigger, through joint prompts that model role interactions.Roles conditioning on each other lowers the difficulty of extracting long-distance arguments.
  • 6.2 Same-role Argument Assignment: PAIE handles multiple arguments sharing one role more effectively and efficiently than QA-based models.Its flexible prompting uses multiple slots for repeated roles, avoiding conventional threshold tuning.
  • 6.3 Few-shot Setting: PAIE remains superior across almost all datasets and training-data ratios, achieving comparable F1 with EEQA-BART at about 20% and EEQA at about 10% of training samples.The gains over baselines become larger as the amount of training data decreases.
  • Efficiency: PAIE usually runs 3–4 times faster than EEQA because it predicts multiple roles simultaneously and avoids heavy threshold tuning.The inference-time comparison uses the ACE05, RAMS, and WIKIEVENTS test sets on one NVIDIA-1080Ti GPU.

7 Conclusion

PAIE is presented as an effective and efficient prompt-tuning model for event argument extraction at both sentence and document levels. Experiments and ablations support its effectiveness, generalization, and behavior across practical scenarios, while future work targets coreference and entity information.

  • 7 Conclusion: PAIE extracts event arguments effectively and efficiently at both sentence and document levels.The model uses role-specific selectors and joint prompts to obtain role knowledge from pretrained language models.
  • 7 Conclusion: Experiments on three standard benchmarks demonstrate PAIE’s effectiveness and generalization across sentence-level and document-level event argument extraction.The paper also reports ablations covering main components, extractive prompting, and several real scenarios.
  • 7 Conclusion: Future work will investigate coreference as an auxiliary task and incorporate entity information to better determine argument boundaries.These directions are stated as extensions of the current approach.

A.1 Dataset statistics

The study evaluates EAE across ACE05, RAMS, and WIKIEVENTS, covering sentence-level and document-level settings. It compares PAIE with joint, generative, and QA-based baselines under the datasets’ differing annotation conditions.

  • Datasets: The experiments use ACE05, RAMS, and WIKIEVENTS as three common EAE benchmarks.ACE05 is used for sentence-level EAE, while RAMS and WIKIEVENTS are document-level benchmarks.
  • Dataset statistics: ACE05 provides English event annotations for sentence-level EAE, with 33 event types, 22 argument roles, and 4,859 training arguments.Its development and test sets contain 605 and 576 arguments, respectively.
  • Dataset statistics: WIKIEVENTS contains 246 documents, 50 event types, and 59 argument roles, with documents drawn from English Wikipedia and related news articles.The experiments use conventional argument annotations rather than its coreference-link annotations.
  • Baseline models: The baseline set includes ONEIE, BART-Gen, EEQA, FEAE, DocMRC, and EEQA-BART.These represent joint extraction, generation, QA-based, and pretrained-model variants.
  • Baseline models: ONEIE is not reported on RAMS or WIKIEVENTS because those datasets lack entity or relation annotations required for a fair comparison.Removing ONEIE’s entity and relation modules produces abnormally low performance on those datasets.

A.3 PAIE implementation and training setup

PAIE is trained with pretrained BART encoders and decoders, document windows centered on triggers, and role-specific span prediction. Bipartite matching aligns predicted and gold spans during training, while inference outputs non-zero predicted spans.

  • Implementation and training: PAIE initializes its encoder-decoder architecture with pretrained BART models and restricts document-level inputs to trigger-centered windows.The window is used when documents exceed the BART encoder constraint or consume excessive memory.
  • Training procedure: The training algorithm encodes context and prompts, decodes role slots, and collects predicted spans for each role.Role-specific outputs are formed from predicted start and end positions.
  • Bipartite matching: For each role, PAIE represents gold and predicted arguments as start–end spans and matches them using the lowest-cost permutation.The matching uses the L1 distance between gold and predicted span boundaries.
  • Bipartite matching: The Hungarian algorithm provides efficient optimal assignment, padding the gold set with (0, 0) when predictions outnumber gold spans.When gold spans outnumber candidates, only optimally matched gold spans are used for the matching loss.
  • Training and inference: Bipartite matching is applied during training, whereas inference outputs all non-zero spans with their corresponding argument roles.The matched prediction logits are used for start/end probability calculation during training.

A.5 Further analysis of Bipartite Matching

PAIE’s bipartite matching loss reassigns predictions across joint prompt slots to optimize multiple arguments, especially when their distinctions depend on syntax rather than sequence. Its contribution is difficult to measure robustly because existing benchmarks contain few genuinely complex multi-argument cases.

  • Matching mechanism: Bipartite matching assigns predicted arguments to prompt slots through a least-cost, permutation-invariant global optimization.This handles multiple arguments of the same role while accounting for different semantic meanings and matching preferences across slots.
  • Matching mechanism: The matching objective is most useful when arguments have subtle distinctions expressed by syntax rather than sequential order.Passive-versus-active voice provides the motivating example: “Peter is taught by John” differs structurally from the prompt “Person teaches Person.”
  • Empirical analysis: Ablations validate bipartite matching loss, but its improvement is not consistently significant relative to PAIE’s other contributions.The authors attribute this partly to limited benchmark coverage of multi-argument examples and their generally simple enumeration.
  • Empirical analysis: Only 8.9% of ACE05, 6.1% of RAMS, and 10.9% of WIKIEVENTS samples involve multiple arguments.The authors therefore expect larger datasets with more multi-arguments and diverse narrative styles to reveal stronger gains.
  • Prompt analysis: Tables 10–12 present benchmark examples, prompt comparisons, and manual templates with repetitive slots for same-role multi-arguments.The prompt comparison covers standard QA, generation, and PAIE’s three joint-prompt types; manual templates are additionally listed for each dataset.
Loading 2202.12109v2…