PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Sentence Argument Linking

Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, Benjamin Van Durme

arXiv:1911.03766v3cs.CL

TL;DR

Sentence-level annotation leaves document-level event arguments incompletely represented, while existing cross-sentence datasets are too small for neural modeling. The paper introduces a span-selection model for argument linking and constructs RAMS, a 9,124-event resource covering 139 event types. The model achieves strong performance on RAMS and related datasets, while its performance is constrained when spans must be selected from sparsely annotated candidates.

  • Problem

    Existing datasets predominantly annotate sentence-level arguments, while cross-sentence argument-linking resources are too small for developing neural models.

  • Method

    The paper formulates document-level event understanding as argument linking and builds a span-selection model, supported by the RAMS dataset.

  • Results

    The model achieves 68.3 F1 on RAMS and 73.3 F1 when event types are known, outperforming strong baselines and transferring to related datasets.

  • Takeaways & Limitations

    RAMS provides broad multi-sentence supervision, and the model demonstrates adaptability across RAMS and several related event-based datasets.

  • Takeaways & Limitations

    Performance is limited when the model must select argument spans from all enumerated spans because annotated spans are sparse relative to that candidate set.

Abstract

from arXiv · show

We present a novel document-level model for finding argument spans that fill an event's roles, connecting related ideas in sentence-level semantic role labeling and coreference resolution. Because existing datasets for cross-sentence linking are small, development of our neural model is supported through the creation of a new resource, Roles Across Multiple Sentences (RAMS), which contains 9,124 annotated events across 139 types. We demonstrate strong performance of our model on RAMS and other event-related datasets.

1 Introduction

The paper frames document-level event understanding as linking event roles to explicit argument mentions across sentences, addressing the limits of sentence-level annotation. It introduces RAMS and a span-selection model that performs strongly across event-related datasets.

  • Motivation: Document-level event understanding remains incomplete because existing large-scale datasets predominantly annotate events and arguments only within individual sentences.The paper argues that even perfect sentence-level semantic role labeling would not fully capture an event described across a document.
  • Task formulation: Argument linking identifies explicit text spans that fill the latent roles evoked by an event trigger.The task connects an event trigger and its role structure to explicit mentions anywhere in the document context.
  • Dataset contribution: RAMS contains 9,124 annotated events across 139 event types, providing broader data for developing neural models of cross-sentence argument linking.The dataset annotates the closest argument span for each role within a 5-sentence window around each trigger.
  • Results: 68.3 F1 is achieved on RAMS, rising to 73.3 F1 when event types are known, and the model outperforms strong baselines.The model is also applied to several other event-based datasets and RAMS is shown useful for pre-training.
  • Contributions: The paper contributes a document-level argument-linking model and a large, broadly covered dataset for advancing automatic event understanding.Experiments emphasize adaptability across multiple datasets.

2 Non-local Arguments

Prior work establishes that event arguments frequently cross sentence boundaries, but available datasets for non-local arguments are small or narrowly covered. RAMS is positioned against these limitations and distinguishes reified event linking from entity-level argument extraction.

  • Data scarcity: Existing non-local argument datasets are too small to support training neural models, unlike large sentence-level SRL resources.Prior datasets focus on either few predicate types or few documents.
  • Prior datasets: SemEval Task 10 contains 1,370 frame instantiations over 438 sentences, while BNB contains 1,247 examples covering 10 nominal predicate types.MS-AMR contains 293 documents with document-level AMR annotations.
  • AIDA-1 coverage: AIDA-1 annotates only AIDA-salient events and covers 88 of the ontology’s 139 event sub-subtypes in its text collection.RAMS uses the same ontology while covering all 139 types and substantially more examples.
  • Non-locality: 38.1% of AIDA-1 events have an argument outside the trigger’s sentence, confirming that non-local arguments are common across datasets.Gerber and Chai also reported a 71% relative increase in role coverage when non-local arguments were added to NomBank annotations.
  • Task distinction: The paper targets links between explicit event mentions and argument mentions, rather than entity-level arguments represented as coreference clusters.This distinguishes the task from the TAC KBP event argument extraction formulation.

3 RAMS

RAMS is a crowdsourced dataset built from vetted news sentences and multi-sentence annotation windows under the AIDA ontology. Its protocol supports broad event coverage, while agreement measurements quantify annotation quality and remaining boundary variation.

  • Dataset description: RAMS contains 9,124 events annotated under the AIDA ontology, with typed trigger spans and role-typed argument spans represented by token offsets.Each example may contain zero or more argument spans in an English document.
  • Annotation context: Annotators may select arguments from any sentence in a five-sentence context window surrounding the event trigger.The window contains two preceding and two following sentences when available.
  • Data collection: Candidate sentences are generated by lexical-unit matching, crowdsourced vetting, and factuality judgments before multi-sentence argument annotation.Approximately 94,000 candidates were produced, 17,500 were vetted, and 52% satisfied the event and factuality constraints, yielding 9,124 trigger sentences.
  • Agreement: 55.3% of annotator span pairs match exactly, increasing to 69.9% when boundaries differing by one token are allowed.At least one boundary matches for 78.0% of pairs; end boundaries agree more often than start boundaries.
  • Related protocols: The protocol differs from prior crowdsourcing by showing two preceding and two following sentences and requiring annotators to highlight mention spans directly in the text.Feizabadi and Padó instead used a window of the predicate sentence plus three preceding sentences and templatic gap filling.

4 Model

The model links event-role representations to explicit argument spans across documents using pruned span selection and role-aware scoring. It supports multiple decoding strategies and achieves competitive sentence-level SRL performance, while document-level pruning can harm recall.

  • Task formulation: Argument linking finds text spans that fill an event’s roles, treating roles as implicit arguments linked to explicit document mentions.The formulation searches for (role, argument) pairs for each event, allowing no explicit argument when appropriate.
  • Representations: Contextualized span representations, trigger representations, and learned role embeddings are combined to score candidate arguments for each event-role pair.Candidate spans are formed from contextualized embeddings and then scored alongside the trigger and role representations.
  • Pruning: O(n^2) possible document spans are reduced by restricting span width and retaining the top λ_A n candidates under a learned unary score.The model then further prunes candidates in relation to each event before event-role scoring.
  • Scoring: The link score combines event-role and argument-role scores with direct candidate–event-role interactions and feature-based components.Features include bucketed token distance, while the model assigns no explicit argument a score of zero as a threshold.
  • Decoding: The model evaluates argmax, greedy, and type-constrained decoding, with type-constrained decoding retaining only the top ontology-permitted arguments for each role.Greedy decoding can use the no-argument probability as a threshold for selecting multiple non-overlapping arguments.
  • Evaluation: 81.4 F1 with gold predicates and 81.2 F1 without gold predicates are achieved on CoNLL 2012 SRL, below stronger reported baselines because document-level pruning harms recall.The model is evaluated on sentence-level SRL as a sanity check despite being designed for document-level linking.

5 RAMS Experiments and Results

The RAMS experiments evaluate document-level argument linking under different decoding, encoder, and scoring choices, including distance, role confusion, and qualitative examples. The model benefits from type constraints and BERT-based representations, while performance remains affected by span-candidate sparsity.

  • Experimental setup: The model predicts trigger-role-argument triples using gold trigger and argument spans, with greedy, argmax, and type-constrained decoding evaluated.Type-constrained decoding uses known event types to filter illegal role predictions.
  • Limitations: The main span-selection setting assumes gold argument spans, while performance is hypothesized to drop when selecting from all spans because annotated spans are sparse.The unpruned setting considers spans up to five tokens long and changes pruning-related hyperparameters.
  • Decoding: Type-constrained decoding generally improves F1 by removing ontology-invalid predictions, although filtering can reduce recall when the correct argument receives a lower score.The model may be more confident in an incorrect argument for a role, causing the correct one to be filtered out.
  • Scoring components: The event-role-to-argument compatibility score is the most important link-score component because it directly models compatibility between explicit and implicit arguments.This component corresponds to sl in the link score.
  • Encoders: BERT generally improves over ELMo, and layers 9–12 often outperform layers 6–9.These comparisons are reported across the tested contextualized encoders.
  • Role analysis: Role errors occur more often between highly correlated roles, with mistakes tending toward the more frequent role.The role analysis compares embedding similarity with row-normalized confusion under argmax decoding.
  • Qualitative analysis: The model links non-local arguments before triggers and can predict multiple arguments for one role under greedy decoding.A qualitative example includes the rare driverpassenger role with 17 training instances.
  • AIDA-1 transfer: RAMS pre-training improves AIDA-1 performance from 19.1 F1 on English AIDA-1 Eval with greedy decoding to 24.4 F1 after fine-tuning.With TCD, the corresponding scores are 18.2 F1 and 24.8 F1.

6 Other Datasets

The model is also evaluated on BNB and GVDB-style document-level extraction settings. GVDB results use broader document-level slot coverage, but direct comparison with prior work is limited by differing splits and evaluation conditions.

  • BNB: BNB is adapted by merging split arguments, retaining only the closest acceptable filler, and modifying data splits to prevent document leakage.The revised splits also provide a development set for hyperparameter tuning.
  • GVDB: GVDB evaluation uses 5,056 training articles, 400 development articles, and 500 test articles from a gun-violence news corpus.The corpus is split chronologically.
  • GVDB: The GVDB model predicts one argument value per slot for a document-level event and uses argmax decoding because each slot has exactly one value.The event is triggered by the full document under the stated assumption.
  • GVDB results: Document-level predictions cover more attributes than the sentence-level baseline, but the reported values are not directly comparable because the datasets, splits, and role combinations differ.The authors nevertheless report the model as suitable for slot-filling information extraction.

7 Conclusion

The paper introduces a document-level argument-linking model and RAMS, a dataset of 9,124 events covering 139 event types. The model outperforms strong RAMS baselines and transfers across related datasets.

  • Conclusion: RAMS contains 9,124 events covering 139 event types and supports neural training for document-level argument linking.The dataset addresses the small amount of existing data for the task.
  • Conclusion: The model outperforms strong baselines on RAMS and is applicable to a variety of related datasets.The authors hope RAMS stimulates further work on multi-sentence argument linking.

A RAMS Data

RAMS combines multi-sentence event annotation with broad coverage of event and role types, while measuring annotator agreement and annotation completeness. Its annotation process selects the closest argument span for each role within a five-sentence context window.

  • Annotation procedure: Annotators judge whether highlighted lexical units evoke the defined event type and assess sentence factuality before selecting arguments.The vetting task evaluates event-evoking lexical units in context.
  • Annotation procedure: RAMS presents annotators with five tokenized sentences around a trigger and asks them to select the closest span for each present role.Annotators use event and role definitions while considering arguments anywhere in the context window.
  • Dataset coverage: RAMS annotations cover a more diverse and less skewed set of event and role types than AIDA-1.More event types and role types are needed to cover a given percentage of RAMS than AIDA-1.
  • Dataset coverage: RAMS training-set role coverage has 25th-, 50th-, and 75th-percentile values of 55.6%, 61.9%, and 68.6%, respectively.Role coverage divides average filled roles per event instance by the ontology’s specified role count.

B RAMS Hyperparameters

The RAMS experiments use span representations and distance-aware scoring, with multiple decoding and data conditions evaluated across related event datasets. The section also documents dataset filtering, model settings, and error-analysis procedures.

  • Model configuration: The model uses learned bucketed distance embeddings and combines candidate-argument and linking scores while omitting boundary scores when spans are given.The selected features resulted from a sweep over possible scoring components.
  • Model configuration: The implementation uses span embeddings with character convolutions, 300-dimensional GloVe vectors, 0.2 dropout, Adam, and patience-based early stopping.The best checkpoint was reached after 19 epochs using F1 for evaluation.
  • Decoding and evaluation: Experiments compare argmax, greedy, and ontology-aware type-constrained decoding, including settings with distractor candidates or all spans up to five tokens.Type-constrained decoding retains the top-scoring number of arguments permitted for each role.
  • Error analysis: The analysis visualizes role-embedding similarity and model errors, aligning correct and predicted role sets before computing remaining errors.This handles cases where one argument span can receive multiple roles for the same event.
  • Decoding and evaluation: AIDA-1 is filtered to English documents with textual triggers and split into 46 training, 17 development, and 69 test documents containing 389 training events.The development and test topics differ from the training topics to emulate topic-distribution mismatch.

D.3 Pre-training on RAMS

Pre-training on RAMS is evaluated for transfer to AIDA-1, where distractor links and ontology mismatches affect decoding behavior. Type-constrained decoding can improve precision but may reduce recall and, without pre-training, lower evaluation performance.

  • Transfer setting: AIDA-1 contains distractor negative links because documents may contain multiple triggers, unlike RAMS examples with one trigger.The model must learn that an argument for one event may not participate in another event in the same document.
  • Decoding effects: Type-constrained decoding can delete correct predictions beyond ontology multiplicity limits, hurting recall when AIDA-1 annotates multiple attackers or targets.AIDA-1 often does not adhere to the ontology’s multiplicity constraints.
  • Decoding effects: Type-constrained decoding hurts AIDA-1 Eval performance without RAMS pre-training, despite its usual precision increase and recall decrease.The reported net effect is a lower F1 score in this condition.

E.1 Data Processing

The comparison data are normalized to the model’s contiguous-span and one-argument-per-role assumptions before document-level splitting. These transformations support consistent processing but constrain comparison with prior work.

  • Data normalization: Gerber and Chai (2012) annotations are aligned to Penn Treebank text, with split arguments manually merged or replaced by their maximal span.The model assumes that arguments are contiguous spans.
  • Data normalization: Argument clusters are reduced to singleton sets containing the span closest to the trigger because the training objective assumes one argument per role.This reformulation limits direct comparison with prior work that gives full credit for finding any cluster member.
  • Data splitting: Examples are assigned by document to 80% train, 10% development, and 10% test splits to avoid information leakage across documents.Documents, rather than individual instances, are bucketed before random assignment.

F GVDB Hyperparameters and Additional Results

The GVDB evaluation uses filtered, sentence-bounded articles and compares BERT- and ELMo-based document encoders across all annotated span fields. BERT generally outperforms ELMo, though the results are not directly comparable to the prior baseline.

  • Data preparation: GVDB preprocessing excludes unreliable articles, separates development and test sets with a 100-article buffer, and removes spans crossing sentence boundaries.The corpus contains 7,366 articles before these filters.
  • Model configuration: The experiments use BERT-base and ELMo contextualized encoders with dropout, Adam optimization, and patience-based early stopping.The best checkpoint was reached after 8 epochs using BERT-base settings described in the passage.
  • Additional evaluation: The model predicts all annotated span-based fields, extending beyond the narrower field subset evaluated by Pavlick et al.The annotated fields include VICTIM, SHOOTER, LOCATION, TIME, and WEAPON attributes.
  • Results: BERT generally improves over ELMo across GVDB fields, but the margin is not sizeable.Table 12 reports BERT results and Table 13 reports ELMo results; the different splits and evaluation conditions prevent direct comparison with the prior baseline.
Loading 1911.03766v3…