PaperlayerView exact revision on arXiv

Source-linked AI summary

EIGEN: Event Influence GENeration using Pre-trained Language Models

Aman Madaan, Dheeraj Rajagopal, Yiming Yang, Abhilasha Ravichander, Eduard Hovy, Shrimai Prabhumoye

arXiv:2010.11764v1cs.CL

TL;DR

Reasoning about event influences is important for understanding processes, yet generating novel influences for unseen contexts remains an open challenge. EIGEN derives a conditioned event-influence dataset and fine-tunes a pretrained language model to generate influences. It outperforms strong baselines in automated and human evaluations, and its generated influences improve downstream QA performance.

  • Problem

    Generating novel event influences for unseen contexts remains an open challenge, despite the importance of implicit event reasoning for process understanding and downstream tasks.

    Source: §1 · p. 1

  • Method

    EIGEN derives a WIQA-based corpus and fine-tunes a pretrained language model to generate event influences conditioned on context, relation, and reasoning-chain distance.

    Source: §1 · pp. 1–2

  • Results

    Approximately 10 ROUGE improvement is reported alongside stronger human judgments of relevance and proximity to reference text, and generated influences improve downstream WIQA QA performance.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Generated influences can be added to a downstream QA task without modifying the underlying model architecture.

    Source: §1 · p. 2

    The dataset uses the relationship types in Table 1, although the framework makes no relation-specific assumptions and is described as applicable to a broader range of relationships.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Reasoning about events and tracking their influences is fundamental to understanding processes. In this paper, we present EIGEN - a method to leverage pre-trained language models to generate event influences conditioned on a context, nature of their influence, and the distance in a reasoning chain. We also derive a new dataset for research and evaluation of methods for event influence generation. EIGEN outperforms strong baselines both in terms of automated evaluation metrics (by 10 ROUGE points) and human judgments on closeness to reference and relevance of generations. Furthermore, we show that the event influences generated by EIGEN improve the performance on a "what-if" Question Answering (WIQA) benchmark (over 3% F1), especially for questions that require background knowledge and multi-hop reasoning.

1 Introduction

Reasoning about event influences matters for process understanding and downstream tasks, but generating novel influences in unseen contexts remains an open challenge. EIGEN derives a context-, relation-, and hop-conditioned dataset and framework, and reports stronger automated and human evaluation than strong baselines.

1 Introduction

  • Event influence reasoning is needed for process understanding and tasks including question answering, process tracking, qualitative-relation reasoning, and physical commonsense reasoning.The relevant influences are often implicit in text.Source: §1 · p. 1
  • Generating novel event influences for unseen contexts remains an open challenge because prior systems primarily extract events already mentioned in text.Such extractive approaches are limited for downstream tasks requiring implicit event reasoning.Source: §1 · p. 1
  • The work derives a large corpus from WIQA and proposes EIGEN to generate an event’s influences in forward and backward directions.Generation is conditioned on context, the relationship between events, and their distance in a reasoning chain.Source: §1 · pp. 1–2
  • The framework generates influence graphs with event nodes and positive or negative relations, while hop denotes the number of edges between events.The overview illustrates generation for “more sunlight” with helps, hurt-by, and related relation types across hops 1 and 2.Source: §1 · p. 2, Figure 1 · p. 2
  • EIGEN outperforms strong baselines by approximately 10 ROUGE and on human judgments of relevance and proximity to reference text.Generated influences can also be integrated into downstream QA without changing the underlying model architecture.Source: §1 · p. 2

3. We also validate our approach by augment-

Adding generated influences to a downstream WIQA QA dataset improves accuracy over the state of the art, with a larger reported gain on questions requiring implicit event-influence reasoning.

3. We also validate our approach by augment-

  • 3% improvement in overall accuracy over the state of the art is reported after adding generated influences to a downstream QA dataset.The benchmark is WIQA QA.Source: §3 · p. 2
  • 8% improvement is reported for questions requiring implicit event-influence reasoning.This is the subset result for the WIQA QA task.Source: §3 · p. 2
  • Adding generated influences to the downstream QA dataset improves performance on both the overall task and the implicit event-influence subset.The reported gains are 3% in overall accuracy and 8% on the subset, respectively.Source: §3 · p. 2

2 Related Work

Prior event-chain and influence work largely extracts explicit events or restricts the task, while WIQA broadens influence prediction to general procedures and multiple hops. EIGEN derives a generation dataset from WIQA and differs from COMET through broader context and hop conditioning and recursive graph generation.

2 Related Work

  • Event-chain approaches extract explicitly mentioned events, while work on recipes and procedural text has used restricted domains or event sets.The cited examples include unsupervised event chains, supervised event chains (Liu et al., 2018), recipes (Bosselut et al., 2018), and procedural text.Source: §2 · p. 2
  • WIQA extends influence study to general procedures by predicting positive, negative, or no-effect relations between events and annotating multiple reasoning hops.Its restricted task formulation limits direct use for event influence generation.Source: §2 · p. 2
  • EIGEN derives a large-scale event-influence dataset from WIQA to address the lack of a dataset for generation.The paper identifies WIQA’s task formulation as restricted for adapting to event influence generation.Source: §2 · pp. 2–3
  • Unlike COMET, EIGEN conditions generation on a larger context and hop information and supports recursively generating influence graphs.The paper also notes that a dataset for EIGEN’s task was not readily available, motivating adaptation of existing datasets.Source: §2 · p. 3

3 Event Influence Generation

EIGEN generates event influences from procedural context, conditioning on the source event, influence relation, and hop distance. The method uses WIQA influence graphs to build generation examples and trains a language model to generate target events.

3 Event Influence Generation

  • EIGEN generates fine-grained event influences conditioned on procedural context, relation, and hop information.The framework's section covers task formulation, dataset collection, and learning to generate influences.Source: §3 · p. 3

3.1 Task

  • Given a procedural passage, source event, relation, and hop length, the task is to generate the target event.The four relations are helps, hurts, helped-by, and hurt-by, capturing positive or negative influence in either direction.Source: §3.1 · p. 3

3.2 Dataset

  • WIQA influence graphs are decomposed into training examples, with each edge yielding passage, source-event, relation, and hop inputs paired with a target event.WIQA graphs capture event interactions and external perturbations; the paper notes their subjectivity alongside high inter-annotator agreement.Source: §3.2 · p. 3
  • Multi-hop samples use transitive composition of influence relations, while inverse-influence augmentation adds examples in the reverse direction.The dataset is described as covering varied relations and hop distances; Table 1 breaks sample counts down by relation, distance, and split, retaining WIQA's train-dev-test split.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3 Learning to Generate Influences

  • EIGEN concatenates the passage, source event, relation, and hop into a query and uses an autoregressive language model to generate the target-event sequence.The model is parameterized with GPT-2, and generation proceeds token by token until an end symbol is produced.Source: §3.3 · p. 4

Related papers

Loading 2010.11764v1…