PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-media Structured Common Space for Multimedia Event Extraction

Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang

arXiv:2005.02472v1cs.MMcs.CLcs.CVcs.LG

TL;DR

Single-modality event extraction does not capture arguments that appear in images but are absent from text. The paper introduces M2E2 and WASE, a weakly supervised approach that embeds structured representations from text and images in a common space. WASE extracts more event mentions than text-only baselines and outperforms the compared uni-modal and multimedia flat methods across tasks.

  • Problem

    Traditional event extraction targets one modality, although 33% of images in a random sample of 100 VOA multimedia news articles contained event arguments not mentioned in text.

    Source: §1 · p. 1

  • Method

    WASE embeds graph-structured event and entity representations from text and images in a common semantic space, using single-modal annotations and image-caption pairs for weakly supervised alignment.

    Source: §1 · pp. 1–2

  • Results

    21.4% more event mentions are extracted than by text-only baselines, and WASE significantly outperforms uni-modal methods and multimedia flat representations across event extraction and argument role labeling.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The 245-article annotated news benchmark supports evaluation of joint event and argument extraction across modalities.

    Source: §1 · pp. 1–2

    Visual argument localization is challenging: attention-based methods lack supervision for attention extraction, and attention can cover the whole image when an argument targets many instances.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

We introduce a new task, MultiMedia Event Extraction (M2E2), which aims to extract events and their arguments from multimedia documents. We develop the first benchmark and collect a dataset of 245 multimedia news articles with extensively annotated events and arguments. We propose a novel method, Weakly Aligned Structured Embedding (WASE), that encodes structured representations of semantic information from textual and visual data into a common embedding space. The structures are aligned across modalities by employing a weakly supervised training strategy, which enables exploiting available resources without explicit cross-media annotation. Compared to uni-modal state-of-the-art methods, our approach achieves 4.0% and 9.8% absolute F-score gains on text event argument role labeling and visual event extraction. Compared to state-of-the-art multimedia unstructured representations, we achieve 8.3% and 5.0% absolute F-score gains on multimedia event extraction and argument role labeling, respectively. By utilizing images, we extract 21.4% more event mentions than traditional text-only methods.

1 Introduction

The paper introduces multimedia event extraction because event arguments can appear in images without being mentioned in text. It presents a benchmark and a weakly supervised structured-embedding method, which outperforms the compared methods and extracts more event mentions than text-only baselines.

1 Introduction

  • 33% of images in a random sample of 100 VOA multimedia news articles contained visual objects serving as event arguments that were not mentioned in text.Traditional event extraction methods target a single modality, while contemporary journalism distributes news via multimedia.Source: §1 · p. 1
  • M2E2 is introduced as a task for jointly extracting events and arguments from multiple modalities, with a first benchmark comprising 245 fully annotated news articles.The benchmark is intended to support deep analysis of cross-media events.Source: §1 · pp. 1–2
  • WASE represents textual and visual semantic information in a structured multimedia embedding space, using graph-based neural networks.Its weakly supervised framework uses existing single-modal annotated corpora and enables joint inference without cross-modal annotation.Source: §1 · pp. 1–2
  • 21.4% more event mentions are extracted than by text-only baselines, and WASE significantly outperforms uni-modal methods and multimedia flat representations across event extraction and argument role labeling.Training and evaluation use heterogeneous datasets from multiple sources, domains, and modalities.Source: §1 · p. 2

2 Task Definition

M2E2 formulates event and argument extraction over text and images, allowing mentions and arguments to be grounded in either or both modalities. Its benchmark maps event ontologies and annotates 245 multimedia news articles.

2.1 Problem Formulation

  • Events may be grounded in a text trigger, an image, or both, with textual and visual mentions treated as one event when they refer to the same event.Arguments are grounded in text entities, image objects, or both; arguments are merged when they refer to the same real-world event.Source: §2.1 · p. 2

2.2 The M2E2 Dataset

  • The event inventory maps 8 ACE types (24% of ACE types) to 98 imSitu types (20% of imSitu types), and its expanded roles cover 52% of ACE events in a news corpus.The role set adds visual arguments from imSitu; the authors reuse the ontologies to train event and argument classifiers without joint multimedia event-annotation training data.Source: §2.2 · p. 2, Table 1 · p. 3
  • From 108,693 VOA articles collected in 2006–2017, the authors selected 245 for annotation; the dataset contains 1,105 text-only, 188 image-only, and 395 multimedia event mentions.Selection favored articles with more event mentions, more than four images, and balanced event-type distribution; 309 cross-media mention pairs align 192 textual and 203 visual mentions.Source: Table 2 · p. 3, §2.2 · p. 3
  • For crowded scenes, annotations use union boxes covering all role constituents or instance boxes for individual participants; eight researchers achieved 81.2% inter-annotator agreement before expert adjudication.The annotation process follows ACE guidelines for text and arguments and uses a designed guideline for multimedia events.Source: §2.2 · p. 3

3 Method

WASE jointly encodes text and images into a structured common space, trains shared event and argument classifiers using separate unimodal resources and weak cross-media alignment, then jointly extracts events and arguments from multimedia articles.

3.1 Approach Overview

  • Three training tasks—text event extraction, visual situation recognition, and cross-media alignment—learn shared encoders and classifiers for joint multimodal inference and cross-modal coreference.At test time, sentences and images are encoded in the structured common space before textual and visual events and arguments are extracted.Source: §3.1 · p. 3

3.2 Text Event Extraction

  • Text representations combine word-sequence and AMR graph context, using a Bi-LSTM and a graph convolutional network to produce common-space word embeddings.Entity representations average the embeddings of their tokens; the shared classifiers predict event types and argument roles.Source: §3.2 · pp. 3–4

3.3 Image Event Extraction

  • Images are represented as situation graphs with a verb node and role-labeled arguments, constructed either from detected objects or role-conditioned attention heatmaps.The object-based model uses detected boxes, while the attention-based alternative addresses object types missing from detector ontologies.Source: §3.3 · pp. 4–5

3.4 Cross-Media Joint Training

  • Separate ACE and imSitu annotations train the modality-specific classifiers, while paired VOA images and captions provide weak supervision for aligning graph nodes without ground-truth node alignments.The alignment objective brings nodes from matching image-caption pairs closer and pushes apart nodes from irrelevant pairs, enabling shared classifiers across modalities.Source: §3.4 · p. 6

3.5 Cross-Media Joint Inference

  • At test time, WASE pairs each sentence with its closest image, combines word features with aligned image representations, and applies shared classifiers to both modalities.It then corefers cross-media events of the same type when their sentence-image similarity exceeds a threshold.Source: §3.5 · pp. 6–7

4 Experiments

WASE outperforms the evaluated baselines across text-only, image-only, and multimedia settings, while qualitative examples show cross-modal context and structured representations at work. Image-argument localization remains a challenge, particularly for attention-based models.

4.1 Evaluation Setting

  • Evaluation uses Precision, Recall, and F1; visual arguments require IoU above 0.5, and baselines cover text-only, image-only, and multimedia models.Attention-based bounding boxes are generated by thresholding heatmaps at 0.75 times their peak value; the common space is 300-dimensional with three GCN layers.Source: §4.1 · p. 7

4.2 Quantitative Performance

  • WASEatt and WASEobj outperform all baselines in F1 across text-only, image-only, and multimedia evaluation settings.The comparison with Flat also supports the effectiveness of structured embedding, while multimedia training improves performance on text-only and image-only events.Source: §4.2 · p. 8
  • WASEobj outperforms multimedia embedding models and event-type matching on cross-media event coreference performance.The evaluation pairs textual and visual event mentions in the same document and compares the pairs with ground truth.Source: Table 4 · p. 8, §4.2 · p. 8

4.3 Qualitative Analysis

  • Qualitative examples show images helping text event extraction, sentences informing visual event extraction, and WASE capturing structures missed by flat embeddings.The examples contrast modality-specific predictions with joint predictions and show structural distinctions such as an artifact positioned on a vehicle and an entity distinguished from an agent.Source: §4.3 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

4.4 Remaining Challenges

  • Image argument localization remains difficult: object-based models are limited by object types, while attention maps can be imprecise or lose focus across many instances.The reported error examples include a correctly identified entity with incorrect localization and heatmaps covering the whole image when an argument targets many instances.Source: §4.4 · p. 8

5 Related Work

Prior work addresses text and visual event extraction separately, and multimedia representation methods often align images with language without modeling event structure and semantic roles.

Text Event Extraction

  • Text event extraction has been extensively studied for general news, and multimedia features have also been used to improve it.The cited text-event literature includes Ji and Grishman (2008), Li et al. (2013), and Wadden et al. (2019).Source: §5 · pp. 8–9

Visual Event Extraction

  • Visual event work often targets simpler activities and may omit argument structure; Situation Recognition predicts events and arguments but does not localize arguments.This paper distinguishes its scope by including event identification and coreference, with argument localization through object detection and attention.Source: §5 · p. 9

Multimedia Representation

  • Multimedia representation methods often align images with captions or regions with words while ignoring structure and semantic roles; UniVSE captures attributes and relations but not graph-level structure.Source: §5 · p. 9

6 Conclusions and Future Work

The paper introduces multimedia event extraction and a benchmark, and develops a weakly supervised structured common space using image-caption pairs and single-modal annotated data. It positions the work as a step toward multimedia event understanding, with video extraction and broader event types among planned extensions.

6 Conclusions and Future Work

  • The paper proposes multimedia event extraction, establishes a benchmark, and develops a structured common space method using existing image-caption pairs and single-modal annotations for weakly supervised training.Future work includes video event extraction, scaling to new event types, expanding annotations, and downstream applications such as cross-media inference and timeline generation.Source: §6 · p. 9

Related papers

Loading 2005.02472v1…