Source-linked AI summary
CLIP-Event: Connecting Text and Images with Event Structures
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, Shih-Fu Chang
TL;DR
Existing vision-language pretraining largely overlooks event semantics and argument structures, despite their importance for distinguishing events. CLIP-Event transfers event knowledge extracted from captions through contrastive learning, hard structural negatives, prompt functions, and optimal-transport graph alignment. It outperforms state-of-the-art vision-language pretraining models on event extraction and several downstream tasks, including under zero-shot settings.
Problem
Existing vision-language pretraining models focus on objects and entities while often ignoring event semantics and argument structures needed to distinguish participant roles.
Method
CLIP-Event extracts event structures from captions, contrasts prompt-generated hard negative descriptions, and aligns image-text event graphs using optimal transport.
Results
CLIP-Event significantly outperforms state-of-the-art vision-language pretraining models on Multimedia Event Extraction and Grounded Situation Recognition in zero-shot and supervised settings.
Takeaways & Limitations
The framework supports zero-shot multimedia event extraction by identifying event structures from natural-language descriptions of unseen event types and argument roles.
Abstract
from arXiv · showhide
Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding objects in images or entities in text, they often ignore the alignment at the level of events and their argument structures. In this work, we propose a contrastive learning framework to enforce vision-language pretraining models to comprehend events and associated argument (participant) roles. To achieve this, we take advantage of text information extraction technologies to obtain event structural knowledge, and utilize multiple prompt functions to contrast difficult negative descriptions by manipulating event structures. We also design an event graph alignment loss based on optimal transport to capture event argument structures. In addition, we collect a large event-rich dataset (106,875 images) for pretraining, which provides a more challenging image retrieval benchmark to assess the understanding of complicated lengthy sentences. Experiments show that our zero-shot CLIP-Event outperforms the state-of-the-art supervised model in argument extraction on Multimedia Event Extraction, achieving more than 5% absolute F-score gain in event extraction, as well as significant improvements on a variety of downstream tasks under zero-shot settings.
1. Introduction
CLIP-Event addresses the limited treatment of visual events and argument roles in vision-language pretraining by transferring event structure from captions to images. It uses hard structural negatives, event-aware alignment, and an event-rich dataset to improve zero-shot event understanding and downstream performance.
- Motivation: Existing vision-language models focus on objects and entities while often ignoring event semantics and argument roles, although role differences can distinguish otherwise similar events.For ATTACK, protester and police can switch between ATTACKER and TARGET roles, changing the event interpretation.
- Approach: CLIP-Event extracts event structures from captions and uses them as distant supervision for interpreting events in associated images.The framework transfers caption-derived event knowledge across modalities through self-supervised contrastive learning.
- Approach: The framework contrasts correct descriptions with hard negatives created by manipulating event types and argument roles through multiple prompt functions.The prompts encode positive and manipulated structures as natural-language descriptions for image-text contrastive training.
- Approach: CLIP-Event introduces event graph alignment based on optimal transport to align image objects with text entities and their argument roles.The alignment extends image-object matching toward event-structure-aware cross-media alignment.
- Results: CLIP-Event significantly outperforms state-of-the-art vision-language pretraining models on Multimedia Event Extraction and Grounded Situation Recognition in both zero-shot and supervised settings.It also improves several downstream tasks under zero-shot evaluation, including image retrieval and visual commonsense reasoning.
- Resources: The released dataset contains 106,875 event-rich images and provides a challenging image-retrieval benchmark for complex, lengthy sentences.The dataset includes extracted event knowledge for evaluating event understanding in real-world applications.
2. Our Approach
CLIP-Event acquires event structures from text, generates hard negatives by manipulating event types and argument roles, and aligns event graphs across images and text. Its approach combines event-aware prompting, entity-level alignment, and optimal-transport-based global graph alignment.
- Event Structural Knowledge Extraction: The approach extracts events from captions and detects image objects, then selects a primary event using dependency structure, argument count, event frequency, and image similarity.The primary event is chosen by ranking candidate events under these criteria and applying majority voting.
- Event Structure Driven Negative Sampling: Hard negatives target visually confusable event types and altered argument roles to force robust representations of event semantics and participant roles.Negative event types come from a CLIP confusion matrix, while argument roles are manipulated by right-rotating their ontology-defined sequence.
- Event Structure Driven Negative Sampling: Multiple prompt functions encode positive and corrupted event structures, including templates, learnable continuous prompts, caption editing, and GPT-3-generated descriptions.The prompts vary in how they serialize arguments and preserve or alter information while expressing event structures as text.
- Event Graph Alignment via Optimal Transport: Event graphs represent event nodes connected to entities through argument roles, enabling alignment of image objects with the entities and roles they fulfill.This structure distinguishes entities with similar visual content but different argument functions, such as an injured man as the transported ENTITY rather than AGENT.
- Event Graph Alignment via Optimal Transport: Optimal transport computes a global graph distance by learning a soft node alignment from a cost matrix combining event, entity, role, and object similarities.The image-text pipeline also computes cosine similarity and distance for modality-level alignment, while entity costs combine mention and type similarity.
3. Evaluation Tasks
The evaluation tests whether CLIP-Event supports event and argument understanding across extraction, retrieval, reasoning, and intent-generation tasks. Several evaluations use zero-shot image-text alignment to assess transfer without task-specific supervision.
- Multimedia Event Extraction: Multimedia Event Extraction classifies images into eight event types and localizes argument roles as image bounding boxes.Its zero-shot setting evaluates open-vocabulary event handling and the effect of event knowledge encoded during pretraining.
- Grounded Situation Recognition: Grounded Situation Recognition selects an event type from 504 verbs and predicts each argument role’s entity name and bounding box.The implementation is described as similar to M2E2, with evaluation details provided in the appendix.
- Image Retrieval: Image retrieval evaluates image-text alignment by ranking images for captions with Recall@1, Recall@5, and Recall@10.The approach aligns images and text at both the global modality level and the event-graph level.
- Visual Commonsense Reasoning: Visual question answering evaluates answer and rationale prediction from four options each, using zero-shot image-text alignment and F-scores.The task architecture is presented as part of the evaluation-task framework.
- VisualCOMET: VisualCOMET ranks candidate participant intents from image-text similarity and reports Accuracy@50.The task conditions on an image and its event participants to generate intents.
4. Experiments
Experiments evaluate CLIP-Event on event extraction, retrieval, and downstream reasoning, including zero-shot settings and ablations of event-graph alignment. Results show gains on event and argument extraction, challenging lengthy-caption retrieval, and several downstream tasks.
- Pretraining Dataset: 106,875 event-rich image-caption pairs form the VOANews pretraining dataset and a challenging retrieval benchmark with lengthy, multi-event sentences.Average captions contain 28.3 tokens, compared with 13.4 for Flickr30k and 11.3 for MSCOCO.
- Event Extraction: 5.5% absolute F-score gain in event extraction and 33.3% relative gain in argument extraction are achieved on M2E2 under zero-shot settings.These results quantify improvements in both event classification and participant-role extraction.
- Event Extraction: News-pretrained CLIP improves 1.9% over vanilla CLIP on M2E2, while CLIP-Event increases that gain to 3.89 times.The comparison isolates the added benefit of structural event knowledge during pretraining.
- Event Extraction: Zero-shot CLIP-Event outperforms the state-of-the-art weakly supervised model on M2E2 argument extraction.The authors attribute this result to optimal-transport alignment capturing argument structures that prior vision-language models miss.
- Downstream Tasks: VOANews retrieval improves more than Flickr30k and MSCOCO retrieval, indicating better handling of lengthy captions containing many events.Downstream analyses also report benefits from fine-grained event-graph alignment in retrieval, VCR, and VisualCOMET.
- Ablation Studies: Removing optimal transport generally lowers performance across evaluation tasks, with the largest gain appearing on argument extraction.The transport plan performs global alignment over event-graph structures and associates roles with visually compatible objects.
5. Related Work
Related work shows that vision-language models and visual-event methods have largely underrepresented event structures and argument roles. CLIP-Event addresses this gap through open-world event extraction and global cross-modal alignment over event graphs.
- Vision-Language Pretraining: Vision-language pretraining models have succeeded with Transformer-based image-text learning, but event structural knowledge remains insufficiently captured.This deficiency is linked in the cited work to weaker performance on tasks involving verb comprehension.
- Visual Event Understanding: Prior visual-event methods often simplify events into Subject-Verb-Object triples or rely on limited event ontologies.The cited approaches include Situation Recognition and Multimedia Event Extraction.
- Cross-media Alignment: Existing cross-media alignment methods maximize modality alignment without modeling the structures of text and images.CLIP-Event uses text event graphs to bridge complicated linguistic structures and image structures through global alignment.
6. Conclusions and Future Work
The paper integrates structural event knowledge into vision-language pretraining by extracting event information from captions, generating structured hard negatives, and aligning event graphs with optimal transport. It reports zero-shot gains and identifies video event evolution as future work.
- Conclusions: CLIP-Event extracts event knowledge from captions and supervises image event-structure understanding through contrastive learning.Hard negatives are generated by manipulating event structures and converting them into natural-language descriptions with event prompts.
- Conclusions: Optimal-transport event-graph alignment transfers argument structural knowledge through global alignment between visual and textual argument structures.The method is presented as a mechanism for cross-media transfer of event knowledge.
- Conclusions: CLIP-Event outperforms state-of-the-art vision-language pretraining models on event extraction and downstream tasks under zero-shot settings.The conclusion summarizes the paper-level result without restricting it to a single benchmark.
- Future Work: The future direction is extending the capability to videos to comprehend event evolution through argument tracking.This scope boundary is explicitly identified by the authors as future work.
A.3. Multimedia Event Extraction Implementation Details
The Multimedia Event Extraction implementation evaluates open-vocabulary event and argument recognition directly from the pretrained model, with supervised fine-tuning using annotated SWiG events for comparison.
- Task Definition: Multimedia Event Extraction classifies images into eight event types and localizes argument roles as bounding boxes.The task is selected as a direct assessment of event-structure understanding.
- Zero-shot Setting: In zero-shot evaluation, the model ranks event types by image-text similarity and evaluates event and argument extraction across images containing eight visual-event types.An OTHER event type and OTHER argument role are added as candidates for out-of-target cases.
- Supervised Setting: Supervised evaluation fine-tunes CLIP-Event on the 125k-image SWiG training dataset using annotated image events instead of extracted text events.The optimal transport plan remains the fine-grained alignment between event graphs.
A.4. Grounded Situation Recognition Implementation Details
Grounded Situation Recognition evaluates event understanding by predicting an image’s event type, participant names, and argument bounding boxes. The implementation ranks candidate verbs and argument descriptions using fixed prompt templates.
- The task selects an event type from 504 verbs and predicts each argument’s entity name and bounding box.
- The evaluation directly measures event-structure understanding across a larger set of event types and argument roles.
- For each image, verbs are ranked with the prompt “An image of ⟨verb⟩”.
- Argument candidates are ranked with prompts linking each candidate name to its role and event verb, such as “The man is a agent of riding”.
- Evaluation separately measures verb prediction, argument-name prediction, and joint argument bounding-box and name prediction.Argument metrics include value, value-all, ground, and ground-all variants.
A.5. VCR Implementation Details
VCR evaluates whether event understanding supports visual question answering and rationale selection. The implementation uses zero-shot image-text alignment together with event-graph alignment to rank candidate answers and rationales.
- VCR contains two four-option tasks: answer prediction and rationale prediction supporting the selected answer.
- The zero-shot evaluation relies solely on image-text alignment to compare pretraining models fairly.
- Answers are ranked by concatenating each answer with the question, while rationales combine the question, answer, and rationale.
- Both image alignment d(i, t) and event-graph alignment d(Gi, Gt) contribute to ranking answers and rationales.
A.6. VisualCOMET Implementation Details
VisualCOMET tests whether event understanding and argument roles support predicting participants’ intents before, during, and after an image event. Candidate intents are ranked using both image and event-graph alignment, although the analysis notes possible errors from text information extraction.
- VisualCOMET generates participant intents describing what they need to do before, want to do during, and will most likely do after an image event.
- The task requires understanding events, their temporal connections, and participant argument roles.
- For each image and participant, candidate intents from training data are ranked using image alignment d(i, t) and event-graph alignment d(Gi, Gt).
- The ranking text concatenates the input event description, a temporal prompt, and a candidate intent.An example uses “because person1 wanted to” before the candidate intent.
- The paper notes that errors in text information extraction are analyzed in subsequent sections.
B.1. Text Event Extraction Performance Table
The extraction analysis reports component performance and characterizes the event distribution used for evaluation. Caption-derived events are described as primarily visually detectable, while the retrieval benchmark reports R@1, R@5, and R@10 across three datasets.
- 72.1% F-score is achieved on event extraction across the evaluated components.
- The component results are organized in a performance table reporting percentage scores for each component.
- B.2. Event Type distribution: Caption-derived events are primarily visually detectable and can be depicted in the corresponding images.
- Image retrieval is reported with R@1(%), R@5(%), and R@10(%) on Flickr30k, MSCOCO, and VOANews.
- B.2. Event Type distribution: The event-type analysis identifies the top frequent event types extracted from VOANews captions.