PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Semantic Role Labeling for Video Understanding

Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Nevatia, Aniruddha Kembhavi

arXiv:2104.00990v1cs.CVcs.CL

TL;DR

Action labels omit participants and their roles, while open-ended captions lack a clear evaluation target. The paper defines VidSRL as structured event and relation prediction and introduces VidSitu with capability-specific baselines; its human-agreement analysis finds room for improvement.

  • Problem

    Action recognition misses event participants and their roles, while open-ended video captioning lacks a clear definition of what information is extracted, making evaluation difficult.

    Source: §1 · p. 2

  • Method

    VidSRL represents a video as related salient events with verbs and role-bearing entities, co-reference across events, and labels for event relations, studied using the VidSitu dataset and capability-specific baselines.

    Source: §1 · p. 2

  • Results

    The benchmark provides capability-specific baselines and finds significant room for improvement in human agreement; video encoders outperform video-blind event-relation baselines.

    Source: §1 · p. 2, §6.2 · p. 8

  • Takeaways & Limitations

    VidSRL offers a structured route toward holistic video understanding by representing participants and links among events, beyond isolated action labels.

    Source: §1 · p. 2, §7 · p. 8

    The task fixes one salient event per two-second interval in a ten-second clip, a design choice made to balance descriptive richness with objective evaluation.

    Source: §3 · p. 3

Abstract

from arXiv · show

We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling. We represent videos as a set of related events, wherein each event consists of a verb and multiple entities that fulfill various roles relevant to that event. To study the challenging task of semantic role labeling in videos or VidSRL, we introduce the VidSitu benchmark, a large-scale video understanding data source with $29K$ $10$-second movie clips richly annotated with a verb and semantic-roles every $2$ seconds. Entities are co-referenced across events within a movie clip and events are connected to each other via event-event relations. Clips in VidSitu are drawn from a large collection of movies (${\sim}3K$) and have been chosen to be both complex (${\sim}4.2$ unique verbs within a video) as well as diverse (${\sim}200$ verbs have more than $100$ annotations each). We provide a comprehensive analysis of the dataset in comparison to other publicly available video understanding benchmarks, several illustrative baselines and evaluate a range of standard video recognition models. Our code and dataset is available at vidsitu.org.

1. Introduction

The paper proposes VidSRL to represent video events with their participants and relations, addressing the missing structure in action labels and the evaluation difficulty of open-ended captions. It introduces VidSitu and capability-specific baselines for studying this task.

1. Introduction

  • VidSRL identifies salient events, their participating actors, objects and locations, co-refers entities across events, and represents how events affect each other over time.The authors present it as a step toward holistic understanding of complex videos, with more precise extracted information than video captioning.Source: §1 · p. 2
  • VidSitu contains over 29K ten-second videos drawn from 3K movies, with five salient-event verb annotations per video at two-second intervals.Each event also has roles whose values are free-form text; expressions can disambiguate entities and are kept consistent across events in a clip.Source: §1 · p. 2
  • Each video has five inter-related events, averaging 4.2 unique verbs, 6.5 unique entities, and 3.8 roles per event.The dataset includes verb-specific roles, free-form entity values, co-reference across events, and event-relation labels for causation and contingency.Source: §1 · p. 2
  • VidSitu spans 1,500 curated verbs, with 200 verbs receiving at least 100 event annotations, and 5,600 unique nouns, with 350 appearing in at least 100 videos.The dataset's situations are characterized as both diverse and complex.Source: §1 · p. 2
  • The paper provides partwise capability evaluation, state-of-the-art component baselines, metrics selected to signal progress, and a human-agreement analysis showing significant room for improvement.The dataset and code are publicly available at vidsitu.org.Source: §1 · p. 2

2. Related Work

Related work spans video tasks with different temporal horizons, datasets, and annotation types. The paper positions VidSRL as combining event-level roles and cross-event understanding while focusing on visual activity in movies.

2. Related Work

  • Video tasks range from atomic actions at one-second intervals to summarizing 180-second videos, while VidSRL operates on ten-second videos at two-second intervals.A task description frames VidSRL as predicting a verb and role-bearing entities for each interval, then relating events over time.Source: §2 · pp. 2–3
  • Table 1 is a non-exhaustive summary of video-understanding tasks, their required annotations, and benchmarks.The paper also states that VidSRL operates at a ten-second horizon with two-second intervals.Source: §2 · p. 2, Table 1 · p. 3
  • The paper situates VidSRL among datasets supporting video analysis, noting a shift from small datasets to large collections that support neural-model training and pretraining.The related-work discussion points to comparisons with relevant datasets in Section 4, Table 3, and Figure 2.Source: §2 · p. 3
  • Unlike image-based role labeling, video role labeling must aggregate interacting entities across frames and co-reference them across events.The related-work discussion also names situation recognition, visual semantic role labeling, and human-object interaction as image-domain work.Source: §2 · p. 3
  • For movie-based tasks, VidSitu focuses on visual activity by actors and objects, without movie scripts, subtitles, or audio.The paper notes other movie-domain uses including detection, description, question answering, retrieval, social graphs, and shot-style classification.Source: §2 · p. 3

3. VidSRL: The Task

VidSRL formalizes a video as related salient events, each described by a verb and role values. Its design fixes event intervals, supports entity co-reference across events, and labels relations between events.

3. VidSRL: The Task

  • VidSRL addresses the impoverished representations of activity recognition and object detection and the open-ended evaluation problem of captioning by producing structured event descriptions.The task is motivated as a way to represent complex interactions among actors, objects, and locations more richly.Source: §3 · p. 3
  • Given a video, VidSRL predicts related salient events, each with a verb and values assigned to roles relevant to that verb.The role assignment notation means that a verb's jth role receives value a; the event pair also has a relation label.Source: §3 · p. 3
  • The task fixes one salient event per two-second interval, yielding five events in a ten-second clip, to balance descriptive richness with objective evaluation.The authors characterize event timescale as ambiguous and subjective, and report that two seconds strikes a qualitative balance.Source: §3 · p. 3
  • Verbs come from a curated PropBank vocabulary with verb senses and roles, while free-form argument values can distinguish entities through referring expressions.Roles may require information from the whole video when an entity is occluded or outside the camera view during the event interval.Source: §3 · p. 4
  • Because grounding entities with bounding boxes is expensive, the task assesses co-reference by requiring phrases for the same entity to match exactly within a clip.An entity may participate in multiple events and play different roles in them.Source: §3 · p. 4
  • The proposed event-relation taxonomy labels a later event as caused by, enabled by, a reaction to, or unrelated to an earlier event.Enabled-by denotes that the earlier event does not cause the later one, but the later event would not occur without it.Source: §3 · p. 4

4. VidSitu Dataset

VidSitu provides videos of diverse, complex situations annotated with verbs, semantic roles, entity co-references, and event relations. The section describes its curation and compares its diversity, complexity, and annotation richness with other video datasets.

4.1. Dataset Curation

  • VidSitu selection targets diverse, visually interesting 10-second clips; its verb-sense vocabulary and argument roles are curated from PropBank and movie descriptions.Annotations use free-form descriptions and support entity reuse across a video.Source: §4.1 · p. 4
  • The 80:5:15 split keeps each movie in one set; validation and test sets have separate task-specific test sets and multiple annotations.The annotation process collects 10 candidate verbs per clip, then role labels for the highest-agreement verb from three workers.Source: §4.1 · pp. 4–5

4.2. Dataset Analysis and Statistics

  • 224 verbs and 336 nouns have at least 100 annotations, and VidSitu offers greater verb and noun diversity than the compared datasets.Verb senses are collapsed for this analysis; movies such as VidSitu and MPII are generally more diverse than open-domain datasets.Source: §4.2 · pp. 5–6
  • About 80% of VidSitu videos have at least 4 unique verbs and 70% have at least 6 entities, versus 20% and 30% for VATEX-en.At least 4 semantic roles occur in 90% of VidSitu events, compared with 55% for VATEX-en.Source: §4.2 · p. 6
  • VidSitu adds structured verbs, semantic roles, entity coreferences, and event relations to video descriptions, supporting evaluation and study of coreference and event relations.Figure 2-f shows the fraction of entity coreference chains by length.Source: §4.2 · p. 6

5. Baselines

The baselines address verb prediction, semantic-role prediction given verbs, and event-relation prediction. They combine video-recognition features with transformer and language-model components.

5. Baselines

  • Verb prediction classifies the most salient event in a 2-second clip using I3D or SlowFast variants, trained from scratch or after Kinetics pretraining.The models use cross-entropy loss; frozen verb classifiers provide features for later stages.Source: §5 · p. 6, Figure 3 · p. 6
  • For each of five events, a transformer encoder contextualizes event features, then a verb-conditioned decoder generates that event’s role and argument sequence.Language-only baselines use the transformer decoder or GPT-2.Source: §5 · pp. 6–7
  • Event-relation prediction concatenates visual and RoBERTa language representations for an ordered event pair, then classifies their relation.The language representation encodes the events’ verbs, arguments, and roles.Source: Figure 3 · p. 6, §5 · p. 7

6. Experiments

The experiments evaluate verb prediction, semantic-role prediction with coreference, and event-relation prediction, using metrics adapted to ambiguity and human agreement. Baselines show promise, but results also reveal substantial gaps, including in coreference and event relations.

6.1. Evaluation Metrics

  • The evaluation separates verb prediction, role prediction with coreference, and event relations; human annotations also show ambiguity in both verb choices and referring expressions.The ambiguity analysis used 25 verb and five role annotations for each of 500 events from 100 videos, and manual inspection attributed most annotation differences to inherent ambiguity rather than poor annotation quality.Source: §6 · p. 7, §6.1 · p. 7
  • For ambiguous verbs and an imbalanced vocabulary, the benchmark favors macro-averaged Recall@k, reports R@5 as preferred, and includes Acc@1/5 for completeness.Source: §6.1 · p. 7
  • Role evaluation retains Arg0, Arg1, Arg2, ALoc, and AScn, reports micro- and macro-averaged CIDEr plus ROUGE-L, and uses LEA-soft to weight coreference cluster matches by CIDEr.Event relations are evaluated with class-averaged top-1 accuracy, using event pairs where two of three annotators agreed.Source: §6.1 · p. 7

6.2. Results

  • ≈10 to 16 points: Kinetics pretraining consistently improves recall across models, and SlowFast significantly outperforms I3D for verb prediction.Source: §6.2 · p. 8
  • Video-conditioned models outperform video-blind baselines for role prediction, and event contextualization improves almost all metrics, while a large gap from human performance remains.Source: §6.2 · p. 8
  • Only Vid TxEncDec beats the language-only GPT2 baseline on coreference metrics on both validation and test sets, indicating current techniques do not learn coreference directly from data.Source: §6.2 · p. 8
  • Video encoders outperform video-blind event-relation baselines, which predict the majority class; adding verb-sense and argument context yields small gains.Source: §6.2 · p. 8
  • Powerful baselines show promise on the three sub-tasks, but VidSitu still poses significant challenges with substantial room for improvement.Source: §6.2 · p. 8

7. Conclusion

The conclusion defines VidSRL as recognizing salient actions, their participants and roles, cross-event co-reference, and relations among actions. The accompanying appendix lists further material on semantic roles and dataset collection.

7. Conclusion

  • VidSRL requires recognizing salient actions and their role-bearing entities, co-referencing entities across time, and identifying how actions affect each other.Source: §7 · p. 8

Appendix

  • The appendix lists supplementary material on roles, dataset curation, statistics, implementation, metrics, the datasheet, and qualitative analysis.Source: Appendix · p. 9

A. Semantic Roles: A Brief Summary

B. Dataset Collection

  • The dataset-collection section covers verb and argument curation, the annotation interface, quality control, and rewards.Source: B. Dataset Collection · p. 9

B.1. Dataset Curation

Dataset curation selects movie clips for diversity and complex situations, then defines the video, verb, role, and event-relation annotations used in VidSitu.

Video Source Selection

  • Movie clips were chosen to provide diverse situations and complex events, after other video domains were judged less suitable on those dimensions.Source: §B.1 · p. 9

Video Selection

  • The selection heuristic ranks clips by detected people’s unique atomic actions, filters routine and martial-arts actions, and samples non-overlapping clips.Source: §B.1 · p. 10
  • Annotating a uniform sample showed fewer unique verbs under the heuristic, supporting its usefulness for selecting richly annotatable segments.Source: §B.1 · p. 10, Figure 1 · p. 10

Curating Verb Senses

  • Verb curation filters PropBank senses for visual relevance and movie-description use, yielding 2,154 verb senses.Source: §B.1 · p. 10

Curating Argument Roles

  • Argument roles draw on frequent roles in movie descriptions, omit infrequent or temporally implicit roles, and add Scene when a verb lacks Location.Source: §B.1 · p. 10

Event Relations

  • Event relations distinguish no relation, causality, contingency, and a Reaction To relation for responses not clearly caused by an earlier event.Source: §B.1 · pp. 10–11

B.2. Annotation pipeline

The annotation pipeline collects verb, role, and event-relation labels from workers, with interval-based annotation and a fixed reference event shaping the task. The section also describes quality controls and validation procedures.

Annotation Interface

  • Workers watch each 10-second video, label the salient event in each 2-second interval, fill role slots with free-form phrases, reuse entity phrases for co-reference, and label event relations.Source: §B.2 · pp. 11–12

Partitioning into 2-second clips

  • Two-second intervals reduce annotation time and improve consensus, but yield imprecise event boundaries and do not represent hierarchical actions.Source: §B.2 · p. 12

Event Relation Annotation w.r.t. Middle Event

  • Event relations are annotated relative to Event 3, avoiding the greater cost of labeling every pair and the inconsistency of choosing a subjective main event.Source: §B.2 · p. 12

Worker Qualification and Quality Control

  • Workers qualify through a verified task, while automated checks and recurring manual sample reviews provide quality control.Source: §B.2 · p. 12

Annotating Validation and Test Sets

  • Human verb agreement rose from 62% with 10 annotations to 71% with 20, motivating Recall@5 rather than accuracy, precision, or F1.Source: §B.2 · p. 12, Table 1 · p. 12
  • Validation and test annotations use two stages—additional verb labels followed by role and relation labels—and cost $3.9 per video, about 5.2× a training video.Source: §B.2 · pp. 12–13

C. Additional Dataset Statistics

Additional statistics describe event-relation agreement and the distributions of verbs, genres, movies, and argument words in VidSitu.

Event Relations

Verb and Genre Distributions

Argument Distributions

D. Implementation Details

The implementation details specify the verb and argument prediction setups, followed by the metrics used to evaluate generated roles and entity co-reference.

D.1. Verb Prediction Models

  • Verb prediction uses Kinetics-pretrained I3D, Slow-only, and SlowFast models, selecting the checkpoint with the best recall@5.A 1,560-verb classifier predicts the top five scoring verbs.Source: §D.1 · p. 13

D.2. Argument Prediction Models

  • Argument models use Byte-Pair Encoding with special role tokens, and GPT2 training uses batch size 8 due to memory restrictions.Other models use batch size 16; GPT2 training takes around 10 hours over eight GPUs.Source: §D.2 · pp. 13, 15
  • Argument prediction uses three-layer transformer encoders and decoders with eight attention heads, trained using cross-entropy and greedy decoding.The decoder attends to the final encoder layer; beam search and alternate temperatures did not improve results.Source: §D.2 · p. 13

E. Evaluation Metrics

  • The evaluation reports BLEU, METEOR, ROUGE, CIDEr, and co-reference metrics, including the proposed LEA-soft.CIDEr and BLEU are among the additional metrics reported for argument generation.Source: E. Evaluation Metrics · p. 16

E.1. Co-Reference Metrics

  • LEA is a link-based co-reference metric, while LEA-soft addresses high scores from correctly linking incorrectly predicted entities by incorporating CIDEr into precision.LEA-soft weights entity importance with CIDEr scores; recall remains unchanged.Source: §E.1 · p. 16
  • The final co-reference score is F1; LEA-soft modifies precision with entity CIDEr scores while leaving recall unchanged.For LEA-soft, scores are computed against each of three ground-truth references and averaged.Source: §E.1 · p. 16

E.2. Evaluation of Arguments

The argument evaluation identifies which free-form roles have reliable human agreement and describes the metrics reported for validation and test performance.

E.2. Evaluation of Arguments

  • Human agreement is high for Arg0, Arg1, Arg2, ALoc, and AScn, so evaluation excludes direction and manner, where agreement and language-only baseline scores are poor.Direction permits multiple valid descriptions, while manner is open to subjective interpretation.Source: §E.2 · p. 16
  • CIDEr scores for all collected arguments were compared using five annotations on 100 videos.The language-only baseline generates role values from an event verb; the passage reports poor agreement for direction and manner.Source: §E.2 · p. 16, Table 3 · p. 16

E.3. All Metrics

  • Validation and test results report BLEU, METEOR, ROUGE, CIDEr, and co-reference metrics, with macro-averaged scores across verbs and arguments.CIDEr alone can use the macro-averaged scores through inverse document frequency reweighting.Source: §E.3 · p. 16

F. VidSitu DataSheet

The VidSitu datasheet describes the dataset’s motivation, composition, collection, processing, uses, and distribution, including boundaries on its suitability for real-world use.

F.1. Motivation

  • VidSitu is intended to bridge the gap between atomic-action learning and holistic caption generation by supporting video semantic-role labeling.The task emphasizes object interactions within and across actions, co-reference over time, and relations among actions.Source: §F.1 · pp. 16, 19

F.2. Composition

  • The dataset has 29.2K ten-second videos available from YouTube, annotated at 2-second intervals with event verbs, co-referenced argument roles, and relations to the middle event.They are split randomly by movie name into training, validation, and test sets in an 80:5:15 ratio.Source: §F.2 · p. 19, dataset) · p. 19
  • The authors note possible redundancy from oversampled dialogue and closely related verbs such as “run” and “jog,” and warn that some clips may contain violence or gore.Action, crime, and horror clips may be sensitive to some viewers.Source: dataset) · p. 19

F.3. Collection Process

  • Crowdworkers collected annotations through a custom Mechanical Turk interface, with validation via a small-scale user study and worker feedback.Video sampling favored clips containing more verbs; collection lasted 2.2 months.Source: §F.3 · pp. 19–20

F.4. Preprocessing/cleaning/labeling

  • Preprocessing used exact string matching for co-referenced entities; spaCy was used for dataset statistics, not downstream tasks.The authors state that the raw data is the same as the cleaned data.Source: §F.4 · p. 20

F.5. Uses

  • The dataset is intended as a video-understanding benchmark, not for real-world cases, because movie stereotypes and unlikely action sequences may not reflect reality.The authors also identify possible downstream uses including retrieval, question answering, forecasting, and long-term reasoning.Source: §F.5 · p. 20

F.6. Distribution

  • VidSitu is publicly available through the project website and GitHub, with data stored in Amazon S3 buckets.Source: §F.6 · p. 20

Related papers

Loading 2104.00990v1…