PaperlayerView exact revision on arXiv

Source-linked AI summary

VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation

Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, Tamara Lee Berg, Mohit Bansal, Jingjing Liu, Lijuan Wang, Zicheng Liu

arXiv:2106.04632v2cs.CVcs.CL

TL;DR

VidL evaluations often focus on a single dataset or task, limiting assessment across diverse settings. VALUE brings together 11 datasets across three tasks and evaluates baseline and multi-task approaches. An all-task model improves meta-average over separately trained models, though the best tested models remain well below human performance.

  • Problem

    VidL methods are often evaluated on distinct datasets and settings, making fair comparison difficult, while limited task coverage makes universal systems hard to measure.

    Source: §1 · p. 1

  • Method

    VALUE compiles 11 datasets across retrieval, question answering, and captioning, and evaluates baselines while studying input channels, fusion, transfer, and multi-task training.

    Source: §1 · p. 2

  • Results

    All-task training improved meta-average by +1.89 points over separately trained models (52.33 vs. 50.44), while the best tested models remained well below human performance.

    Source: §1 · p. 2, §4.5 · p. 9

  • Takeaways & Limitations

    VALUE provides a broad evaluation platform for studying general-purpose VidL models, which the authors say remain challenging to design.

    Source: §5 · p. 10

    VALUE currently covers a selected set of datasets and tasks, and all included datasets are in English.

    Source: §5 · p. 10

Abstract

from arXiv · show

Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.

1 Introduction

VidL methods are often compared on different datasets and experimental settings, while many evaluations cover only a limited set of tasks. VALUE addresses this gap with a multi-task benchmark and studies how input channels and joint training affect model performance.

  • VidL methods are often evaluated on distinct datasets and settings, making fair comparison difficult, and limited task coverage makes universal systems hard to measure.Source: §1 · p. 1
  • VALUE compiles 11 VidL datasets across retrieval, question answering, and captioning, with varied genres, video lengths, data volumes, and task difficulties.It provides video frames and accompanying subtitle sentences as inputs, and offers a universal meta-average target metric across tasks.Source: §1 · p. 2
  • The benchmark evaluates baselines with and without pre-training and investigates input channels, fusion methods, video representations, task transfer, and multi-task training.Source: §1 · p. 2

2 Related Work

VidL research lacks a large-scale benchmark for systematically tracking progress, and methods are often assessed on different tasks, datasets, and experimental settings. VALUE broadens task coverage and welcomes different model types.

  • NLP benchmarks such as GLUE, SuperGLUE, XTREME, and XGLUE cover varied task settings, while VALUE focuses on understanding and generation in VidL.SentEval targets sentence-level vector representations, whereas GLUE allows any type of representation.Source: §2 · p. 2
  • VidL understanding lacks a large-scale benchmark for systematically tracking advances, while methods are evaluated on different tasks, datasets, and settings.Source: §2 · pp. 2–3
  • The Pentathlon Challenge combines five text-based video retrieval tasks but restricts evaluation to offline extracted features, whereas VALUE includes diverse VidL tasks.VALUE is model-agnostic and welcomes methods of all kinds.Source: §2 · p. 3

3 VALUE Benchmark Tasks

VALUE assembles 11 datasets spanning retrieval, question answering, and captioning, with video from varied domains and a range of lengths. The benchmark specifies task settings, dataset splits, and evaluation measures for these task groups.

3 VALUE Benchmark Tasks

  • VALUE combines 11 datasets across three VidL tasks, drawing on multi-channel and single-channel videos from varied genres, domains, and lengths.Average video length ranges from 10 to 76 seconds, and every dataset except VATEX has a high proportion of videos with subtitles.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Text-based Video Retrieval Tasks

  • Retrieval includes video retrieval and video-corpus moment retrieval, with the latter also requiring localization of the relevant moment.Performance uses average recall at 1, 5, and 10; moment retrieval additionally requires high temporal overlap with the ground-truth span.Source: §3.1 · pp. 4–5

3.2 Video Question Answering Tasks

  • The QA group uses classification accuracy and includes full-video-clip QA, video-and-language inference, and future-event prediction.For TVQA and How2QA, the benchmark considers QA on the full clip rather than grounded-clip QA or moment localization.Source: §3.2 · p. 5

3.3 Video Captioning Tasks

  • Captioning covers video-moment descriptions, clip-level cooking captions, and English VATEX captions, evaluated primarily with CIDEr-D.TVC uses subtitles as additional input and captions moments rather than entire videos; YC2C is evaluated at clip level.Source: §3.3 · pp. 5–6

4 Experiments and Analysis

Experiments evaluate multi-channel inputs and fusion, visual representations, task transfer, and multi-task training across VALUE. Joint training can improve benchmark performance, but the best reported meta-average comes from single-task finetuning and remains far from human parity.

4.1 Features and Baseline

  • Experiments use HERO as a multi-channel video encoder, with task-specific heads for VALUE tasks.HERO fuses subtitle sentences with accompanying local video segments, then uses a temporal transformer to contextualize video segments globally.Source: §4.1 · p. 6

4.2 Impact of Input Channels and Video-Subtitle Fusion Methods

  • Using both video and subtitle channels gives the highest meta-average score, 52.52, and the best performance across VALUE tasks.QA tasks generally benefit more from subtitles than video, unlike retrieval and captioning; adding subtitles also helps tasks originally collected without them.Source: §4.2 · p. 7, Table 3 · p. 7
  • HERO fusion has the highest meta-average score, 52.52, but sequence concatenation performs best on VATEX and also beats HERO on How2R, YC2R, and VLEP.The authors select HERO for subsequent experiments because the goal is a fusion method that performs well across all 11 tasks.Source: §4.2 · pp. 7–8, Table 4 · p. 7

4.3 Impact of Visual Representations

  • Without pre-training, CLIP-ViT+SlowFast performs best, suggesting VALUE tasks require both appearance and motion information; with HERO pre-training, ResNet+SlowFast performs best.Experiments use ResNet+SlowFast features extracted every 1.5 seconds, alongside subtitles, with HERO encoding video segments and subtitle sentences hierarchically.Source: §4.1 · p. 6, §4.3 · p. 8

4.4 Task Transferability Evaluation

  • Transfer across tasks of the same type yields low performance, reflecting differences in domain, video length, and task formalization.Examples include TV versus instructional videos, 60–90-second versus 10-second videos, and retrieval tasks with different task formalizations.Source: §4.4 · p. 8

4.5 Multi-Task Learning Evaluation

  • Training one model on all 11 datasets (AT, L5) outperforms separately trained single-task models on 8 of 11 tasks, raising meta-average by +1.89 points (52.33 vs. 50.44) with approximately 11 times fewer parameters.AT also outperforms task- and domain-based multi-task baselines on meta-average despite using fewer parameters.Source: §4.5 · p. 9, Table 6 · p. 9
  • Task-based and domain-based multi-task training each improve meta-average over single-task training.Task-based training scores 52.00 vs. 50.44 (+1.56 points), with gains on retrieval and QA, especially smaller datasets; domain-based training scores 51.97 vs. 50.44 (+1.53 points).Source: §4.5 · p. 9
  • Single-task finetuning with pre-training achieves the best reported meta-average score of 57.00, while pre-training and multi-task finetuning often do not complement each other.Multi-task finetuning degrades captioning performance more severely; the best model remains far from human parity, especially on QA tasks.Source: Table 6 · p. 9, §4.5 · p. 10

5 Conclusion and Discussion

VALUE brings together 11 multi-channel VidL datasets across three tasks, spanning varied genres, video lengths, difficulties, and data volumes. The authors conclude that general-purpose VidL models remain challenging to design, while noting the benchmark’s selected-task and English-only scope.

5 Conclusion and Discussion

  • VALUE combines 11 multi-channel VidL datasets across three tasks and a broad range of video genres, lengths, difficulties, and data volumes.The authors see general-purpose VidL modeling as an ongoing challenge and plan to add diagnostic datasets and quantitative and qualitative model analysis.Source: §5 · p. 10
  • The benchmark currently covers only a selected set of datasets and tasks, with future additions proposed to broaden that coverage.The authors also identify multilingual VidL datasets as a future complement because the current benchmark is English-only.Source: §5 · p. 10
  • All datasets in VALUE are in English, so multilingual VidL evaluation is not currently included.The authors attribute this scope boundary to limited availability of multilingual VidL datasets.Source: §5 · p. 10

A Additional Data Statistics

This section reports video-length distributions and frequent words in subtitles and annotations, including examples from YouCook2.

A Additional Data Statistics

B.1 Impact of Visual Representations

The visual-representation analysis finds that CLIP-ViT features outperform other tested 2D or 3D features, while combining 2D and 3D features performs best overall in the reported comparison. Results also vary by task and pre-training domain, and HERO pre-training benefits multiple visual representations.

B.1 Impact of Visual Representations

  • CLIP-ViT features outperform other tested 2D or 3D features, and combining CLIP-ViT with SlowFast achieves the best performance.The results are reported in Table 9; combining 2D and 3D features does not always help, as ResNet+S3D performs worse than S3D alone.Source: §B.1 · pp. 14–15
  • Video-text pre-trained S3D features benefit VALUE tasks, but most of the improvement comes from YC2 tasks, whose videos resemble its pre-training videos.The authors infer that the video domain of pre-training data is critical to downstream performance.Source: §B.1 · p. 15
  • Retrieval and captioning depend more on visual-representation quality, while QA performance remains relatively stable across the tested features.The authors relate the QA pattern to greater reliance on subtitle information, as observed in Table 3.Source: §B.1 · p. 15
  • HERO pre-training improves performance in almost all tasks across the examined visual representations, including representations different from those used during pre-training.The authors interpret this as evidence that video-language alignment learned through HERO pre-training transfers across visual representations.Source: §B.1 · p. 15

B.2 Zero-Shot Evaluation on CLIP

The zero-shot evaluation applies CLIP to retrieval and question-answering tasks using encoded video frames and text, with an optional subtitle channel. CLIP performs decently on two retrieval tasks but struggles with moment retrieval and QA, and adding subtitles does not guarantee better results.

B.2 Zero-Shot Evaluation on CLIP

  • For video-only evaluation, CLIP encodes sampled frames and text, then uses similarity scores for retrieval, moment localization, and QA predictions.The procedure uses mean-pooled frame features for global video representations and a sliding window for moment localization.Source: §B.2 · p. 16
  • The subtitle variant max-pools encoded subtitle sentences and averages subtitle-only and video-only similarity scores.Captioning is not directly evaluated because CLIP has no trained decoder.Source: §B.2 · p. 16
  • CLIP achieves decent performance on YC2R and VATEX-EN-R, but struggles with video moment retrieval and QA.The authors report that CLIP cannot distinguish semantically similar video segments for moment retrieval and struggles to select the correct answer on similar QA queries.Source: §B.2 · pp. 16–17, Table 10 · p. 16
  • Adding subtitles to video-only CLIP does not guarantee improved performance.The authors suggest ineffective prediction-level video-subtitle fusion or limited capacity to align subtitle information with the query as possible explanations.Source: Table 10 · p. 16, §B.2 · p. 17

B.3 Additional Results on Multi-Task Baselines

AT →ST with pre-training achieves the highest reported meta-average score among the multi-task baselines. Human baselines are collected separately for QA and captioning, with a captioning exception for YC2C.

B.3 Additional Results on Multi-Task Baselines

  • 57.58 is the highest meta-average score, achieved by AT →ST with pre-training (L11).Source: §B.3 · p. 17
  • Validation results and detailed retrieval and captioning metrics are reported for validation and test leaderboard splits.Source: §B.3 · p. 17

C Collection of Human Baselines

  • For QA human baselines, annotators answer multiple-choice questions, judge VIOLIN hypotheses, or choose likely next events; majority vote is used for VIOLIN and VLEP.Source: C Collection of Human Baselines · p. 18
  • Captioning human performance uses one randomly sampled ground-truth caption against the remaining references; it is not reported for YC2C, which has one caption per clip.Source: C Collection of Human Baselines · p. 19

D Additional Experimental Details

HERO is adapted to retrieval, QA, hypothesis or event selection, and captioning with task-specific heads or decoders. The appendix also describes early video-subtitle fusion baselines and shared-encoder multi-task training.

D.1 Downstream Adaptation

  • Retrieval adds a query encoder head; moment retrieval uses local and global alignment losses, while video retrieval uses the combined hinge loss.Source: §D.1 · p. 19
  • For multiple-choice QA, question-answer pairs are fused with subtitle and video context, then scored by an MLP and softmax; timestamp supervision adds a span-prediction loss.Source: §D.1 · p. 19, §D.2 · p. 19
  • VIOLIN hypotheses and VLEP event candidates are fused with subtitle and temporal context, then pooled into query-aware representations for cross-entropy training.Source: §D.1 · p. 19
  • Captioning uses a Transformer decoder attending to a captioned moment for TVC or the whole clip for YC2C and VATEX-EN-C.Source: §D.1 · p. 19

D.2 Video-Subtitle Fusion Methods

  • Three early-fusion baselines combine video and subtitle embeddings by sequence concatenation, temporally aligned summation, or temporally aligned feature concatenation before temporal-transformer encoding.Source: §D.2 · p. 19

D.3 Multi-Task Baselines

  • Multi-task models share a HERO encoder and use one head per task type, including a shared Transformer decoder across captioning tasks.Source: §D.3 · p. 20

D.4 Implementation Details

Implementation uses specified optimization, hardware, and task-sampling procedures, with Azure-generated subtitles for YC2 and VATEX. The benchmark appendix also describes licenses, dataset bias, and the settings covered by detailed multi-task results.

D.4 Implementation Details

  • Experiments run on 4 or 8 Nvidia V100 GPUs, with mixed precision and gradient accumulation used to speed training and reduce communication overhead.Source: §D.4 · p. 20
  • AdamW uses an initial learning rate from {3e −5, 5e −5, 1e −4}, 10% warmup, and linear decay.Source: §D.4 · p. 20
  • Single-task schedules use dataset-specific learning rates and steps; multi-task batches sample tasks approximately in proportion to training-set size, with checkpoints selected by validation meta-average.Source: §D.4 · p. 20
  • Subtitles for YC2 and VATEX are generated using Azure Cognitive Service ASR.Source: §D.4 · p. 20

E License and Usage

  • Dataset annotations use CC BY-NC-SA 4.0 or CC BY 4.0 licenses, while released features use CC BY-NC-SA 4.0 and code uses MIT.Source: E License and Usage · p. 20
  • The datasets contain biases in videos and annotations, so the authors caution against relying completely on systems trained on them for real-world decisions.Source: E License and Usage · p. 20
  • Table 13 covers retrieval and captioning results on validation and test leaderboard splits, comparing single-task, multi-task, all-task, and AT →ST settings with and without pre-training.Source: Table 13 · p. 21
Loading 2106.04632v2…