PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Pretraining for Dense Video Captioning

Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, Radu Soricut

arXiv:2011.11760v1cs.CVcs.CLcs.LG

TL;DR

Instructional videos are difficult to skim, and time-stamped step annotations can help users locate relevant content. The paper introduces ViTT and pretrains segment-captioning models with text-only or multimodal signals; these models improve results across YouCook2 and ViTT, with strong performance and reported generalization across varied instructional videos.

  • Problem

    Instructional videos are harder to skim than text, while time-stamped annotations for their main steps can help users find relevant moments.

    Source: §1 · p. 1

  • Method

    The paper releases ViTT and explores text-only and multimodal pretraining for segment-level video captioning using unsupervised signals.

    Source: §1 · p. 2

  • Results

    Pretraining improves performance over no pretraining on YouCook2 and ViTT, with state-of-the-art YouCook2 results and strong ViTT performance.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results indicate that the models generalize well and are robust across a wide variety of instructional videos.

    Source: §1 · p. 2

    YouCook2's almost uniform distribution over 89 recipes makes generalization to unseen topics difficult to judge.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Learning specific hands-on skills such as cooking, car maintenance, and home repairs increasingly happens via instructional videos. The user experience with such videos is known to be improved by meta-information such as time-stamped annotations for the main steps involved. Generating such annotations automatically is challenging, and we describe here two relevant contributions. First, we construct and release a new dense video captioning dataset, Video Timeline Tags (ViTT), featuring a variety of instructional videos together with time-stamped annotations. Second, we explore several multimodal sequence-to-sequence pretraining strategies that leverage large unsupervised datasets of videos and caption-like texts. We pretrain and subsequently finetune dense video captioning models using both YouCook2 and ViTT. We show that such models generalize well and are robust over a wide variety of instructional videos.

1 Introduction

Instructional videos can be difficult to skim, so time-stamped step annotations could help users find relevant content. This paper introduces ViTT and explores pretraining for segment-level video captioning, reporting gains on YouCook2 and ViTT.

1 Introduction

  • Because videos are harder to skim than text, users can benefit from time-stamped annotations that identify steps and link to relevant moments.The paper frames automatic generation of this metadata as a way to scale easier access to video content.Source: §1 · p. 1
  • ViTT adds around 8,000 instructional videos across topics such as cooking, repairs, and maintenance, with an average of 7.1 time-stamped segments per video.Each segment has a short free-text description.Source: §1 · p. 2
  • The paper investigates pretraining a video-segment captioning model using unsupervised signals and evaluates on YouCook2 and ViTT.The task assumes segment boundaries are given.Source: §1 · p. 2
  • Text-only and multimodal pretraining both significantly improve performance over no pretraining on YouCook2 and ViTT, with state-of-the-art YouCook2 results and strong ViTT performance.The authors conclude that the models generalize well and are robust across a wide variety of instructional videos.Source: §1 · p. 2

2 Related Work

Prior work spans language-only and multimodal pretraining, as well as video captioning at whole-video and segment levels. This paper situates its approach among methods that use video and ASR for segment-level captions.

Text-only Pretraining

  • MASS is a joint encoder-decoder pretraining strategy for sequence-to-sequence tasks, while BERT-style approaches separately pretrain encoder and decoder components.The cited work reports MASS outperforming those separate approaches for sequence-to-sequence tasks.Source: §2 · p. 2

Multimodal Pretraining

  • Prior multimodal work uses BERT-style objectives for video and ASR encoders or joint video-ASR representations; UniViLM uses separate encoder and decoder pretraining, unlike this paper's joint MASS approach.The passage identifies UniViLM as concurrent work sharing some pretraining objectives, setup, and the YouCook2 benchmark (Luo et al., 2020).Source: §2 · p. 2

Dense Video Captioning

  • Dense video captioning generates fine-grained captions for predefined event segments, unlike video captioning that produces one summary for the entire video.The paper focuses on segment-level captioning (Krishna et al., 2017; Wang et al., 2019).Source: §2 · pp. 2–3

Instructional video captioning data sets

  • Related instructional datasets include densely labeled action segments in Epic Kitchen and creator-authored video-level descriptions in How2, alongside YouCook2 (Damen et al., 2018; Sanabria et al., 2018; Zhou et al., 2018a).Prior segment-captioning approaches include models using video and ASR as well as methods that do not use ASR directly.Source: §2 · p. 3

3 Data

The data section covers the segment-caption datasets used for finetuning and evaluation, the unannotated video and text sources used for pretraining, and differences between those sources and target captions.

3 Data

  • The data section uses segment-level caption datasets for finetuning and evaluation and unannotated datasets for pretraining.The pretraining sources include paired ASR+video and unpaired instruction-step text.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4

3.1 Dense Video-Captioning Datasets

  • ViTT annotations mark the start of each main step and provide a concise free-text tag; YouCook2's near-uniform distribution across 89 recipes limits conclusions about generalization to unseen topics.ViTT tags average 2.97 words, compared with 8.8 words per YouCook2 description.Source: §3.1 · pp. 3–4, §3.4 · p. 5
  • ViTT contains 8,169 videos, including 3,381 cooking videos, with an average of 7.1 segments per video and 88,455 segments overall.Its 56,027 unique tags form two variants: ViTT-All and the cooking subset ViTT-Cooking.Source: §3.1 · p. 4

3.2 Pretraining Datasets: ASR+Video

  • ASR paired with video provides two pretraining sources: YT8M-cook has 186K segments, while HowTo100M has 7.99M segments after preprocessing.Their average segment lengths are 64 words and 78 words, respectively.Source: §3.2 · p. 4

3.3 Pretraining Datasets: CAP-style

  • Recipe1M recipe steps and WikiHow step summaries serve as separate CAP-style text-pretraining examples, totaling 10,767,594 and 1,360,145 segments, respectively.The sources average 12.8 words per Recipe1M segment and 8.2 words per WikiHow segment.Source: §3.3 · p. 4

3.4 Differences between Pretraining and Finetuning Datasets

  • Pretraining and finetuning segments can differ in boundaries, caption length, style, and granularity, yet the authors report that the pretraining data helps the task.ViTT captions average 2.97 words, shorter than the CAP-style sources and YouCook2 captions.Source: §3.4 · pp. 4–5

4 Method

The paper uses MASS-style Transformer objectives for segment-level caption generation, with text-only and multimodal variants. It combines separate-modality encoding with reconstruction and alignment/ordering pretraining, then finetunes models for captioning.

4 Method

  • Segment-level caption generation uses MASS-style Transformer pretraining and fine-tuning in text-only and multimodal variants.Source: §4 · p. 5

4.1 Separate-Modality Architecture

  • Separate text and video Transformers exchange cross-modal attention when available, and the text decoder attends to both encoders’ final outputs; text inputs also include style embeddings.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 Pretraining with Text-only MASS

  • Text-only MASS alternates masked-subsequence reconstruction of ASR and caption-like text, using unpaired examples from independent sources.Source: §4.2 · p. 5

4.3 Pretraining with Multimodal MASS

  • MASSvid alternates CAP→CAP reconstruction with masked-ASR reconstruction conditioned on co-occurring video; MASSdrop hides text-encoder output for some training examples.Source: §4.3 · pp. 5–6

4.4 Pretraining with Alignment and Ordering Tasks

  • MASSalign alternates text-only reconstruction with same-segment alignment classification, same-video negatives at least two segments away, and before/after ordering.Source: §4.4 · p. 6

4.5 Finetuning on Video Captioning

  • Finetuning maps ASR to caption-like text, optionally with video; some bidirectional variants are beneficial whether training from scratch or from pretrained models.Source: §4.5 · p. 6

5 Experiments

Multimodal MASS pretraining improves segment-captioning results across YouCook2 and ViTT, while the experiments qualify evaluation through short-tag metrics and cross-paper comparability limits.

5.1 Implementation Details

  • Text was truncated to 240 subwords and video sequences to 40 frames (40 seconds); 128-dimensional Compact 2D features were projected into the embedding space.Source: §5.1 · p. 7
  • Training ran for 200 pretraining and 30 finetuning epochs; ViTT evaluation emphasizes ROUGE-L and BLEU-1 because 67% of tags contain fewer than four words.Segments from any one video were kept together in either validation or test, and each segment was treated as a separate instance with one reference caption.Source: §5.1 · p. 7

5.2 Main Results

  • Comparisons with prior YouCook2 methods are not always direct because processing, tokenization, splits, dataset shrinkage, pretraining data, and segmentation strategies can differ.Source: §5.2 · p. 7
  • 4.37 ROUGE-L for E2vidD6-MASSvid-BiD over E2vidD6-BiD on YouCook2; gains also reach +4.22 on ViTT-Cooking and +2.97 on ViTT-All.Source: §5.2 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 9
  • Most pretraining improvement is achievable with text-only MASS, and multimodal pretraining exceeds text-only pretraining by under one ROUGE-L point across all three datasets.Source: §5.2 · p. 8
  • MASSalign validation accuracy reached 90% for alignment and 80% for ordering on YT8M-cook, and 87% and 71.4%, respectively, on HowTo100M, versus 50% for random guessing.The authors interpret these results as evidence that the video features are reasonably strong and that the reported findings are not due to weak visual representations.Source: §5.2 · p. 8
  • D6 with bidirectional finetuning tends to perform best, but decoder and direction differences are under one ROUGE-L point; 2D and 3D features show no large performance difference on YouCook2.The paper uses simpler 2D features for multimodal pretraining and leaves more extensive visual-feature experiments for future work.Source: §5.2 · p. 8
  • 31.5 ROUGE-L on ViTT versus 39.0 ROUGE-L on YouCook2 reflects the new benchmark’s greater difficulty, while similar pretraining and finetuning conditions maximize performance across its broader instructional-video coverage.Source: §5.2 · p. 9

6 Conclusions

The paper introduces ViTT, a larger and more diverse instructional-video dataset, and evaluates multimodal pretraining for segment-level captioning. MASS-style pretraining is the most decisive factor in the reported benchmark gains.

6 Conclusions

  • ViTT is larger and more diverse than YouCook2, covering a wider range of instructional videos.The dataset was collected to improve information-seeking capabilities for videos.Source: §6 · p. 9
  • The paper investigates several multimodal pretraining strategies for segment-level video captioning and conducts extensive ablations.The experiments use both YouCook2 and ViTT.Source: §6 · p. 9
  • MASS-style pretraining is the most decisive factor in improving performance across all benchmarks used.The results also indicate that most performance can be attributed to leveraging the ASR signal.Source: §6 · p. 9
  • The models achieve new state-of-the-art results on YouCook2 and establish strong performance baselines for ViTT.The authors present these baselines as starting points for further progress.Source: §6 · p. 9

A Appendix

This section identifies itself as supplementary material for the paper.

A Appendix

  • The section is titled supplementary material for “Multimodal Pretraining for Dense Video Captioning.”Source: A Appendix · p. 13

A.1 The ViTT dataset

ViTT was built from sampled YouTube videos and annotated with segment boundaries and concise labels for instructional content. Its 8,169 videos span varied topics, while short, diverse tags create evaluation and modeling considerations.

A.1 The ViTT dataset

  • Videos were sampled from YouTube-8M to mirror topic distributions, excluding videos without English ASR and those labeled “Game.”The sampling aimed for roughly half cooking videos and half randomly sampled non-cooking videos verified as instructional.Source: §A.1 · p. 13
  • Annotators judged whether videos showed real-world human actions explained by procedural language, then identified segments, start times, and labels.They could adjust suggested start times when needed.Source: §A.1 · p. 13
  • Guidelines targeted segments useful for rewatching or navigation, suggesting five to ten segments when the video’s structure permitted.For short videos, annotators were told to prioritize narrative quality over reaching five segments.Source: §A.1 · p. 13
  • To suggest segment start times, ASR tokens were split at gaps over two seconds, then a punctuation model identified sentence boundaries.Annotators saw each sentence with the timestamp of its first token and could adjust the start time.Source: §A.1 · p. 13
  • ViTT contains 8,169 annotated instructional videos, averaging 7.1 segments per video, with descriptions averaging 2.97 words.Of the videos, 3,381 are cooking-related; the dataset contains 88,455 segments and descriptions reach a maximum of 16 words.Source: §A.1 · pp. 13–14
  • Frequent intro and ending paraphrases can be penalized by automatic metrics, so the authors standardized three groups among the top 20 tags.The passage notes variants such as “intro” and “introduction.”Source: §A.1 · p. 14, Table 6 · p. 14
  • The 88,455 segments contain 56,027 unique tags, including 51,474 that appear once, making pure classification infeasible.Source: §A.1 · p. 14

A.2 Separated vs. Concatenated-Modality Architecture

The paper uses separated unimodal transformers rather than concatenating modalities, citing flexibility and concerns about joint text-video processing.

A.2 Separated vs. Concatenated-Modality Architecture

  • Separated unimodal transformers avoid requiring video and text embeddings to share a space and allow variable-length text-only, video-only, or joint inputs.The authors also cite concerns about shared attention weights and joint layer normalization in concatenated architectures.Source: §A.2 · p. 15

A.3 Additional Implementation Details

The implementation uses a V100 GPU, Adam with an inverse square root schedule, and dropout; pretraining takes 3–6 days and bidirectional finetuning up to 1.5 days.

A.3 Additional Implementation Details

  • Models are optimized on an nVidia V100 GPU using Adam with an inverse square root schedule and batch size 32.Source: §A.3 · p. 15
  • Training uses 4,000 warm-up iterations, a maximum learning rate of 0.0001, randomly initialized positional embeddings, and dropout and attention dropout of 0.1.Source: §A.3 · p. 15
  • With E2vidD6, pretraining takes 3–6 days and bidirectional finetuning takes up to 1.5 days, though data-pipeline optimization could improve those times.Source: §A.3 · p. 15

A.4 Example Predictions

The section presents good and bad prediction examples from YouCook2 and ViTT-All, generated by models with and without text-only MASS pretraining.

A.4 Example Predictions

  • The examples cover predictions on both YouCook2 and ViTT-All.Source: §A.4 · p. 15
  • The displayed examples include both good and bad predictions.Source: §A.4 · p. 15
  • The captions compare E2vidD6-BiD without pretraining against E2vidD6-MASS-BiD with text-only MASS pretraining.Source: §A.4 · p. 15

A.5 Full result tables

The reported ablations associate pretraining, especially use of ASR signals at both pretraining and finetuning, with accuracy improvements and faster convergence. The section also provides prediction examples and full result tables across YouCook2 and ViTT variants.

A.5 Full result tables

  • Pretraining improves accuracy across all reported metrics, with the results indicating value from leveraging ASR signals during both pretraining and finetuning.Source: §A.5 · p. 15
  • On ViTT-All, MASS-pretrained finetuning reaches its best ROUGE-L score at epoch 2, versus around 11 epochs to converge from scratch.Source: §A.5 · p. 15
  • Prediction examples indicate that pretrained models are generally, but not always, better on YouCook2 and ViTT.Source: Figure 3 · p. 16, Figure 4 · p. 17, Figure 5 · p. 18
  • YouCook2 was curated to contain actual recipe steps, so it has no intro- or outro-like labels.Source: Figure 3 · p. 16
  • Table 10 reports YouCook2 captioning results using YT8M-cook/Recipe1M pretraining; video features are Compact 2D except where marked S3D.Source: Table 10 · p. 19
  • Table 11 reports ViTT-All captioning results using HowTo100M/WikiHow pretraining and includes an estimate of human performance.Source: Table 11 · p. 20
  • Table 12 reports ViTT-Cooking captioning results, with YT8M-cook and Recipe1M available for optional pretraining.Source: Table 12 · p. 21

Related papers

Loading 2011.11760v1…