PaperlayerView exact revision on arXiv

Source-linked AI summary

Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid

arXiv:2302.14115v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Unified dense video captioning models require large-scale training data that current annotated datasets do not provide. Vid2Seq uses narrated-video speech and timestamps as weak supervision, and achieves state-of-the-art results across dense, paragraph, and clip captioning benchmarks.

  • Problem

    Unified dense video captioning models require large-scale training data that current annotated datasets do not provide.

    Source: §1 · p. 1

  • Method

    Vid2Seq uses transcribed speech sentences and their timestamps from narrated videos as weak supervision to train a model that jointly predicts event boundaries and captions.

    Source: §1 · p. 2, §3.2.1 · p. 5

  • Results

    Vid2Seq achieves state-of-the-art results across dense, paragraph, and clip captioning benchmarks, improving CIDEr by 18.2 points on YouCook2 over PDVC.

    Source: §1 · p. 2, §4.3 · p. 7

  • Takeaways & Limitations

    The authors suggest Vid2Seq's sequence-to-sequence design may extend to tasks such as temporally grounded video question answering [51] [56] [57] and temporal action localization [67] [123].

    Source: §5 · p. 8

    Transcribed-speech supervision is weak because transcripts are not always visually grounded and are often temporally misaligned.

    Source: §3.2.1 · p. 5

Abstract

from arXiv · show

In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. The Vid2Seq architecture augments a language model with special time tokens, allowing it to seamlessly predict event boundaries and textual descriptions in the same output sequence. Such a unified model requires large-scale training data, which is not available in current annotated datasets. We show that it is possible to leverage unlabeled narrated videos for dense video captioning, by reformulating sentence boundaries of transcribed speech as pseudo event boundaries, and using the transcribed speech sentences as pseudo event captions. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset improves the state of the art on a variety of dense video captioning benchmarks including YouCook2, ViTT and ActivityNet Captions. Vid2Seq also generalizes well to the tasks of video paragraph captioning and video clip captioning, and to few-shot settings. Our code is publicly available at https://antoyang.github.io/vid2seq.html.

1. Introduction

Dense video captioning localizes and describes every event in an untrimmed video, a harder task than captioning a short clip but useful for large-scale video search and indexing. Vid2Seq unifies event timing and caption generation and learns from narrated videos as weak supervision.

  • Dense video captioning localizes and captions all events in untrimmed videos [46] [102], unlike standard video captioning, which produces one caption for a short clip [63] [70] [80].The task is more difficult because it requires localization across minutes-long videos, but can support large-scale video search and indexing where content is not segmented into clips.Source: §1 · p. 1
  • Existing methods often separate localization and captioning or retain task-specific components, and their reliance on limited manual annotations hinders learning [37] [46] [100] [102] [35] [46] [130].Vid2Seq draws inspiration from sequence-to-sequence models pretrained on Web data [4] [11] [13] [105].Source: §1 · p. 1
  • Vid2Seq augments a Web-pretrained language model with time tokens to jointly generate event boundaries and captions as one sequence from video frames and transcribed speech.The unified sequence model can learn multimodal dependencies among events through attention.Source: §1 · pp. 1–2
  • The model is pretrained on narrated videos by treating speech sentence boundaries and sentences as pseudo event boundaries and captions, using generative and denoising objectives.Because speech can be noisy or misaligned with video, Vid2Seq jointly models narrations and timestamps to learn from this supervision.Source: §1 · p. 2
  • Vid2Seq improves the state of the art on three dense-captioning datasets, two paragraph-captioning benchmarks, and two clip-captioning datasets, and generalizes to few-shot settings.For paragraph captioning, it outperforms prior approaches without ground-truth event proposals at inference time.Source: §1 · p. 2

2. Related Work

Related work frames Vid2Seq as a unified generative approach to dense captioning: it emits event timestamps and captions in one sequence, and its pretraining targets detailed multi-event generation in untrimmed videos. It also casts temporal event localization as language modeling.

Dense video captioning

  • Vid2Seq emits event timestamps and captions sequentially, conditioned on prior events, unlike paragraph grounding, separate localization and captioning [125], or speech-timestamp inference.Because timestamps are generated as special tokens rather than inferred from speech, it can caption videos with limited speech, as demonstrated on ActivityNet Captions.Source: §2 · p. 2
  • Most dense captioners localize events before captioning, while joint methods improve interaction between the tasks and Wang et al. [102] predict event-caption sets in parallel.Source: §2 · p. 2

Video and language pretraining

  • Video-text pretraining mostly learns global representations for retrieval, question answering, and clip captioning, whereas Vid2Seq targets detailed multi-event generation in untrimmed videos.Prior work also explores long-form and temporal-localization pretraining, but focuses on video understanding rather than generative description of multiple events.Source: §2 · p. 2

Pretraining for dense video captioning

  • Prior dense-caption pretraining used ActivityNet annotations [125], ground-truth proposals on instructional videos [35], or domain-specific text [35]; Vid2Seq uses a generic video corpus [121].The paper reports benefits on multiple downstream datasets and domains.Source: §2 · pp. 2–3

Unifying tasks as language modeling

  • Prior work casts computer-vision tasks such as object detection [11], grounded image captioning, and visual grounding as language modeling; Vid2Seq applies it to temporal event localization in untrimmed videos.This differs from prior language-modeling work on image-level spatial localization.Source: §2 · p. 3

3. Method

Vid2Seq casts dense video captioning as generation of a sequence combining event timestamps and descriptions, and uses transcribed speech from narrated videos as weak pseudo-event supervision.

3.1. Model

  • Vid2Seq predicts an ordered event sequence of timestamp tokens and text, using visual frames and a similarly structured speech transcript to model event relationships.A multimodal encoder-decoder encodes video and speech, then autoregressively predicts event boundaries and captions in one sequence.Source: §3 · p. 3, §3.1 · p. 4

3.2.1 Pretraining on untrimmed narrated videos

  • Narrated speech provides weak supervision because transcripts may be visually ungrounded or temporally misaligned and are far denser than standard event annotations.YT-Temporal-1B videos average 120 speech sentences; Vid2Seq contextualizes speech boundaries and semantics across minutes-long videos to learn long-term relationships between segments.Source: §3.2.1 · p. 5
  • Pretraining combines a visual-only generative task that predicts the transcript with denoising of corrupted speech and time tokens to train the visual encoder, text encoder, and decoder.The generative task omits transcript input to avoid text-only shortcuts; denoising asks the decoder to recover masked spans from video and corrupted speech context.Source: §3.2.1 · p. 5, Figure 3 · p. 5

3.2.2 Downstream task adaptation

  • For downstream tasks, Vid2Seq is finetuned to predict event sequences from video and speech transcripts, and beam search generates sequences that are converted into event predictions.Removing output time tokens adapts the model to video paragraph captioning, while the same finetuning and inference recipe supports video clip captioning.Source: §3.2.2 · pp. 5–6

4. Experiments

Experiments evaluate Vid2Seq through ablations and comparisons across dense, paragraph, clip, and few-shot captioning, alongside qualitative examples.

4.1. Experimental setup

  • Vid2Seq is pretrained on 18 million YT-Temporal-1B narrated videos and evaluated on YouCook2, ViTT, ActivityNet Captions, MSR-VTT, and MSVD.The model has 314M trainable parameters and is pretrained for 200,000 iterations; dense-captioning evaluation includes CIDEr, METEOR, and SODA c, while localization uses F1 across IoU thresholds.Source: §4.1 · p. 6

4.2. Ablation studies

  • Untrimmed video-speech pretraining improves performance over trimmed inputs, and integrating speech sentence-boundary times with time tokens yields a further drastic improvement.The comparison also finds gains over a baseline without video-text pretraining.Source: Table 1 · p. 6, §4.2 · p. 6
  • Generative pretraining benefits visual-only models, denoising improves multimodal reasoning, and joint captioning and boundary prediction localize better than localization-only training.The visual-only model drops in performance when fine-tuned with both visual and speech inputs.Source: Table 2 · p. 7, Table 3 · p. 7, §4.2 · p. 7
  • T5-Base outperforms T5-Small, larger narrated-video pretraining subsets improve performance, and HowTo100M pretraining achieves the best YouCook2 results.The paper also reports benefits from its pretraining method with only 150K narrated videos and notes the similar domains of HowTo100M and YouCook2.Source: Table 4 · p. 7, §4.2 · p. 7

4.3. Comparison to the state of the art

  • Vid2Seq sets a new state of the art on YouCook2, ViTT, and ActivityNet Captions, improving CIDEr over PDVC by 18.2 and 0.8 points on YouCook2 and ActivityNet Captions, respectively.The comparison uses cross-entropy training.Source: Table 5 · p. 7, §4.3 · p. 7
  • Vid2Seq localizes events better than prior work on YouCook2 and ViTT, but underperforms PDVC and UEDVC on ActivityNet Captions.The paper notes that PDVC and UEDVC use more task-specific temporal-localization components.Source: Table 6 · p. 7, §4.3 · p. 7

4.5. Qualitative examples

  • A qualitative ActivityNet example shows Vid2Seq predicting meaningful event boundaries and captions that differ considerably from the transcribed speech input.The paper attributes this difference as showing the importance of visual tokens in the input.Source: Figure 4 · p. 8, §4.5 · p. 8

4.3. Comparison to the state of the art

  • Vid2Seq outperforms prior methods on both paragraph-captioning datasets, improves over prior methods within their respective pretraining-data settings for clip captioning, and is evaluated few-shot by fine-tuning on small fractions of downstream training data.The paragraph-captioning comparison includes methods using ground-truth event-boundary proposals at inference time.Source: Table 7 · p. 8, Table 8 · p. 8, Table 9 · p. 8

5. Conclusion · Appendix

Vid2Seq generates text and time tokens in one sequence from multimodal inputs, benefits from pretraining on narrated videos, and achieves state-of-the-art results on several captioning benchmarks. The appendix lists supplementary examples, setup details, and experiments, while the conclusion notes possible extensions to other video tasks.

5. Conclusion

  • Vid2Seq achieves state-of-the-art results on dense event captioning datasets and video paragraph and standard video clip captioning benchmarks.Source: §5 · p. 8
  • Vid2Seq generates a single sequence of text and time tokens from multimodal inputs, using transcribed speech sentences and their temporal boundaries for large-scale pretraining.Source: §5 · p. 8
  • The sequence-to-sequence design may extend to temporally-grounded video question answering [51] [56] [57] and temporal action localization [67] [123].Source: §5 · p. 8

Appendix

  • The appendix provides additional qualitative examples of dense video captioning predictions in Section A.Source: Appendix · p. 14
  • The appendix provides additional information about the experimental setup.Source: Appendix · p. 14
  • The appendix includes additional experimental results in Section C.Source: Appendix · p. 14
  • Section C.1 reports an ablation on the importance of pretraining for few-shot dense video captioning.Source: Appendix · p. 14
  • Section C.2 presents additional ablation studies in the standard fully-supervised dense video captioning setting.Source: Appendix · p. 14

A. Qualitative examples of dense video captioning predictions

Vid2Seq produces meaningful dense captions and event boundaries across varied examples, but can also hallucinate events that are not visually grounded.

A. Qualitative examples of dense video captioning predictions

  • Vid2Seq predicts meaningful dense captions and event boundaries across diverse scenarios, with or without transcribed speech, including cooking instructions and sports or leisure activities.The examples are shown in Figures 5 and 6 on YouCook2 and ActivityNet Captions.Source: A. Qualitative examples of dense video… · p. 14
  • In the last Figure 6 example, Vid2Seq hallucinates events that are not visually grounded, such as “one man hats off to the camera.”Source: A. Qualitative examples of dense video… · p. 14

B. Experimental setup

Section B supplements the dataset information in Section 4.1 and provides additional implementation details.

B. Experimental setup

B.1. Datasets

The section defines narrated-video pretraining corpora and evaluation datasets whose scale, annotation formats, and evaluation procedures vary across tasks.

  • YT-Temporal-1B [121] provides 18.821M narrated videos spanning about 150 years across broader domains, while HowTo100M [72] provides 1.221M instructional videos spanning about 15 years for pretraining.Source: §B.1 · p. 14
  • YouCook2 [130] contains 1,790 untrimmed cooking videos averaging 320s, each annotated with 7.7 localized imperative sentences and split into 1,333 training and 457 validation videos.Source: §B.1 · p. 14
  • ViTT [35] contains 7,672 YouTube-8M videos intended to reflect wild instructional-video distributions, averaging 250s and 7.1 localized tags across training, validation, and test splits.For evaluation, each annotation set is treated as a separate example, and videos with more than three annotation sets are discarded.Source: §B.1 · p. 14
  • ActivityNet-Captions [46] contains 14,934 activity videos averaging 120s and 3.7 localized sentences, and 68% lack transcribed narration.Its validation annotations use both sets for evaluation; paragraph-caption results follow the val-ae split of 2,460 videos.Source: §B.1 · p. 14
  • MSR-VTT contains 10,000 open-domain clips lasting 10–30 seconds, each with 20 manually annotated descriptions and training, validation, and test splits.Source: §B.1 · p. 14
  • MSVD contains 1,970 open-domain clips lasting 10–30 seconds, each with roughly 40 manually annotated captions and training, validation, and test splits.Source: §B.1 · p. 14

B.2. Implementation details

Vid2Seq uses a uniform 12-layer transformer architecture and specific decoding, optimization, and regularization settings. Its pretraining and finetuning schedules vary by training stage and dataset.

B.2. Implementation details

  • The visual encoder, text encoder, and text decoder each use 12 layers, 12 heads, 768-dimensional embeddings, and an MLP hidden dimension of 2048.Sequences are padded or truncated to L = S = 1000 tokens during pretraining and S = 1000 and L = 256 during finetuning; inference uses beam search with the top 4 sequences and length normalization of 0.6.Source: §B.2 · p. 14
  • Pretraining uses Adam with β = (0.9, 0.999), no weight decay, and a learning rate of 1e−4.The learning rate warms up linearly for the first 1000 iterations, then remains constant; gradient norm is clipped to 0.1 during pretraining.Source: §B.2 · pp. 14–15
  • Finetuning uses a learning rate of 3e−4, batch size 32 across 16 TPU v4 chips, and gradient-norm clipping at 1.The learning rate warms up linearly for 10% of iterations, then decays cosine-wise to 0 over the remaining 90%; training lasts 40 epochs on YouCook2, 20 on ActivityNet Captions and ViTT, 5 on MSR-VTT, and 10 on MSVD.Source: §B.2 · p. 15
  • Training uses random temporal cropping, label smoothing with value 0.1, and dropout with probability 0.1.Source: §B.2 · p. 15

C. Experiments

Section C adds experiments complementing Section 4, covering pretraining in the proposed few-shot setting and ablations in the fully supervised setting.

C.1

  • Section C.1 examines the importance of pretraining in Vid2Seq’s proposed few-shot setting.The supplied passage states the question examined but does not report the experiment’s results.Source: C. Experiments · p. 15

C.2

  • Section C.2 ablates long-narrated-video pretraining, the pretraining dataset, and visual-backbone size in the fully supervised setting.The passage identifies these as factors considered in the ablation studies, without giving their results.Source: C. Experiments · p. 16
  • The fully supervised ablations also examine time-tokenization, the number of time tokens, sequence construction, temporal positional embeddings, and language-model initialization.The passage lists these factors but does not report their results.Source: C. Experiments · p. 16

C.1. Importance of pretraining in few-shot settings

Vid2Seq’s pretraining is important in few-shot dense event captioning, enabling non-zero performance with only 1% of the downstream training data.

C.1. Importance of pretraining in few-shot settings

  • Pretraining enables Vid2Seq to achieve non-zero performance with only 1% of the downstream training dataset.Table 10 presents the impact of pretraining when finetuning with a small fraction of the downstream data.Source: Table 10 · p. 17, §C.1 · p. 17

C.2. Additional ablation studies

Additional ablations show benefits from long narrated-video pretraining, stronger visual features, relative time tokens, and temporal positional embeddings. They also find that sequence ordering has mixed effects and text-only language-model initialization matters less after narrated-video pretraining.

Pretraining on long narrated videos

  • Pretraining on longer untrimmed narrated videos significantly outperforms cropping videos to at most 10 narration sentences.The default sampling can span hundreds of transcribed speech sentences.Source: §C.2 · p. 17

Visual features

  • CLIP-pretrained ViT-B/16 improves over ImageNet-pretrained ViT-B/16, and scaling up to ViT-L/14 brings additional improvements.The comparisons hold the visual backbone constant for the pretraining comparison, then scale backbone size.Source: §C.2 · p. 17

Time tokenization and number of time tokens

  • Relative time tokens with N = 100 yield the best dense video captioning results among the tested time-token variants.Relative tokens encode timestamps within each video; absolute tokens encode seconds, and increasing N affects their coverage differently.Source: §C.2 · pp. 17–18, Table 13 · p. 17

Sequence construction

  • Inserting time tokens before event text slightly improves localization F1 but slightly lowers overall dense captioning, while event-separation dots have low importance.The alternative inserts time tokens after each corresponding sentence; the variants achieve similar results overall.Source: Table 14 · p. 17, §C.2 · p. 18

Temporal positional embeddings

Language model initialization and pretraining

  • Text-only language-model initialization helps without narrated-video pretraining, but after that pretraining its effect is small and mixed across datasets.It slightly improves ActivityNet Captions while slightly reducing performance on YouCook2.Source: §C.2 · p. 18, Table 16 · p. 18

Related papers

Loading 2302.14115v2…