PaperlayerView exact revision on arXiv

Source-linked AI summary

Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid

arXiv:2302.14115v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Dense video captioning lacks large-scale annotated data and must jointly localize and describe events in long videos. Vid2Seq unifies both outputs as a time-token-augmented sequence and pretrains on narrated videos using speech-derived weak supervision. It achieves state-of-the-art results across dense, paragraph, and clip captioning benchmarks and generalizes to few-shot settings.

  • Problem

    Dense video captioning requires jointly localizing and describing many events in untrimmed videos, but existing annotated datasets are limited and expensive to create.

  • Method

    Vid2Seq generates captions and temporal boundaries in one sequence using special time tokens, pretrained with generative and denoising objectives on narrated-video transcripts.

  • Results

    Vid2Seq achieves state-of-the-art results across dense video captioning, video paragraph captioning, and video clip captioning benchmarks, while generalizing to few-shot settings.

  • Takeaways & Limitations

    Unlabeled narrated videos can provide weak supervision for a unified model that handles temporal localization and caption generation across multiple video-captioning tasks.

  • Takeaways & Limitations

    Speech transcripts are weak supervision because they may not faithfully describe the video and are often temporally misaligned.

Abstract

from arXiv · show

In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. The Vid2Seq architecture augments a language model with special time tokens, allowing it to seamlessly predict event boundaries and textual descriptions in the same output sequence. Such a unified model requires large-scale training data, which is not available in current annotated datasets. We show that it is possible to leverage unlabeled narrated videos for dense video captioning, by reformulating sentence boundaries of transcribed speech as pseudo event boundaries, and using the transcribed speech sentences as pseudo event captions. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset improves the state of the art on a variety of dense video captioning benchmarks including YouCook2, ViTT and ActivityNet Captions. Vid2Seq also generalizes well to the tasks of video paragraph captioning and video clip captioning, and to few-shot settings. Our code is publicly available at https://antoyang.github.io/vid2seq.html.

1. Introduction

Vid2Seq addresses dense video captioning by unifying temporal localization and caption generation, then pretraining on large-scale narrated videos using speech as weak supervision. The pretrained model achieves strong performance across dense, paragraph, and clip captioning benchmarks.

  • Motivation: Dense video captioning must localize and describe all events in untrimmed videos, making it harder than standard short-clip captioning.Its applications include large-scale video search and indexing.
  • Research gap: Existing approaches often use two stages or task-specific components and rely on manually annotated datasets of limited size.These constraints motivate a sequence-to-sequence model pretrained on Web-scale data.
  • Approach: Vid2Seq jointly predicts event captions and temporal boundaries as one sequence of text tokens interleaved with special time tokens.The model takes video frames and transcribed speech as multimodal inputs.
  • Pretraining: Unlabeled narrated videos provide pseudo event boundaries from speech sentence boundaries and pseudo captions from transcribed speech sentences.Pretraining combines generative prediction from visual inputs with denoising of masked speech spans.
  • Results: Vid2Seq achieves state-of-the-art performance on dense video captioning benchmarks and also performs strongly on paragraph and clip captioning tasks.The experiments additionally examine modality use, objectives, joint localization and captioning, model size, dataset scale, and few-shot settings.

2. Related Work

Prior dense video captioning methods commonly separate localization from captioning or jointly train specialized modules. Vid2Seq instead casts the task as language modeling over multimodal sequences with temporal tokens.

  • Dense video captioning: Most prior methods localize events before captioning, while newer methods jointly train localization and captioning modules.Some joint approaches still require task-specific components such as event counters.
  • Pretraining: Earlier pretraining work relied on manual annotations, ground-truth event proposals, or domain-specific text data, limiting its generality.Vid2Seq pretrains on a generic video corpus without manual annotation.
  • Unified formulation: Vid2Seq represents captions and temporal boundaries in a single sequence using visual tokens, time tokens, and text tokens.This unifies event localization and captioning within a sequence-to-sequence formulation.

3. Method

Vid2Seq formulates dense video captioning as multimodal sequence generation, jointly modeling event text and temporal boundaries. It is pretrained on narrated videos using speech-derived weak supervision and complementary generative and denoising objectives.

  • Method: Dense captioning requires modeling relationships among multiple events while localizing and describing each event in long, untrimmed videos.The method targets dependencies such as predicting later events from earlier events.
  • Sequence construction: Vid2Seq constructs input and output sequences with natural-language tokens and special time tokens representing temporal event locations.The output sequence contains each event’s textual description and timestamps.
  • Architecture: The multimodal encoder-decoder embeds video frames and timestamped speech, then autoregressively predicts event boundaries and captions.The visual encoder combines a frozen spatial encoder with a temporal transformer, while the decoder cross-attends to visual and speech embeddings.
  • Pretraining data: Narrated videos supply weak supervision because speech sentences and timestamps can substitute for manually annotated event captions and boundaries.Transcripts may be visually ungrounded, temporally misaligned, and much denser than standard event annotations.
  • Pretraining objectives: The generative objective predicts transcribed speech from visual inputs without text encoder input, avoiding text-only shortcuts.The denoising objective masks spans of transcribed speech and trains the model to recover them, including the text encoder.
  • Downstream use: The generic language-modeling formulation also supports paragraph captioning by removing time tokens and adapts to clip captioning with the same finetuning and inference recipe.Beam search is used because it improves captioning quality over argmax and nucleus sampling.

4. Experiments

Experiments evaluate Vid2Seq through dataset setup, ablations, state-of-the-art comparisons, and qualitative results. The model improves dense captioning broadly, while localization performance varies by dataset and design choice.

  • Experimental setup: Experiments use YT-Temporal-1B for pretraining and evaluate on YouCook2, ViTT, and ActivityNet Captions, with additional paragraph and clip-captioning benchmarks.The setup also includes ablations, few-shot dense captioning, and qualitative evaluation.
  • Ablation studies: Untrimmed narrated-video pretraining integrates transcript sentence boundaries as temporal information through special time tokens.Table 1 isolates the impact of untrimmed inputs and time-token integration during pretraining.
  • Ablation studies: Joint captioning and localization improves localization over a localization-only variant, while denoising loss benefits multimodal reasoning.The authors attribute the localization gain to contextualizing noisy transcript timestamps with speech semantics.
  • Ablation studies: T5-Base outperforms T5-Small, and pretraining on HowTo100M generalizes well, achieving the best YouCook2 results because of domain similarity.The experiments also examine pretraining-data scale and language-model size.
  • Comparison to the state of the art: Vid2Seq sets the state of the art on YouCook2, ViTT, and ActivityNet Captions, improving CIDEr by 18.2 and 0.8 points over PDVC on YouCook2 and ActivityNet Captions.For event localization, Vid2Seq outperforms prior work on YouCook2 and ViTT but underperforms PDVC and UEDVC on ActivityNet Captions.
  • Generalization and qualitative results: Vid2Seq generalizes to video paragraph and clip captioning, outperforming prior methods on the reported paragraph-captioning datasets and improving over prior methods on MSR-VTT and MSVD.Qualitative predictions show meaningful boundaries and captions that can differ considerably from transcribed speech, highlighting visual input use.

5. Conclusion

The conclusion presents Vid2Seq as a single-sequence visual language model and identifies weakly supervised narrated-video pretraining as its central training strategy. It reports state-of-the-art results across dense, paragraph, and clip video captioning benchmarks and points to broader extensions.

  • Conclusion: Vid2Seq generates one sequence interleaving caption tokens and time tokens from transcribed speech and video inputs.This design performs dense video captioning while representing event timestamps and text together.
  • Conclusion: Unlabeled narrated videos provide weak supervision through transcribed speech sentences and their corresponding temporal boundaries.The conclusion identifies this large-scale pretraining strategy as a central contribution.
  • Conclusion: Vid2Seq achieves state-of-the-art results on dense event captioning datasets and performs well on video paragraph and standard video clip captioning benchmarks.The paper also reports performance in few-shot dense captioning settings in the supplied abstract context.
  • Conclusion: The sequence-to-sequence design is presented as extensible to temporally grounded video question answering and temporal action localization.These are proposed directions rather than evaluated conclusions in the supplied passage.

A. Qualitative examples of dense video captioning predictions

Qualitative examples show Vid2Seq producing dense captions and event boundaries across cooking, sports, and leisure scenarios, with or without transcribed speech input. They also expose a hallucination failure case.

  • Qualitative examples: Vid2Seq predicts meaningful dense captions and event boundaries across diverse scenarios, including cooking instructions and human sports or leisure activities.The examples include settings both with and without transcribed speech input.
  • Qualitative examples: A failure case shows hallucinated events that are not visually grounded, such as a man removing his hat toward the camera.This example marks a concrete limitation of the qualitative predictions.

B. Experimental setup

The appendix section supplements the paper’s experimental setup with dataset information and implementation details.

  • Experimental setup: The supplementary experimental-setup section covers dataset descriptions and additional implementation details.These materials are organized into Sections B.1 and B.2.

B.1. Datasets

The paper evaluates on dense video captioning datasets spanning instructional, general human-activity, and open-domain video clips, with varied durations and annotation schemes.

  • Pretraining data: YT-Temporal-1B contains 18.821M unlabeled narrated videos spanning about 150 years of video content across broader domains than HowTo100M.HowTo100M contains 1.221M narrated instructional videos covering about 15 years.
  • Dense captioning benchmarks: ViTT contains 7,672 videos split across training, validation, and testing, averaging 250s and annotated with 7.1 temporally localized short tags.The passage specifies splits of 5,476, 1,102, and 1,094 videos, respectively.
  • Dense captioning benchmarks: YouCook2 contains 1,790 untrimmed cooking-procedure videos averaging 320s, while ActivityNet-Captions contains 14,934 human-activity videos averaging 120s.ActivityNet-Captions averages 3.7 temporally localized sentences per video.
  • Clip captioning benchmarks: MSR-VTT provides 10,000 open-domain clips lasting 10–30 seconds, each with 20 manually annotated descriptions.The dataset is split into 6,513 training, 497 validation, and 2,990 testing videos.
  • Clip captioning benchmarks: MSVD contains 1,970 open-domain clips lasting 10–30 seconds, with roughly 40 manually annotated captions per clip.The dataset is split into 1,200 training, 100 validation, and 670 testing videos.

B.2. Implementation details

The implementation uses transformer-based visual and textual components with fixed sequence lengths, Adam optimization, scheduled learning rates, and regularization during fine-tuning.

  • Architecture: The visual temporal transformer, text encoder, and text decoder each use 12 layers, 12 heads, 768-dimensional embeddings, and 2048-dimensional MLP hidden states.These architectural settings are shared across the three components.
  • Architecture: Pretraining truncates or pads encoder and decoder sequences to 1000 tokens, while fine-tuning uses 1000-token and 256-token limits.The passage specifies L = S = 1000 during pretraining and S = 1000, L = 256 during fine-tuning.
  • Decoding and regularization: Inference tracks the top 4 beam-search sequences with length normalization of 0.6, while training applies random temporal cropping, label smoothing of 0.1, and dropout of 0.1.These choices cover decoding, augmentation, and regularization.
  • Optimization: Training uses Adam with β = (0.9, 0.999), a 1e−4 warmed-up constant pretraining learning rate, and a 3e−4 warmup-plus-cosine-decay fine-tuning schedule.Fine-tuning warms up for 10% of iterations and decays during the remaining 90%.
  • Optimization: Fine-tuning uses batches of 32 videos across 16 TPU v4 chips, with dataset-specific schedules ranging from 5 to 40 epochs.The passage lists 40 epochs for YouCook2, 20 for ActivityNet Captions and ViTT, 5 for MSR-VTT, and 10 for MSVD.

C. Experiments

The additional experiments examine pretraining and architectural choices, including long narrated videos, data and backbone scale, time tokenization, sequence construction, and language-model initialization.

  • Supplementary experiments: The supplementary experiments first evaluate the importance of pretraining in the few-shot setting before conducting fully supervised ablations.The ablations cover multiple pretraining, tokenization, sequence, positional-embedding, and initialization factors.
  • Qualitative results: ActivityNet Captions examples include successful predictions and a failure case in which Vid2Seq hallucinates an event that is not visually grounded.The failure example is “one man hats off to the camera,” and these videos contain no transcribed speech.
  • Ablation studies: The ablations vary pretraining on long narrated videos, the pretraining dataset, and the size of the visual backbone.They also examine time-tokenization choices and the number of time tokens.

C.1. Importance of pretraining in few-shot settings

The experiments assess pretraining under reduced downstream supervision and report that it remains important in few-shot dense event captioning.

  • Few-shot evaluation: Table 10 reports the impact of pretraining on few-shot dense event captioning by fine-tuning Vid2Seq on a small fraction of downstream data.The supplied passage identifies the table’s evaluation focus but does not provide its individual metric values.
  • Related ablations: The supplementary ablations separately examine the maximum number of narration sentences covered by cropped videos, pretraining data, and visual-backbone model size.Unrestricted cropping can sample arbitrarily long videos.
  • Few-shot evaluation: Vid2Seq pretraining has considerable importance when fine-tuning with only a smaller fraction of the downstream training dataset.The few-shot setting is defined by using less than 100% of downstream training data.
  • Few-shot evaluation: Non-zero performance is achieved with only 1% of the downstream training dataset when using the pretrained Vid2Seq model.The passage attributes this result to the pretraining method and identifies rows 1 and 2 of Table 10.

C.2. Additional ablation studies

Additional ablations show that Vid2Seq benefits from long narrated-video pretraining, stronger visual features, relative time tokens, and temporal positional embeddings. Sequence ordering has little effect overall, while language-model initialization matters less after narrated-video pretraining.

  • Pretraining on long narrated videos: Longer narrated videos outperform crops limited to 10 narration sentences, demonstrating that Vid2Seq benefits from long-context pretraining.The default sampling can span hundreds of transcribed speech sentences.
  • Visual features: CLIP-pretrained visual features outperform ImageNet-pretrained features, and scaling from ViT-B/16 to ViT-L/14 brings additional improvements.Both comparisons use the same ViT-B/16 backbone for the pretraining comparison before scaling backbone size.
  • Time tokenization and number of time tokens: Relative time tokens with N = 100 yield the best dense video captioning results.The ablation compares relative and absolute timestamp representations across different numbers of time tokens.
  • Sequence construction: Placing time tokens before or after each event sentence produces similar results, with the default ordering slightly improving event localization F1 Score.The sequence inserts each segment’s start and end time tokens before its corresponding text sentence by default.
  • Temporal positional embeddings: Temporal positional embeddings improve performance by communicating temporal information from the visual stream to Vid2Seq.The comparison is reported in Table 15.
  • Language model initialization and pretraining: After narrated-video pretraining, initializing the language model from text-only pretrained weights has little importance, despite helping without narrated-video pretraining.The text-only initialization effect is reported across rows 1–4 of Table 16.
Loading 2302.14115v2…