PaperlayerView exact revision on arXiv

Source-linked AI summary

Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, Sergey Tulyakov

arXiv:2402.19479v1cs.CV

TL;DR

High-quality video-text data is difficult to collect because manual labeling is time-consuming and video content is temporally varied. Panda-70M automatically splits videos into coherent clips, combines multimodal teacher captions, and selects annotations with fine-grained retrieval; pretraining on the dataset benefits three downstream tasks.

  • Problem

    High-quality video-text data is difficult to obtain because videos require time-consuming manual inspection and contain temporally varying, multi-scene content that metadata may not precisely describe.

  • Method

    Panda-70M splits 3.8M high-resolution videos into semantically consistent clips, generates multiple captions from multimodal teachers, and uses fine-grained retrieval trained on 100K human-selected examples to choose annotations.

  • Results

    Pretraining on Panda-70M benefits video captioning, video and text retrieval, and text-to-video generation, while jointly collecting teacher captions yields good captions for 84.7% of videos.

  • Takeaways & Limitations

    Panda-70M provides a large-scale, high-resolution dataset with semantically coherent video clips and caption annotations for multimodal learning.

  • Takeaways & Limitations

    The dataset is dominated by vocal-intensive videos from HD-VILA-100M and emphasizes semantically consistent short clips, limiting content diversity and potentially constraining long-video generation and dense captioning.

Abstract

from arXiv · show

The quality of the data and annotation upper-bounds the quality of a downstream model. While there exist large text corpora and image-text pairs, high-quality video-text data is much harder to collect. First of all, manual labeling is more time-consuming, as it requires an annotator to watch an entire video. Second, videos have a temporal dimension, consisting of several scenes stacked together, and showing multiple actions. Accordingly, to establish a video dataset with high-quality captions, we propose an automatic approach leveraging multimodal inputs, such as textual video description, subtitles, and individual video frames. Specifically, we curate 3.8M high-resolution videos from the publicly available HD-VILA-100M dataset. We then split them into semantically consistent video clips, and apply multiple cross-modality teacher models to obtain captions for each video. Next, we finetune a retrieval model on a small subset where the best caption of each video is manually selected and then employ the model in the whole dataset to select the best caption as the annotation. In this way, we get 70M videos paired with high-quality text captions. We dub the dataset as Panda-70M. We show the value of the proposed dataset on three downstream tasks: video captioning, video and text retrieval, and text-driven video generation. The models trained on the proposed data score substantially better on the majority of metrics across all the tasks.

1. Introduction

Panda-70M addresses the difficulty of obtaining high-quality video-text data by automatically combining multimodal captioning teachers with fine-grained retrieval. The resulting dataset contains 70M captioned clips and supports multiple downstream tasks.

  • Motivation: Video-text annotation is harder than image-text annotation because videos require viewing temporal, multi-scene content, while subtitles and metadata may be broad or misaligned.Existing ASR-based annotations can miss the main content and actions shown in videos.
  • Contribution: Panda-70M contains 70M high-resolution video clips with captions averaging 13.2 words.The dataset is automatically annotated because manually labeling 70M videos would be prohibitively expensive.
  • Motivation: 84.7% of videos have at least one good caption when captions from multiple cross-modality models are jointly collected, versus at most 31% for any single model.The evaluation is based on human judgments of captions generated for video samples.
  • Method: The pipeline splits 3.8M high-resolution long videos into semantically consistent clips, generates candidate captions with multimodal teachers, and uses a retrieval model to select annotations.The teachers use inputs including video descriptions, subtitles, frames, and video content; a 100K human-selected subset supervises caption selection.
  • Results: Pretraining on Panda-70M benefits video captioning, video-text retrieval, and text-to-video generation.A knowledge-distilled student can outperform any teacher model by more than 7.7% preference ratio, with further gains from text inputs such as descriptions and subtitles.

2. Related Work

The paper situates Panda-70M among large-scale vision-language datasets and multimodal annotation models. It combines existing vision-language teachers with fine-grained retrieval for video caption annotation.

  • Vision-Language Datasets: Large image-text datasets have enabled powerful image foundation models, whereas manually annotated video-language datasets remain costly and limited in scale.Prior video-language datasets address tasks including action recognition, video understanding, VQA, and retrieval.
  • Naming: Panda is named by analogy to Panda Po, who learns from multiple martial arts teachers.The name reflects the dataset’s use of several vision-language models for annotation.
  • Video Annotation through Multi-modal Models: Panda-70M uses BLIP-2, MiniGPT-4, Video-LLaMA, VideoChat, and Unmasked Teacher for multimodal captioning and caption selection.The models receive combinations of frames, video, video descriptions, subtitles, and prompts; Unmasked Teacher is fine-tuned for fine-grained retrieval.

3. Methodology

Panda-70M constructs semantically coherent clips, captions them with diverse cross-modality teachers, and selects annotations using fine-grained retrieval. A distilled student model addresses annotation cost while leveraging multimodal inputs.

  • Dataset Construction: 70.8M semantically coherent clips are created from 3.8M high-resolution long videos through semantics-aware splitting and caption selection.The pipeline splits source videos, generates candidate captions with multiple teachers, and selects the most accurate caption using a finetuned retrieval model.
  • Semantics-aware Video Splitting: The splitting algorithm balances semantic consistency with clip duration by detecting shot boundaries, stitching incorrectly separated adjacent clips, and using frame embeddings.Max Running LPIPS measures the largest perceptual change within a clip; the proposed splitting outperforms subtitle-alignment splitting on semantic consistency while retaining longer clips than vanilla shot detection.
  • Captioning with Cross-Modality Teachers: Multiple cross-modality teachers combine video, text metadata, subtitles, and frames because different teachers caption different kinds of videos.The teacher pool includes video VQA, image captioning, and related models using different pretraining weights and input modalities.
  • Captioning with Cross-Modality Teachers: 17.85% is the highest selective rate achieved by one teacher, showing that no single model consistently provides the best caption across diverse videos.A user study asks participants to select the best caption from eight candidates, and the strongest individual teacher is BLIP-2 with opt6.7b.
  • Multimodal Student Captioning Model: A student captioning model distills multiple teachers to reduce the cost of running 8 + 1 models per clip and uses visual and text branches for multimodal inputs.The student architecture includes a vision branch and a text branch for subtitles and metadata; its zero-shot captioning evaluation is reported on MSR-VTT and MSVD.

4. Experiments

Experiments evaluate Panda-70M pretraining across video captioning, video-text retrieval, and text-to-video generation, with improvements over official baselines across these tasks.

  • 4.1. Video Captioning: 17.7% and 18.5% B-4 improvements are achieved on MSR-VTT and MSVD, respectively, with Panda-2M Video-LLaMA pretraining.The model uses the same backbone as the official comparison, while the student additionally uses metadata and subtitles.
  • 4.1. Video Captioning: Panda-70M-generated captions better align with video content than original Video-LLaMA captions, which contain irrelevant and generic details.The qualitative comparison visualizes annotations and predictions from Video-LLaMA and student models with video-only or video-and-text inputs.
  • 4.2. Video and Text Retrieval: Panda-5M pretraining improves zero-shot text-to-video retrieval R@1 by 7.0%, 0.6%, and 4.9% on MSR-VTT, DiDeMo, and MSVD.It also outperforms the official checkpoint in finetuned retrieval settings and existing methods pretrained on more than 100M vision-text pairs.
  • 4.3. Text-to-Video Generation: 77.4 lower FVD on UCF101 is obtained with Panda-2M pretraining, which outperforms the official AnimateDiff weight on both reported metrics.The comparison uses AnimateDiff weights trained on 2.5M videos and evaluates FVD on UCF101 and CLIPSim on MSR-VTT.
  • 4.3. Text-to-Video Generation: Panda-2M generation weights produce more meaningful motion and photorealistic appearance without a watermark.This qualitative observation accompanies the reported quantitative superiority over the official AnimateDiff weight.

5. Conclusion and Limitations

The paper concludes that Panda-70M provides high-resolution, semantically coherent captioned clips for three downstream tasks, while identifying vocal-content and clip-duration limitations.

  • 5. Conclusion: Panda-70M contains high-resolution, semantically coherent video samples with caption annotations and supports captioning, retrieval, and text-to-video generation.The dataset contains 70M captioned video clips and uses multimodal information including descriptions, subtitles, and static frames.
  • 5. Limitations: Most Panda-70M samples are vocal-intensive because the videos are collected from HD-VILA-100M, limiting coverage of nonvocal content.The authors identify collecting more nonvocal videos as an important extension because the annotation pipeline itself does not require subtitles.
  • 5. Limitations: Semantically consistent fine-grained clips improve caption specificity but limit within-video content diversity and reduce average duration.The authors note potential harm to long video generation and dense video captioning, motivating future long-video, dense-caption datasets.

Supplementary Material

The supplementary material describes the video-splitting algorithm as a two-stage process that creates and then merges semantically coherent clips.

  • Supplementary Material: The video-splitting algorithm uses separate splitting and stitching stages to divide long videos into semantically coherent clips.Further implementation details are provided in the supplementary subsections for each stage.

A.1. Stage1: Splitting based on Shot Boundary Detection

Stage 1 detects shot boundaries and adds post-processing for transitions and unedited footage, then filters clips using duration, motion, and semantic criteria.

  • A.1. Stage1: Splitting based on Shot Boundary Detection: PySceneDetect ContentDetector identifies initial cuts with a threshold of 25 and a minimum scene length of 15 frames.The method is followed by post-processing to address complex transitions and semantic changes without explicit cut-scenes.
  • A.1. Stage1: Splitting based on Shot Boundary Detection: Clips without detected cut-scenes are recursively split into five-second segments to expose semantic changes in unedited footage.ImageBind features near clip beginnings and ends are then used to remove clips with dramatically different boundary content.
  • A.1. Stage1: Splitting based on Shot Boundary Detection: Adjacent clips are merged when their semantic feature distance is at most 0.6, reducing consecutive fragments with the same content.The merging rule operates on neighboring clips in sequence.
  • A.1. Stage1: Splitting based on Shot Boundary Detection: Post-processing excludes clips shorter than 2 seconds, clips with only slight motion, and content beyond the first 60 seconds of longer videos.It also retains semantically different clips using average ImageBind representations.
  • A.1. Stage1: Splitting based on Shot Boundary Detection: The captioning pipeline uses multiple cross-modality teachers that combine visual inputs with subtitles or metadata, selecting eight models from an initial pool of 31.The teacher pool includes video VQA, image VQA, and image-captioning models, with inputs such as video, static frames, subtitles, and metadata.

B.3. Selecting 8 Captioning Models based on a Human Evaluation

The authors select a compact set of captioning teachers through human evaluation, measuring how often models produce good captions and choosing models that jointly cover videos.

  • Model selection: 8 selected teacher models are highlighted in Figure 10 alongside each model’s good-caption ratio and the “All Bad” ratio.Bars are colored by input information, while the selected teachers are marked in gray.
  • Coverage results: 30.8% is the maximum good-caption ratio achieved by any single captioning model.This motivates combining complementary models rather than relying on one teacher.
  • Coverage results: 84.7% of videos receive at least one good caption when all 31 captioning models are used.The corresponding “All Bad” ratio is 15.3%.
  • Coverage results: 76.8% of videos receive a good caption from the 8 selected teachers, comparable to all 31 models while requiring less computation.The selected subset is intended to jointly cover good captions across video types.
  • Dataset analysis: Figure 11 shows the distribution of source teacher models for captions in Panda-70M.The figure summarizes which selected teachers supply the final captions.

C. Details of Fine-Grained Video-to-Text Retrieval: Dataset, Training, and Inference

Because generic retrieval models cannot reliably distinguish among several highly relevant captions, the authors create a fine-grained retrieval setting tailored to selecting the best caption.

  • Retrieval challenge: Generic retrieval models cannot reliably choose the best caption from 8 teacher-generated candidates.The candidates are all highly relevant to the video, so optimal selection requires discerning subtle caption differences.
  • Supervision: The authors manually annotate a subset of video samples by selecting the best caption for each video.This supervision targets the paper’s fine-grained retrieval task rather than generic video-text matching.
  • Selection objective: The resulting retrieval model is used to select one best caption as the annotation for each video.This preserves the captioning algorithm’s use of multiple cross-modality models while producing a single dataset annotation.

C.1. Collection of Dataset

The dataset collection procedure combines human preference labels, retrieval-model finetuning, and multimodal captioning outputs to select annotations across Panda-70M.

  • Collection of supervision: 100K video samples are manually labeled by choosing the caption that is most faithful and descriptive of each clip.Annotators are instructed to select the textual summary that best describes the video content.
  • Collection of supervision: 12,064 videos marked “All Bad” are filtered out, leaving 86,131 training and 1,805 validation videos.The “All Bad” option is used when every candidate caption contains wrong information.
  • Retrieval training: Unmasked Teacher is finetuned with video-text contrastive and matching losses, using hard negative mining for fine-grained caption discrimination.The model uses ViT-L/16 and BERTlarge initialized from weights pretrained on 25M image-text and video-text pairs.
  • Inference and analysis: The finetuned UMT retrieves the best caption for all 70M videos in Panda-70M.Figures report the selected-caption source distribution, caption-length distribution, and a word cloud of 100K annotations.
  • Inference and analysis: 89.6% of Panda-70M samples have UMT matching scores higher than 0.43.A score above 0.43 usually represents a strong association between the video and caption.
  • Student model: The student captioning model combines a vision branch with a text branch for subtitle and metadata inputs.The architecture uses video representations, text representations, and an LLM to predict captions.

D.2. Training Details

Training uses video-caption pairs together with optional metadata and subtitles, while optimization follows a specified AdamW schedule on the full Panda-70M dataset.

  • Training inputs: The training data pairs each video with a caption and additional metadata or subtitle text embedded in the prompt.Part of the text input is randomly dropped so the model can operate with either video-only or video-text inputs.
  • Optimization: The model is optimized with AdamW using a learning rate warmed from 1e−6 to 1e−4 over 2,500 steps, then cosine-annealed to 5e−5.The configuration uses β = [0.9, 0.99] and weight decay 0.05.
  • Compute: Training runs for 300K steps on Panda-70M with batch size 48 across 48 Nvidia A100 GPUs with 80GB memory.The model is trained on the whole dataset.

E. Visualization of Panda-70M Dataset

This section visualizes video-text pairs in Panda-70M by category.

  • Video-text pairs in Panda-70M are organized and visualized by category.
Loading 2402.19479v1…