Source-linked AI summary
VidChapters-7M: Video Chapters at Scale
Antoine Yang, Arsha Nagrani, Ivan Laptev, Josef Sivic, Cordelia Schmid
TL;DR
Long-video chaptering could help users navigate to information of interest, but public datasets for studying it have been scarce. The paper introduces VidChapters-7M, a large web-scraped collection of user-chaptered videos and three related tasks. Models trained on the dataset transfer to dense video captioning, improving YouCook2 and ViTT results.
Problem
Long-video chaptering has been understudied because publicly released datasets are scarce, while existing datasets lack scalable, explicit chapter structure.
Source: §1 · pp. 1–2Method
The paper automatically scrapes user-annotated chapters from online videos to build VidChapters-7M and defines generation, boundary-conditioned title generation, and chapter-grounding tasks.
Source: §1 · p. 2Results
Models trained on VidChapters-7M transfer well to dense video captioning and largely improve the state of the art on YouCook2 and ViTT.
Source: §1 · p. 2Takeaways & Limitations
VidChapters-7M provides a benchmark for chaptering tasks and a resource for pretraining generic video-language models.
Source: §5 · p. 10VidChapters-7M inherits biases in video-category distribution from YT-Temporal-180M.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Segmenting long videos into chapters enables users to quickly navigate to the information of their interest. This important topic has been understudied due to the lack of publicly released datasets. To address this issue, we present VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters in total. VidChapters-7M is automatically created from videos online in a scalable manner by scraping user-annotated chapters and hence without any additional manual annotation. We introduce the following three tasks based on this data. First, the video chapter generation task consists of temporally segmenting the video and generating a chapter title for each segment. To further dissect the problem, we also define two variants of this task: video chapter generation given ground-truth boundaries, which requires generating a chapter title given an annotated video segment, and video chapter grounding, which requires temporally localizing a chapter given its annotated title. We benchmark both simple baselines and state-of-the-art video-language models for these three tasks. We also show that pretraining on VidChapters-7M transfers well to dense video captioning tasks in both zero-shot and finetuning settings, largely improving the state of the art on the YouCook2 and ViTT benchmarks. Finally, our experiments reveal that downstream performance scales well with the size of the pretraining dataset. Our dataset, code, and models are publicly available at https://antoyang.github.io/vidchapters.html.
1 Introduction
Long-video navigation is difficult and existing datasets do not provide scalable, explicit chapter structure. VidChapters-7M supplies user-chaptered videos for three chaptering tasks, and models trained on it transfer to dense video captioning benchmarks.
1 Introduction
- Long-video search is understudied, and available datasets lack scalable, explicit chapter structure: short-video caption sets are unsuitable, while ASR is only weakly related to visual content.Existing manually annotated moment-retrieval and dense-caption datasets are small and do not focus on creating explicit structure.Source: §1 · p. 2
- The dataset contains 817K videos and 7M user-annotated chapters, collected automatically from the Web.Videos average 23 minutes and include a starting timestamp and title for each chapter.Source: §1 · p. 2
- The paper defines chapter generation, title generation given ground-truth boundaries, and chapter grounding from an annotated title.The first task segments a video and generates a title per segment; the other two isolate title generation and temporal localization.Source: §1 · p. 2
- VidChapters-7M chapter-generation models transfer well to dense video captioning, largely improving YouCook2 and ViTT and showing performance that scales with chapter-dataset size.The paper also reports that pretraining with speech transcripts and chapter annotations significantly outperforms pretraining based only on speech transcripts.Source: §1 · p. 2
- The authors evaluate simple baselines and state-of-the-art video-text methods on the three tasks, which they report remain far from solved.They frame the tasks as involving long-video parsing and multimodal reasoning.Source: §1 · p. 2
2 Related Work
VidChapters-7M differs from existing video-language datasets through its long videos and user-annotated chapters aligned over time. Chapter generation combines full-video segmentation with short title generation, while chapter grounding focuses on locating chapter starts.
Large-scale vision-language datasets
- VidChapters-7M uses long videos with user-annotated chapters aligned over time, unlike short-video caption datasets and narrated-video datasets with aligned speech transcripts.It is collected from the Web through a scalable pipeline without expensive manual annotation; ChapterGen also uses user-annotated chapters but is much smaller and was not open-sourced at the time of writing.Source: §2 · p. 3
Video tasks
- Video chapter generation partitions the full video into chapters and generates a short natural-language title for each, unlike dense video captioning's event localization and captioning.Chapter grounding focuses on locating a chapter starting point, whereas temporal language grounding methods commonly localize a start-end window.Source: §2 · p. 3
3 VidChapters-7M: a large-scale dataset of user-chaptered videos
VidChapters-7M is built by collecting user-entered chapter metadata from online video descriptions and processing the videos for chapter modeling. The analysis reports the dataset’s scale, chapter and transcript characteristics, language and gender patterns, and title informativeness.
3.1 Data collection
- The collection procedure downloads descriptions for candidate videos, extracts user-entered chapter timestamps and titles, and yields 817K videos, or 0.9% of candidates.Candidates came from an extended pool of 92 million YouTube video IDs; automatically generated YouTube chapters do not appear in descriptions and are excluded.Source: §3.1 · p. 4
3.2 Data processing
- Processing adds time-aligned speech transcripts using Whisper-Large-V2 and WhisperX, and extracts visual features with CLIP ViT-L/14 at 1 FPS.The authors publicly release the transcripts and code for extracting visual features.Source: §3.2 · p. 4
3.3 Data analysis
- 817,076 videos contain 6,813,732 chapters and 4,894,855 distinct titles; videos average 8.3 chapters and last 1,354 seconds.Figure 3 presents distributions of chapters per video, chapter duration, title length, and video category. The random split contains 801K training, 8.2K validation, and 8.2K test videos.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
- 97.3% of videos contain speech transcripts, but chapters are much coarser: 8.3 titles versus 269.8 speech sentences per video on average.Speech sentences average 3.9 seconds and 11.5 words, compared with 142.0 seconds and 5.4 words for chapter titles.Source: §3.3 · p. 5
- Chapter titles and speech transcripts are predominantly English (92.9% and 93.9% of videos, respectively), while chapter-language counts include a long tail.The authors report female/male/non-binary gendered-word percentages of 19.7%/39.7%/40.7% in chapters and 11.6%/35.6%/52.8% in ASR.Source: §3.3 · p. 5
- Automated filtering flagged 0.70% of videos as visually NSFW, 0.04% for toxic chapter titles, and 0.17% for toxic ASR; flagged samples were retained.The authors caution that the automated filtering techniques are imperfect and harmful content may pass.Source: §3.3 · p. 5
- In a random sample of 100 videos, 83% of chapter titles related to video content, 14% referred only to structure, and 3% were unrelated.The assessment considered whether titles related to speech, vision, or raw audio, or referred only to video structure.Source: §3.3 · p. 6, Table 2 · p. 6
4 Experiments
The experiments evaluate chapter generation, title generation with known boundaries, chapter grounding, and transfer to dense video captioning. Training on VidChapters-7M improves chapter-task performance, while chapter pretraining also benefits dense-caption benchmarks.
4 Experiments
- The experiments cover full chapter generation, title generation with known boundaries, grounding, and dense-caption transfer; overall chapter evaluation uses SODA_c to penalize redundant chapters.The evaluation also reports caption metrics and separate chapter-localization measures.Source: §4 · p. 6
4.1 Video chapter generation
- VidChapters-7M-trained models outperform zero-shot baselines; Vid2Seq achieves the best overall generation and recall, PDVC the best precision, and speech plus vision yields Vid2Seq’s best performance.HowTo100M pretraining improves Vid2Seq, with greater benefit for vision-aware models than for the speech-only model.Source: Table 3 · p. 7, Table 4 · p. 7, §4.1 · p. 7
4.2 Video chapter generation given ground-truth boundaries
- With ground-truth boundaries, VidChapters-7M-trained models substantially outperform zero-shot models; Vid2Seq benefits from speech plus vision and HowTo100M pretraining.LLaMA underperforms the random baseline, while BLIP-2 slightly improves on it.Source: Table 5 · p. 8, §4.2 · p. 8
4.3 Video chapter grounding
- VidChapters-7M-trained Moment-DETR outperforms zero-shot baselines for start-time and start-end-window localization; ASR baselines find start times more readily than windows.Moment-DETR cannot handle speech inputs.Source: Table 6 · p. 8, §4.3 · p. 9
4.4 Transfer learning on dense video captioning
- 14.0 additional points in CIDEr are obtained on YouCook2 by pretraining Vid2Seq on VidChapters-7M after C4 + HowTo100M pretraining; combined pretraining gives best overall performance on YouCook2 and ViTT.The authors also report that larger VidChapters-7M pretraining subsets improve downstream dense-captioning performance.Source: Table 7 · p. 9, §4.4 · p. 10
- In zero-shot dense captioning, ASR plus chapter annotations outperforms baselines not trained on VidChapters-7M, while chapter-only training does not teach the model to use speech at inference.Chapter-only annotations work better than ASR annotations in the visual-only setting; increasing chapter pretraining data also benefits this setting.Source: Table 8 · p. 9, §4.4 · p. 10
5 Conclusion, Limitations, and Societal Impacts
VidChapters-7M provides a benchmark for video chapter tasks and a resource for pretraining video-language models. The paper reports qualitative gains from multimodal chapter generation and pretraining, while noting inherited category bias and potential societal risks.
5 Conclusion, Limitations, and Societal Impacts
- VidChapters-7M serves as both a benchmark for video chapter tasks and a resource for pretraining generic video-language models.The paper evaluates chapter generation with and without ground-truth boundaries, chapter grounding, and dense video captioning.Source: §5 · p. 10
- VidChapters-7M inherits biases in video-category distributions from YT-Temporal-180M [117].Source: §5 · p. 10
- Video chapter generation could facilitate harmful downstream applications such as surveillance, and models may reflect biases in YouTube videos.The authors urge care when deploying, analyzing, and building on these models.Source: §5 · p. 10
Appendix
- Appendix materials cover additional dataset examples, qualitative predictions, data analysis, implementation details, language-split results, and a datasheet.The datasheet presents hosting, licensing, and maintenance plans.Source: Appendix · p. 18
A Additional examples from VidChapters-7M
- Additional examples show chapters tracking visual events, conversations, or raw-audio events across varied video scenarios.Source: A Additional examples from… · p. 18
B Qualitative examples of video chapter generation and dense video caption prediction
- The speech-and-visual model can handle videos without ASR input and use speech cues; VidChapters-7M-pretrained captioning is more accurate and hallucinates less.These are qualitative comparisons against speech-only, visual-only, and non-pretrained variants.Source: B Qualitative examples of video… · p. 18
C Additional data analysis of VidChapters-7M
The additional analysis characterizes chapter-title patterns and text-video alignment in VidChapters-7M. It finds that chapter titles differ from ASR vocabulary and generally align with corresponding frames.
C Additional data analysis of VidChapters-7M
- 54.6% average CLIP cosine similarity, with fewer than 1% of chapters below 30%, indicates strong title-to-frame alignment in VidChapters-7M.Source: C Additional data analysis of… · p. 18
- Generic structural titles such as Intro, Outro, and Conclusion appear more than 10K times, while Unboxing, Review, and Tips are also common.Source: C Additional data analysis of… · p. 18
- The most common ASR words differ substantially from the most common chapter-title words, distinguishing the two text sources.Source: C Additional data analysis of… · p. 18
D Additional implementation details
The appendix supplies additional implementation details for ethical tagging and models used across chapter-generation and grounding tasks. It also presents qualitative examples comparing modalities and pretraining.
D.2 Video chapter generation and dense video captioning
- Qualitative examples show that multimodal chapter generation can generalize without ASR input and exploit speech cues, while VidChapters-7M pretraining improves caption accuracy and reduces hallucinations.Source: Figure 5 · p. 20, Figure 6 · p. 21
D.1 Tagging for ethical considerations
- The ethical tagging procedure flags videos with average NSFW visual or toxicity scores above 0.5.NSFW scores are computed per frame at 1 FPS; toxicity scores are computed per chapter title or ASR sentence.Source: §D.1 · p. 20
D.2 Video chapter generation and dense video captioning
- The appendix details LLaMA transcript summarization, BLIP-2 middle-frame prompting, and PDVC vocabulary adaptation and training settings.Source: §D.2 · pp. 20–21
- Vid2Seq is reimplemented in PyTorch, pretrained on HowTo100M due to computational limitations, then trained on VidChapters-7M and downstream captioning datasets.The training objectives and sequence lengths are described in the implementation details.Source: §D.2 · pp. 21, 23
D.3 Video chapter generation with ground-truth boundaries
- For chapter generation with ground-truth boundaries, Vid2Seq removes time-token weights and trains on single chapter titles paired with video clips.Source: §D.3 · p. 23
D.4 Video chapter grounding
- Moment-DETR is trained for video chapter grounding using 1/3 FPS frames, up to 1200 frames, and chapter-title queries.Source: §D.4 · p. 23
E Video chapter generation results split by language
The language-split results show that VidChapters-7M training benefits both English and German chapter generation, although German results are lower. HowTo100M pretraining helps both languages, while the multilingual variant trades a small English decline for a slight German gain.
E Video chapter generation results split by language
- VidChapters-7M training benefits both English and German video chapter generation, while German quantitative results are lower than English results.The reported global metrics are for the language-restricted test sets.Source: E Video chapter generation results… · p. 23, Table 9 · p. 24, Table 10 · p. 24
- HowTo100M pretraining improves results for English and German, despite HowTo100M being an English dataset.Source: E Video chapter generation results… · p. 23
- The mT5/mC4 Vid2Seq variant performs a bit worse on English but slightly better on German than the T5/C4 variant.Source: E Video chapter generation results… · pp. 23–24
F Datasheet for VidChapters-7M
VidChapters-7M was created to study video chapter generation, which helps users navigate to information of interest. It contains 817K YouTube videos.
F Datasheet for VidChapters-7M
- The datasheet organizes information about VidChapters-7M’s motivation, creation process, composition, and intended uses.Source: F Datasheet for VidChapters-7M · p. 24
Motivation
- The dataset was created to explore video chapter generation as a way to help users navigate to information of interest.Source: Motivation · p. 24
Composition
- Each instance represents a YouTube video, and the dataset contains 817K instances.Source: Composition · p. 25
Q7. Does the dataset contain all possible instances or is it a sample (not necessarily random)
VidChapters-7M is a selected subset of YouTube videos rather than a representative sample of the platform. Its content, acquisition, and verification have specific scope and access boundaries.
Q7. Does the dataset contain all possible instances or is it a sample (not necessarily random)
- 817K chaptered videos were selected from 92 million candidates, so VidChapters-7M does not fully represent YouTube.Source: Q7. Does the dataset contain all… · p. 25
- Each instance includes a video ID, URL, aligned ASR, and chapters; chapters serve as labels in this work.Source: Q7. Does the dataset contain all… · p. 25
- Chapter annotations are inherently noisy because YouTubers can write whatever chapters they want.Source: Q7. Does the dataset contain all… · p. 25
- The dataset distributes YouTube URLs rather than video files, and user deletions can make some URLs unavailable.Source: Q7. Does the dataset contain all… · pp. 25–26
Q20. Does the dataset contain data that might be considered sensitive in any way (e.g., data
- The authors could not manually verify all videos and chapters; they describe sensitive content as highly unlikely and use mitigation measures.Source: Q7. Does the dataset contain all… · p. 26, Q20. Does the dataset contain data… · p. 26
- The dataset does not explicitly identify subpopulations, though chapters may identify people and videos include celebrities or other YouTube-famous people.Source: Q7. Does the dataset contain all… · p. 26
Collection Process
- Video IDs came from YT-Temporal-180M’s providers, chapters were extracted from YouTube descriptions, and Whisper-Large-V2 produced ASR.Source: Collection Process · p. 27
Q23. What mechanisms or procedures were used to collect the data (e.g., hardware apparatus
- Data collection was fully automatic and required no human annotators; the collection code was checked against several dataset samples.Source: Q23. What mechanisms or procedures… · p. 27
Q26. Over what timeframe was the data collected? Does this timeframe match the creation
The videos were uploaded between 2005 and 2022 and collected in early 2023. The authors report no formal ethics review and no dataset impact analysis.
Q26. Over what timeframe was the data collected? Does this timeframe match the creation
- Videos were uploaded to YouTube between 2005–2022, and the dataset was collected in early 2023.Source: Q26. Over what timeframe was the data… · p. 27
- The authors did not conduct a formal institutional review board process, though they used filtering mechanisms to tag potentially problematic instances.Source: Q26. Over what timeframe was the data… · p. 27
- The authors did not notify video authors, citing the practical difficulty of contacting millions of users without contact information.Source: Q26. Over what timeframe was the data… · p. 28
- Users did not explicitly consent to dataset use, but could remove videos or request removal through an opt-out form.Source: Q26. Over what timeframe was the data… · p. 28
- The authors did not conduct a dataset impact analysis.Source: Q26. Over what timeframe was the data… · p. 28
Q35. Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucket-
Preprocessing converts chapter timestamps and extracts speech transcripts and visual features, while some instances receive ethics-focused tags. The dataset was used to train chapter-generation and related models.
Q35. Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucket-
- Preprocessing converts chapter timestamps to seconds, extracts speech transcripts and visual features, and tags some instances for ethical considerations.Source: Q35. Was any… · p. 28
- Raw descriptions and preprocessing code are released through the dataset website.Source: Q35. Was any… · p. 28
Uses
- The authors trained models for chapter generation, generation with ground-truth boundaries, chapter grounding, and transfer to dense video captioning.Source: Uses · p. 29
Q41. What (other) tasks could the dataset be used for?
The dataset may support other video-and-language tasks, including text-to-video retrieval, but its English-heavy composition and stated use restrictions bound future applications. The authors describe how it is distributed, maintained, updated, and extended.
Q41. What (other) tasks could the dataset be used for?
- VidChapters-7M could support other video-and-language tasks, such as text-to-video retrieval.Source: Q41. What (other) tasks could the… · p. 29
- The dataset is mainly composed of English videos, and its YouTube user demographics may influence the videos, languages, and ideas represented.Source: Q41. What (other) tasks could the… · p. 29
- The authors restrict use to non-commercial academic research and exclude tasks involving identification of people or decisions affecting people.Source: Q41. What (other) tasks could the… · p. 29
Distribution
- The dataset is publicly available as JSON/PICKLE annotation files; users download the videos themselves using the collection code.Source: Distribution · p. 29
Q48. Will the dataset be distributed under a copyright or other intellectual property (IP)
- The data and code use an MIT license, while video use is subject to YouTube API terms and commercial redistribution of videos is prohibited.Source: Q48. Will the dataset be distributed… · p. 30
Q50. Do any export controls or other regulatory restrictions apply to the dataset or to individual
- The authors report that no export controls or other regulatory restrictions apply.Source: Q50. Do any export controls or other… · p. 30
Q52. Who will be supporting/hosting/maintaining the dataset?
- The authors will maintain the dataset on Inria servers and Google Drive, update it annually, and remove videos requested through the opt-out form.Source: Q52. Who will be… · p. 30, Q55. Will the dataset be updated… · p. 30
Q58. If others want to extend/augment/build on/contribute to the dataset, is there a mechanism
- Anyone can extend the dataset using its collection code, and independent derivative works are permitted with attribution for non-commercial academic research.Source: Q58. If others want to… · p. 31