PaperlayerView exact revision on arXiv

Source-linked AI summary

MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions

Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba Heilbron, Chen Zhao, Silvio Giancola, Bernard Ghanem

arXiv:2112.00431v2cs.CVcs.AI

TL;DR

Video-language grounding datasets have limited assessment of their fitness and can contain biases that encourage overfitting. MAD addresses this gap by aligning professional movie audio descriptions with long-form videos, producing a large and diverse grounding benchmark. Its long-form setting makes short-moment localization more challenging, while experiments show that scale can offset annotation noise and strong baselines struggle on the task.

  • Problem

    Existing video-language datasets have received limited assessment for grounding fitness, and recent work identifies significant dataset limitations and hidden biases.

  • Method

    MAD crawls and aligns professional audio descriptions from mainstream movies with long-form videos, using a scalable automatic annotation pipeline.

  • Results

    MAD contains more than 384K grounded sentences across more than 1.2K video hours, while baseline results show that long-form grounding remains difficult.

  • Takeaways & Limitations

    MAD provides a large-scale benchmark requiring short temporal moments to be localized in diverse long-form videos without relying on trivial temporal location priors.

  • Takeaways & Limitations

    MAD’s videos cannot be publicly released because of copyright constraints, although the authors provide necessary features for reproducibility.

Abstract

from arXiv · show

The recent and increasing interest in video-language research has driven the development of large-scale datasets that enable data-intensive machine learning techniques. In comparison, limited effort has been made at assessing the fitness of these datasets for the video-language grounding task. Recent works have begun to discover significant limitations in these datasets, suggesting that state-of-the-art techniques commonly overfit to hidden dataset biases. In this work, we present MAD (Movie Audio Descriptions), a novel benchmark that departs from the paradigm of augmenting existing video datasets with text annotations and focuses on crawling and aligning available audio descriptions of mainstream movies. MAD contains over 384,000 natural language sentences grounded in over 1,200 hours of videos and exhibits a significant reduction in the currently diagnosed biases for video-language grounding datasets. MAD's collection strategy enables a novel and more challenging version of video-language grounding, where short temporal moments (typically seconds long) must be accurately grounded in diverse long-form videos that can last up to three hours. We have released MAD's data and baselines code at https://github.com/Soldelli/MAD.

1. Introduction

MAD addresses hidden biases and mismatched data-collection assumptions in video-language grounding by aligning professional movie audio descriptions with long-form films. The resulting benchmark combines diverse, densely grounded language with short temporal moments in videos averaging over 110 minutes.

  • 1. Introduction: Existing grounding datasets often use short, single-scene videos with language covering most of the video and temporally biased anchors, limiting assessment of grounding progress.Many were originally collected for captioning or retrieval rather than dense temporal localization.
  • 1. Introduction: MAD uses professional, grounded audio descriptions to create an authentic untrimmed-video setup for long-form movie grounding.The collection strategy avoids standard crowd-sourced annotation pipelines and grounds descriptions directly in movies.
  • 1. Introduction: MAD videos average over 110 minutes, while annotations cover short, approximately 4.1-second moments with low video coverage and diverse vocabulary.The dataset’s grounded sentences are distributed throughout the video, requiring finer video-language understanding.
  • 1. Introduction: Long-form grounding reduces trivial temporal-location priors, increases the number of segment proposals, and makes efficient inference and training necessary.These challenges are linked to applications such as live-stream analysis and retrieval in large video collections.
  • 1. Introduction: MAD contains more than 384K natural-language sentences anchored on more than 1.2K hours of video, collected through a scalable audio-description pipeline.The pipeline transcribes professional audio descriptions and automatically removes sentences associated with actors’ speech.

2. Related work

Prior grounding benchmarks largely adapt datasets built for other tasks, producing constrained domains, short videos, and simplified localization choices. MAD instead spans diverse movie content and supports grounding in long-form videos using audio descriptions.

  • 2. Related work: Existing benchmarks often repurpose datasets for activity localization or action recognition, restricting visual content and language to specific actions, objects, verbs, and nouns.ActivityNet-Captions and Charades-STA are identified as common examples of this paradigm.
  • 2. Related work: MAD spans more than 22 genres across 90 years of cinema, covering diverse actions, locations, scenes, and linguistic content unlike narrowly scoped alternatives.TACoS, for example, is small and limited to cooking actions recorded with a static camera.
  • 2. Related work: DiDeMo constrains grounding to trimmed videos of at most 30 seconds and 21 discrete proposals, whereas MAD supports videos lasting up to 3 hours.The long-form setup prevents grounding from being reduced to selecting among a small fixed set of proposals.
  • 2. Related work: Many state-of-the-art grounding methods were designed for short videos and assume the entire video can be provided during prediction, an assumption incompatible with MAD’s long-form setting.MAD creates room for methods that address proposal generation, context modeling, and multimodal fusion under longer-video constraints.
  • 2. Related work: QuerYD also uses audio descriptions but targets short-form YouTube videos averaging under 5 minutes and relies on volunteer narrators.Its authors likewise observed that audio descriptions are more visually grounded and descriptive than earlier annotations.

3. Collecting the MAD Dataset

MAD combines professionally produced audio descriptions with scalable alignment and transcription to create a large, long-form grounding dataset. Its statistics and analyses show unusually broad vocabulary, short moments, and reduced temporal-location bias compared with existing benchmarks.

  • Dataset statistics: MAD contains 1,207 hours of video and 384.6K language queries, with 110.77-minute average videos and 4.1-second average grounding moments.
  • Data collection: The training pipeline crawls audio descriptions, aligns them to movies by maximizing audio cross-correlation, and verifies consistent delays across 20 temporal windows.
  • Data collection: After alignment, speech-to-text timestamps ground descriptions to the video, while subtitle timestamps help remove actors’ speech from the retained annotations.
  • Dataset statistics: MAD’s training, validation, and test splits contain 280.5K, 32.1K, and 72.0K sentences, totaling 61.4K unique words and exceeding other benchmarks in vocabulary diversity.
  • Bias analysis: Existing datasets show start/end location peaks, whereas MAD has no preferred temporal start or end, reducing a prominent grounding bias.

4. Experiments

MAD evaluates video-language grounding on untrimmed, long-form movies, where short moments must be localized among many proposals. On this setup, baseline performance deteriorates as the evaluation window expands, exposing limitations of methods developed for short videos.

  • Grounding Performance on MAD: At R@100-IoU=0.1, Random Chance reaches only 8.47%, while CLIP and VLG-Net approach 50%; at R@1-IoU=0.5, their relative improvement is two orders of magnitude.These results quantify the difficulty of grounding within the large proposal pool of long-form videos.
  • Grounding Performance on MAD: CLIP performs best at IoU=0.1 for K={1, 5, 10}, whereas VLG-Net is comparable or better only under the stricter IoU=0.5 criterion.CLIP is evaluated zero-shot, while VLG-Net is trained for grounding.
  • Short-video Setup: In the three-minute setup, Random Chance rises from 0.44% to 15.69% for R@5-IoU=0.1, and VLG-Net achieves the best performance in all cases.Reducing proposal counts makes the short-video configuration less challenging than MAD’s original long-form setting.
  • The Challenges of Long-form Video Grounding: Performance steadily drops for both CLIP and VLG-Net as evaluation windows grow from 30 seconds to entire movies averaging two hours.The trend is reported for R@{1, 5}-IoU=0.5.
  • The Challenges of Long-form Video Grounding: VLG-Net’s short-video advantage does not transfer to long-form movies, indicating that current grounding methods are not ready for MAD’s setting.The paper attributes the challenge to long sequences and the need to compare queries against all movie segments.

5. Ablation Study

The ablation study tests how MAD data scale and automatic annotation noise affect grounding and retrieval. Larger MAD training sets recover or improve performance despite noise, and also benefit the related retrieval task.

  • Improving Grounding Performance with MAD Data: Using 100% of MAD data yields a 20% relative improvement for R@5-IoU=0.5, while progressively adding MAD samples to LSMDC-G steadily increases performance.At 32% of MAD training data, performance drops, but the full dataset overcomes the issue associated with noisy automatic annotations.
  • Improving Grounding Performance with MAD Data: Training with the complete MAD set produces grounding performance comparable to training only on clean LSMDC-G data, despite MAD’s automatic annotation noise.The smaller 32% MAD regimen instead shows a performance drop.
  • Improving Grounding Performance with MAD Data: Table 5 compares VLG-Net trained on manually curated LSMDC-G data with regimens using automatically collected MAD training data on the same LSMDC-G test split.The study varies the amount and composition of training data while holding the evaluation split fixed.
  • Improving Retrieval Performance with MAD Data: On text-to-video retrieval, training with all of LSMDC16 or MAD gives very similar performance, and adding more data boosts performance.This experiment uses CLIP4Clip on LSMDC16 after formatting MAD as trimmed clips around annotated timestamps.
  • Takeaway: Overall, MAD improves both grounding and retrieval, supporting dataset scale as a way to compensate for noise from automatic annotation.The paper presents this as the takeaway across the two related tasks.

6. Conclusion

MAD is a large-scale video grounding benchmark built from movie audio descriptions, addressing limitations of prior datasets while introducing long-form grounding challenges.

  • MAD combines high-quality movie audio descriptions with a large-scale video grounding benchmark.Its automatic annotation pipeline supports reproducible research despite unavailable video files.
  • The benchmark’s videos are unavailable because of copyright constraints, but necessary experimental features are provided for reproducibility.

A.1. MAD Detailed Statistics

This section reports additional MAD statistics covering split similarities and differences, sentence repetition, vocabulary, and visual and language diversity.

  • MAD’s additional statistics compare automatically curated training data with manually curated validation and test data.
  • The analysis assesses repetitive sentences that could create ambiguity for video-language grounding.
  • The section concludes by highlighting MAD’s large visual and language diversity.

A.1.1 Data splits comparison

MAD’s training and validation/test splits have broadly similar temporal and duration distributions, while training contains substantially more data and vocabulary.

  • The average temporal span differs by only 0.1 seconds, while training videos are 6.2% shorter on average than validation/test videos.
  • Training sentences average 2.9 additional words, partly because the annotation pipeline can join consecutive sentences in short temporal spans.
  • Training vocabulary reaches 57.6K tokens, nearly three times the 21.9K-token validation/test vocabulary.
  • The training split contains 2.7× more queries and approximately two-thirds of the video content than the validation/test split.
  • Sentence-length distributions are measured in tokens, providing a separate view of query-length variation across the dataset.
  • Training and validation/test moments follow similar relative start, end, and duration distributions, with slightly more training moments at video boundaries.

A.1.2 Sentences uniqueness

This section examines sentence repetition as a possible source of grounding ambiguity and characterizes MAD’s broad cinematic diversity.

  • Repeated sentences within a movie can create ambiguity, so the analysis measures pairwise sentence similarity using METEOR.
  • MAD spans 22 genres and 90 years of cinema history, demonstrating broad dataset diversity.
  • The sentence-similarity measure is presented as a useful approximation that requires further research.

A.1.3 Additional language statistics

MAD contains about 384K query sentences with variable lengths following a long-tailed distribution.

  • About 384K query sentences comprise MAD, averaging 12.7 tokens with a standard deviation of 8.1.

A.1.4 Diversity

MAD spans diverse movie content and query language, while adapting VLG-Net to ground sentences in long-form videos through windowed processing, temporal jittering, and negative sampling.

  • Diversity: MAD covers movies from the 1940s through the last decade and includes a large variety of genres.These dimensions relate to differences in picture quality, filming, editing, attire, actions, behavior, storytelling, scene setup, and pacing.
  • Input: VLG-Net is adapted for long-form MAD videos by processing independent windows of 128 consecutive frame features instead of entire videos.
  • Input: Temporal jittering changes the ground-truth-enclosing window each epoch, regularizing training against intra-window biases and encouraging temporal extent prediction.
  • Input: At inference, a 128-frame window slides across the movie with a stride of 64 frames, and window proposals are ranked by confidence for video-level recall.
  • Negative samples: Because local windows may not contain the ground truth, training adds negative or unpaired video-language pairs so incorrect windows receive low confidence.
  • Negative samples: Randomly selecting same-movie negative samples 70% of the time yields the best performance, while cross-movie negatives are excluded.
  • Modules: Using CLIP visual and language features creates a tokenizer mismatch that prevents retrieving the syntactic dependencies required by VLG-Net’s SyntacGCN module.
Loading 2112.00431v2…