PaperlayerView exact revision on arXiv

Source-linked AI summary

TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval

Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal

arXiv:2001.09099v2cs.CVcs.CLcs.IR

TL;DR

TVR addresses the need for video moment retrieval that uses both visual and subtitle context. The paper introduces a large, tightly annotated multimodal dataset and XML, a late-fusion model with ConvSE, with reported gains over baselines and improved efficiency.

  • Problem

    TVR addresses multimodal moment retrieval by requiring systems to use videos and associated subtitles rather than visual context alone.

  • Method

    The paper constructs TVR with freely localized query moments and modality labels, and proposes XML with late fusion and ConvSE for VCMR.

  • Results

    XML surpasses all presented baselines by a large margin and runs with better efficiency.

  • Takeaways & Limitations

    TVR and TVC provide publicly available resources for multimodal moment retrieval and captioning, containing 109K queries and 262K captions.

Abstract

from arXiv · show

We introduce TV show Retrieval (TVR), a new multimodal retrieval dataset. TVR requires systems to understand both videos and their associated subtitle (dialogue) texts, making it more realistic. The dataset contains 109K queries collected on 21.8K videos from 6 TV shows of diverse genres, where each query is associated with a tight temporal window. The queries are also labeled with query types that indicate whether each of them is more related to video or subtitle or both, allowing for in-depth analysis of the dataset and the methods that built on top of it. Strict qualification and post-annotation verification tests are applied to ensure the quality of the collected data. Further, we present several baselines and a novel Cross-modal Moment Localization (XML ) network for multimodal moment retrieval tasks. The proposed XML model uses a late fusion design with a novel Convolutional Start-End detector (ConvSE), surpassing baselines by a large margin and with better efficiency, providing a strong starting point for future work. We have also collected additional descriptions for each annotated moment in TVR to form a new multimodal captioning dataset with 262K captions, named TV show Caption (TVC). Both datasets are publicly available. TVR: https://tvr.cs.unc.edu, TVC: https://tvr.cs.unc.edu/tvc.html.

1 Introduction

TVR frames moment retrieval as a multimodal task requiring systems to use both video and subtitle context. It contributes a large dataset and XML, an efficient late-fusion model with ConvSE localization.

  • Motivation: Existing moment-retrieval datasets and methods primarily use visual context, limiting multimodal video-subtitle retrieval.VCMR extends retrieval from a single video to a large corpus, while proposal-based methods can be temporally imprecise and computationally expensive.
  • TVR Dataset: TVR collects queries grounded in videos and subtitles, with freely annotated tight timestamps and query-type labels for video, subtitle, or both.The dataset uses TV shows to capture activities and dialogue, and applies qualification and post-annotation verification tests.
  • Contributions: 109K high-quality queries on 21.8K videos from six TV shows establish TVR as a large-scale multimodal moment-retrieval dataset.The dataset emphasizes linguistic diversity and supports deeper analysis through modality-related query types.
  • XML: XML uses independently encoded videos, subtitles, and queries, then applies late fusion to generate temporal query-clip similarity scores.Pre-encoding videos reduces test-time computation to new-query encoding and optimized matrix multiplication.
  • XML: XML’s ConvSE module detects start-end edges in one-dimensional similarity signals with two convolution filters.The authors report that XML surpasses presented baselines by a large margin while running more efficiently.
  • Additional Dataset: TVC extends TVR with 262K additional captions for annotated moments, forming a multimodal video-captioning dataset.Both TVR and TVC are publicly available.

2 Related Work

Prior work studies language-based moment retrieval from single videos or video corpora, but existing datasets and methods generally rely on visual context and coarse proposals.

  • Retrieval Tasks: SVMR retrieves a moment from one video, whereas VCMR retrieves relevant moments from a large video corpus.VCMR generalizes the setting by requiring corpus-level search rather than assuming the ground-truth video is known.
  • Datasets: Existing datasets such as DiDeMo, ActivityNet Captions, CharadesSTA, and TACoS localize queries using video alone.TVR differs by requiring subtitle information in addition to visual context.
  • Datasets: Fixed video chunks and paragraph-to-query conversions have been used for annotation, but fixed segments may not align with natural moments.TVR instead freely selects temporal windows to improve localization accuracy.
  • Methods: Proposal-based methods rank handcrafted or sliding-window candidates, which can be temporally imprecise and difficult to scale to long videos.Regression offsets alleviate imprecision but remain constrained by the initial proposals.

3 Dataset

TVR is built from paired TV-show videos and subtitles with individually annotated, tightly localized query moments. Its analysis emphasizes data quality, linguistic diversity, multimodal complexity, and interactions among people and actions.

  • Dataset: TVR contains 21,793 videos from six TV shows across sitcom, medical, and crime genres, paired with subtitles and averaging 76.2 seconds per video.The videos are sourced from TVQA.
  • Data Collection: Workers wrote unique-match queries from video and/or subtitles, marked start and end timestamps, and assigned modality-based query types.The collection used separate visual and textual rounds to balance queries requiring visual, subtitle, or combined information.
  • Data Quality: A 12-question qualification test, automatic checks, manual review, and post-annotation verification were used to ensure data quality.Workers needed at least 9 correct qualification answers; automatic checks required at least 8 words and rejected subtitle-copying.
  • Data Quality: 92% of 3,600 verified query-moment pairs received ratings of at least neutral.Among strongly disagree ratings, 80% were still acceptable because of issues such as slightly mismatched timestamps of ≤1 sec.
  • Data Analysis and Comparison: 109K query-moment pairs have average moment length of 9.1 seconds, shorter normalized moments, longer queries, and greater linguistic diversity than existing datasets.Almost every TVR query is unique, making textual understanding more challenging.
  • Data Analysis and Comparison: 66% of TVR queries involve at least two people and 67% involve at least two actions, while the dataset also supports 262K additional captions in TVC.These properties make TVR a testbed for multimodal interactions between people and provide a captioning extension.

4 Cross-modal Moment Localization (XML)

XML addresses video corpus moment retrieval with a hierarchical architecture that separately encodes modalities, fuses query-clip similarities, and predicts precise temporal moments efficiently.

  • XML architecture: XML performs video retrieval in shallower layers and fine-grained moment retrieval in deeper layers for video corpus moment retrieval.Its late-fusion design combines video, subtitle, and query representations while supporting hierarchical retrieval.
  • Input representations: Video representations combine appearance and motion features, while subtitles and queries are represented with contextualized RoBERTa embeddings projected into a shared hidden space.Appearance uses ResNet-152 features and motion uses I3D features; subtitle features are temporally pooled to clips.
  • Query encoding: Modular query encoding uses trainable modality-specific weights to aggregate query-word information separately for video and subtitle representations.The encoded query is transformed into modality-specific vectors through attention over query words.
  • Cross-modal fusion: Late fusion computes one-dimensional query-clip similarity scores by combining video and subtitle similarities, after independently encoding videos or subtitles and queries.Independent encoding requires only M+N neural-network operations and allows video features to be pre-encoded.
  • Moment prediction: ConvSE uses two one-dimensional convolution filters to detect start and end edges in similarity signals instead of ranking handcrafted sliding-window proposals.The resulting start and end scores are softmax-normalized into probabilities, and predictions can be generated in linear time using dynamic programming.
  • Training and inference: At inference, XML first retrieves the top 100 videos using video-retrieval scores, then ranks moments within those videos using the aggregated VCMR score.The aggregation balances video-level and moment-level scores with α=20.

5 Experiments

Experiments evaluate TVR baselines and XML for VCMR, showing that late fusion and ConvSE provide stronger, faster moment localization while exploiting both video and subtitles.

  • Data, Metrics and Implementation Details: 5,445 queries across 1,089 videos form the TVR test-public VCMR evaluation set, with runtime averaged over 3 RTX 2080Ti runs.Feature extraction and preprocessing are excluded because they are expected to be similar across methods.
  • Baselines Comparison: 9.85× performance gain and 51.3× speedup over MEE+ExCL are achieved by XML at R@1 IoU=0.7.The comparison is 3.25 vs. 0.33 for performance and 25.5s vs. 1307.2s for runtime.
  • Baselines Comparison: 287× speedup is obtained on a 1M-video corpus with pre-encoded video representations.The larger-scale result further highlights the efficiency of XML’s late fusion design over ExCL’s early fusion design.
  • Baselines Comparison: The full video+subtitle XML model outperforms single-modality variants, while modality-specific models perform poorly on queries associated with the other modality.This demonstrates that both modalities contribute to multimodal VCMR performance.
  • ConvSE Analysis: 3, 5, or 7-sized ConvSE kernels perform reasonably, but performance drops significantly at k=1.The learned filters respond strongly to upward Start and downward End edges, indicating that neighboring information supports boundary detection.
  • Qualitative Analysis: XML qualitative examples include correct top-ranked moments from the same video and failures involving moments retrieved from another video.The examples also visualize query-word attention and subtitle associations for the predictions.

6 Conclusion

The paper presents TVR as a large-scale multimodal moment-retrieval dataset and XML as an efficient VCMR model. It concludes that TVR is high quality and challenging, while XML is suitable for VCMR.

  • 6 Conclusion: TVR is presented as a large-scale dataset for multimodal moment retrieval, and XML as an efficient model for VCMR.The conclusion also states that detailed analyses show TVR is high quality and more challenging than previous datasets.

A Additional TVR Data Details

TVR data collection uses qualification and post-annotation verification procedures, with reported ratings indicating high-quality query-moment pairs and additional TV show statistics.

  • Data Collection: TVR’s data collection procedure is summarized in Fig. 9 and includes qualification, annotation, and post-annotation verification stages.The paper directs readers to the data-collection section for details of each step.
  • Data Collection: 1,055 workers took a 12-question qualification test, and 67% passed by answering at least 9 questions correctly.The qualification test was designed to ensure workers understood the annotation requirements.
  • Post-Annotation Verification: 92% of 3,600 verified query-moment pairs received a rating of at least neutral.Workers rated relevance and unique-match quality on a five-option Likert scale.
  • Post-Annotation Verification: 80% of strongly-disagree and 90% of disagree pairs were still judged acceptable, including cases with timestamps mismatched by at most 1 second.The authors use this verification to support the data’s high quality.

A.2 Data Analysis

TVR spans 21,793 videos from six TV shows and contains short, visually and linguistically diverse moments whose queries frequently require video context.

  • Statistics by TV Show: 21,793 videos from six long-running TV shows constitute TVR’s source collection.The shows span The Big Bang Theory, Friends, How I Met Your Mother, Grey’s Anatomy, House, and Castle.
  • Moments and Queries: 9.1 seconds is the average TVR moment length, compared with 76.2 seconds for the average video.Most annotated moments are relatively short, and moment centers occur more often near video beginnings.
  • Moments and Queries: Around 91% of queries need video context, according to the query-type analysis.The analysis distinguishes query types by their reasoning requirements.
  • Frequent Words in Queries: TVR queries cover common objects, scenes, and actions alongside genre-specific terms such as “patient” and “hospital.”The word-cloud analysis lemmatizes words and removes stop words.
  • Video Comparison: TVR videos typically have more visual diversity than videos in existing datasets, including more camera viewpoints, activities, and people.The comparison covers videos from six TV shows and three genres.

B.1 More VCMR Experiments

Additional VCMR experiments examine dataset bias, proposal strategies, architecture and feature choices, scalability, and retrieval efficiency. XML consistently benefits from direct temporal modeling, multimodal features, and corpus-level decomposition.

  • Dataset and feature analysis: Temporal Endpoint Features improve most models, indicating moment-distribution and language-timestamp biases in TVR.Moments occur more often near video beginnings, while words such as “first” can indicate temporal location.
  • Proposal generation: 3.82 vs. 0.97, R@1 IoU=0.7, XML (sw) outperforms MEE+CAL using the same sliding-window proposals.The authors attribute the gap partly to ranking difficulties over 1.5M proposals and describe XML’s video-level and moment-level decomposition.
  • Model architecture: Transformer encoders outperform bidirectional LSTM and CNN alternatives, which the authors associate with better long-term dependency capture.The ablation also compares modularized queries with a single max-pooled query.
  • Feature ablation: Combining ResNet appearance and I3D action features outperforms using either feature alone, supporting recognition of both objects and actions.The feature ablation is conducted for VCMR with both videos and subtitles.
  • Retrieval efficiency: XML is 18× faster in retrieval, 4.5× faster in feature encoding, and uses 77% less disk space than MEE+MCN.On the TVR test-public set, XML also records 3.25 vs. 0.42, IoU=0.7, R@1.
  • Retrieved-video sensitivity: XML’s R@1, R@5, and R@10 performance is not sensitive to retrieving 10–200 videos in the tested range.For R@100 at IoU=0.5, retrieving more videos helps.

C TVR DiDeMo Experiments

Experiments extend evaluation to DiDeMo and introduce TVC, a multimodal captioning dataset built from TVR moments. The captioning setup uses both video and subtitle context through a multimodal Transformer.

  • TVR DiDeMo Experiments: XML outperforms all baselines on DiDeMo, despite using only ResNet features and no subtitle stream.The result indicates XML can generalize to datasets where only video context is available.
  • TVC dataset: TVC adds extra descriptions to TVR moments, with annotators encouraged to cover video-only and video-plus-subtitle content types.Training moments receive one extra description, while validation and test moments receive additional descriptions as specified in the collection procedure.
  • TVC dataset: 262K descriptions paired with 108K moments make TVC distinct from visual-only captioning datasets because captions may describe dialogue and subtitles.TVC also includes description-type annotations for training and analysis.
  • TVC model: MMT encodes video and subtitle inputs together with a Transformer encoder-decoder to generate captions.Video uses concatenated I3D and ResNet-152 features, while subtitles use trainable 300D word embeddings projected into a shared space.

D.3 Experiments

TVC experiments compare input modalities and feature configurations using standard captioning metrics, while the dataset release preserves the TVR split structure.

  • TVC Experiments: Across all reported metrics, MMT using both videos and subtitles performs better than models using only one modality.The comparison uses BLEU@4, METEOR, Rouge-L, and CIDEr-D on TVC.
  • TVC Experiments: Using both ResNet-152 appearance features and I3D motion features outperforms using either visual feature alone.All models in this feature ablation use both videos and subtitles.

E Data Release and Public Leaderboards

TVR and TVC are publicly released with split protocols, leaderboards, and qualitative examples illustrating multimodal retrieval and captioning behavior. The splits prevent associated videos and queries from crossing partitions.

  • Data Release and Public Leaderboards: TVR and TVC are publicly available through their websites, with public leaderboards provided for system comparison.The paper directs readers to the dataset websites for release and usage details.
  • Data Release and Public Leaderboards: TVR uses 80% train, 10% validation, 5% test-public, and 5% test-private splits, keeping each video and its queries in one split.Validation is intended for parameter tuning, while test-private is reserved for future challenges.
  • Data Release and Public Leaderboards: TVC follows TVR’s split structure but assigns 2 descriptions to training moments and 4 descriptions to moments in other splits.The same split-usage rules apply to TVC.
  • Qualitative examples: TVR examples compare retrieval predictions with ground-truth temporal regions, while TVC examples pair captions with description-type tags and subtitle context.XML figures show top-3 VCMR and SVMR predictions; TVC figures illustrate caption descriptions and sampled moment frames.
Loading 2001.09099v2…