PaperlayerView exact revision on arXiv

Source-linked AI summary

MovieNet: A Holistic Dataset for Movie Understanding

Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, Dahua Lin

arXiv:2007.10937v1cs.CV

TL;DR

Understanding story-based movies remains difficult because they combine long temporal structure, artistic style, and multimodal content. MovieNet addresses this gap with a holistic dataset, annotations, and benchmarks spanning multiple aspects of movie understanding. Its experiments reveal limitations of current approaches and support the value of holistic annotations for movie analytics.

  • Problem

    Story-based movie understanding remains challenging because movies combine long-form stories, artistic styles, complex situations, and multimodal information.

  • Method

    MovieNet builds a holistic dataset from 1,100 movies with multimodal data, diverse manual annotations, and benchmarks covering multiple movie-understanding aspects.

  • Results

    Experiments across challenging benchmarks reveal a gap between current approaches and comprehensive movie understanding and show benefits from holistic annotations.

  • Takeaways & Limitations

    MovieNet provides a broad resource for research on cinematic style, middle-level entities, stories, and related video analytics.

  • Takeaways & Limitations

    The current dataset covers 1,100 movies, with planned extensions to more movies, annotations, approaches, and topics.

Abstract

from arXiv · show

Recent years have seen remarkable advances in visual understanding. However, how to understand a story-based long video with artistic styles, e.g. movie, remains challenging. In this paper, we introduce MovieNet -- a holistic dataset for movie understanding. MovieNet contains 1,100 movies with a large amount of multi-modal data, e.g. trailers, photos, plot descriptions, etc. Besides, different aspects of manual annotations are provided in MovieNet, including 1.1M characters with bounding boxes and identities, 42K scene boundaries, 2.5K aligned description sentences, 65K tags of place and action, and 92K tags of cinematic style. To the best of our knowledge, MovieNet is the largest dataset with richest annotations for comprehensive movie understanding. Based on MovieNet, we set up several benchmarks for movie understanding from different angles. Extensive experiments are executed on these benchmarks to show the immeasurable value of MovieNet and the gap of current approaches towards comprehensive movie understanding. We believe that such a holistic dataset would promote the researches on story-based long video understanding and beyond. MovieNet will be published in compliance with regulations at https://movienet.github.io.

1 Introduction

MovieNet addresses the complexity of understanding story-based movies by combining multimodal data, holistic annotations, and benchmarks across movie-understanding tasks. Its experiments expose gaps in current approaches and indicate benefits from integrating diverse annotations.

  • Motivation: Movie understanding is challenging because movies combine complex stories, artistic styles, varied situations, and rich multimodal content.The paper frames movie understanding as closely related to understanding the real world and as an arena for high-level machine intelligence.
  • Contribution: MovieNet combines data, annotation, and benchmark components into a holistic dataset for comprehensive movie understanding.The three aspects are presented as connected parts of the dataset.
  • Contribution: The dataset includes multiple modalities such as movies, trailers, photos, subtitles, scripts, and metadata, totaling 3K hours of video, 3.9M photos, 10M text sentences, and 7M metadata items.These resources cover visual, textual, and structured movie information.
  • Contribution: MovieNet provides annotations for characters, scenes, actions, places, aligned descriptions, and cinematic styles to support different levels of movie understanding.Cinematic-style annotations include view scale and camera motion for more than 46K shots.
  • Benchmarks: Benchmarks span genre analysis, cinematic style prediction, character analysis, scene understanding, and movie segment retrieval.The benchmarks are designed to cover different aspects of movie understanding.
  • Results: Experimental analysis shows a gap between current approaches and comprehensive movie understanding, while holistic annotations benefit video analytics.The paper positions these findings as evidence for the value of combining multiple annotation types.

2 Related Datasets

Prior movie-understanding datasets typically target one element or task and are limited in scale and annotation richness. MovieNet is positioned as a broader alternative for comprehensive understanding.

  • Existing datasets: Existing movie datasets commonly focus on a single element, including genre, character, action, scene, or description.This task-specific design limits the range of movie-understanding research each dataset can support.
  • Existing datasets: Many prior datasets are small, using only several TV episodes, clips from twelve movies, or three movies for particular tasks.The paper contrasts these scales with the demands of data-hungry learning.
  • Research gap: Existing datasets do not support the progression from middle-level elements to high-level story understanding within one comprehensive resource.The paper identifies this as a difficulty for comprehensive movie understanding.
  • Comparison: Tables 1 and 2 compare MovieNet with related datasets in terms of data and annotation.These comparisons are used to characterize MovieNet's relative scale and annotation coverage.

3 Visit MovieNet: Data and Annotation

MovieNet assembles a large movie collection with multimodal resources and manually curated annotations for semantic elements and cinematic style. It uses these resources to support genre-analysis and other movie-understanding benchmarks.

  • Data: MovieNet includes 1,100 selected color movies longer than one hour, spanning diverse genres, years, and countries.Metadata includes titles, release dates, countries, genres, ratings, runtimes, directors, casts, and storylines.
  • Data: The dataset gathers movies, metadata, subtitles, trailers, photos, scripts, and other multimodal resources, with subtitles manually checked for alignment.Metadata is collected from IMDb and TMDb, while trailers are downloaded using linked sources.
  • Annotation: MovieNet manually annotates characters, scenes, events, and cinematic styles to support varied movie-understanding research topics.Cinematic style is treated as relevant because it influences how a movie tells its story.
  • Annotation: Character annotation uses key-frame bounding boxes, detector-assisted expansion, manual cleaning, and identity annotation.The process begins with 758K selected key frames and retains the top 10 credited cast members.
  • Annotation: Manual scene-boundary annotation yields 42K scenes and captures the movie's hierarchical shot–scene temporal structure.Shots are minimal visual units, while scenes are semantically related sequences of shots.
  • Annotation: Action and place tags are assigned to scene-derived clips or action-containing sub-clips, while natural-language descriptions address more complex events.The annotation process uses multiple tags for places and actions.
  • Benchmarks: Table 3 compares MovieNet with other genre-analysis benchmarks and reports baseline genre-classification results.The table separates benchmark comparison from baseline performance.
  • Benchmarks: Genre analysis includes a framework for analyzing long movies and samples of genre-guided trailer generation for Titanic.The framework and generation examples are presented as separate parts of Figure 3.

4 Play with MovieNet: Benchmark and Analysis

MovieNet supports benchmarks spanning genre, cinematic style, character, scene, and story understanding, exposing challenges and gains from large-scale, multimodal, holistic annotations.

  • Benchmark scope: MovieNet benchmarks five aspects of movie understanding: genre, cinematic style, character, scene, and story.The benchmarks use currently popular techniques and analyze experimental results across these research topics.
  • Genre analysis: Genre analysis becomes harder at MovieNet scale because labels are extremely imbalanced and genre depends on high-level semantics beyond visual appearance.Drama has 40 times as many instances as Sport, while genre also depends on action, clothing, facial expression, and background music.
  • Cinematic style analysis: MovieNet is the first dataset covering both view scale and camera motion for cinematic style prediction, and subject maps improve performance.The benchmark applies TSN and I3D, with saliency-based subject detection helping prediction.
  • Character recognition: MovieNet provides more than 1.1M character instances from 3,087 identities, and a MovieNet-trained Cascade R-CNN reaches 95.17% mAP for character detection.The dataset is designed to address domain gaps between movie characters and general object or pedestrian datasets.
  • Scene analysis: With more than 43K scene boundaries and 65K action/place tags, MovieNet supports both scene segmentation and tagging; multimodality and multiple semantic elements improve segmentation.The MS-LSTM combines audio, character, action, and scene information, while large scale and diversity improve performance.
  • Story understanding: Movie segment retrieval improves when it uses middle-level entities, multimodal features, and graph formulations of temporal and spatial story structure.Stories are represented through character, action, and place graphs, with graph matching further boosting retrieval performance.

5 Discussion and Future Work

MovieNet is presented as a holistic dataset for comprehensive movie understanding, with benchmarks spanning filming art, middle-level entities, and high-level stories. Future work will extend its coverage and explore additional approaches and topics.

  • MovieNet provides annotations covering different aspects of movie understanding.The paper frames these annotations as supporting comprehensive movie understanding.
  • Its benchmarks address filming art, middle-level entities, and high-level story semantics.The listed areas include discovering filming art, recognizing entities, and understanding stories.
  • Integrating filming art and middle-level entities according to movies’ internal structure helps story understanding in segment retrieval.The paper presents this result as evidence for the effectiveness of holistic annotations.
  • Future work will expand the dataset beyond 1,100 movies, add annotations, improve approaches, and address topics such as movie deoldify and trailer generation.

Supplementary Material

The supplementary material documents MovieNet’s data, annotations, benchmarks, and toolbox. It covers collection procedures, annotation workflows, experimental tasks, and software for managing the dataset.

  • Data: The supplementary material details MovieNet’s data content, collection, and cleaning procedures.It organizes these materials under metadata, movies, subtitles, trailers, scripts, synopses, and photos.
  • Annotation: Annotation sections describe workflows for character boxes and identities, cinematic styles, scene boundaries, action and place tags, and synopsis alignment.The workflows include semi-automatic image annotation, optimized scene-boundary labeling, joint action/place labeling, and coarse-to-fine synopsis alignment.
  • Annotation: Trailer–movie alignment automatically links trailer shots to the original movies.This annotation supports tasks such as trailer generation.
  • Benchmarks: The benchmarks cover genre classification, cinematic-style analysis, character detection and identification, scene segmentation, action recognition, place recognition, and story understanding.Story understanding uses MovieNet segment retrieval to explore analytics combining different dataset aspects.
  • Benchmarks: Scene segmentation is formulated as boundary detection, while action and place recognition are multi-label classification tasks.
  • Toolbox: A codebase provides data-management and benchmark-processing tools, and the toolbox is released with the benchmark code.

A Data in MovieNet

MovieNet combines multimodal data with high-quality annotations across multiple aspects of movie understanding, with comparisons to related datasets provided in Table A1.

  • MovieNet contains multimodal data and high-quality annotations covering different aspects of movie understanding.The dataset’s overall data comparison with related datasets appears in Table A1.

A.1 Meta Data

MovieNet’s metadata spans a large movie collection and diverse attributes, while its movie resources include multiple representations, aligned subtitles, and script-alignment procedures. The supplementary material also reports distributions and qualitative alignment results.

  • Metadata: MovieNet provides metadata for 375K movies, including the 1,100 movies with video sources.The larger metadata collection is intended to support tasks beyond the video dataset itself.
  • Movie collection: MovieNet contains 28 unique genres, and its selected movies cover a wide range of years, countries, and genres.The supplementary statistics show distributions for these attributes and describe the movie collection as diverse across them.
  • Metadata: Metadata includes identifiers, titles, genres, release information, countries, versions, ratings, directors, writers, cast, and plot descriptions.The metadata uses IMDb, TMDb, and, for some Asian films, Douban Movie information.
  • Movie collection: The 1,100 movies are selected to be colored, longer than one hour, and diverse in genres, years, and countries.The dataset also provides runtime and description information for multiple movie versions.
  • Feature representation: MovieNet offers shot-based visual and audio features, plus frame-based features for tasks requiring motion information.Movies are divided into shots, with key-frame visual features and within-shot audio features extracted for representation.
  • Alignment: Subtitles are aligned to movie sources, and dynamic programming aligns dialog snippets with subtitle sentences before transferring timestamps to corresponding script snippets.The qualitative Titanic example shows connections between storyline snippets, timestamps, and images without human assistance.

B.1 Character Bounding Box and Identity

MovieNet builds large-scale character annotations through a staged workflow that combines manual labeling, detector assistance, identity matching, and temporal cleaning. The broader annotation framework also covers cinematic style, scenes, actions, and places.

  • Character annotation: 758K key frames were manually annotated to produce 1.3M character bounding boxes.The key frames were sampled at three frames per shot.
  • Character annotation: A Cascade R-CNN detector reached 95% mAP and assisted identity annotation, resulting in 1.1M character instances.Identity labels were assigned using one frame per shot, with manual cleaning of false-positive boxes.
  • Character annotation: The identity workflow used cast portraits plus face and body features to generate candidate character matches for annotators.Annotators labeled positive or negative samples and later cleaned results using temporal context.
  • Character annotation: Character instances follow a long-tail distribution, with substantially more instances for famous actors such as Leonardo Dicaprio.The statistics and examples indicate that the annotations contain diverse characters but uneven identity frequencies.
  • Scene annotation: Scene boundaries were annotated with a shot-based workflow because each scene is treated as a continuous subsequence of shots.Annotators compared neighboring shots with surrounding frames, and inconsistent labels received additional judgments.
  • Action and place annotation: Action and place labels were assigned to scene segments, with multiple place tags per segment and multiple action tags per detected human-action sub-clip.MovieNet reports 41,259 action clips, 45,033 action tags, 13,700 place clips, and 19,642 place tags.

C Experiments

MovieNet evaluates movie-understanding tasks using disjoint movie-based training, validation, and test splits. The same split determines the corresponding annotation partitions.

  • Dataset split: 1,100 movies are split into training, validation, and test sets at a 3:1:1 ratio.The split is applied by movie rather than independently by annotation.
  • Dataset split: Annotations inherit the movie split for each task.This prevents movies from overlapping across the training, validation, and test sets.
  • Dataset split: The benchmark design keeps train, validation, and test movies separate across tasks unless otherwise specified.The passage states that the 1,100-movie split is the default experimental setting.

C.1 Genre Classification

The genre-classification benchmarks use filtered MovieNet photos and trailers to evaluate multi-label genre prediction. Performance varies by genre, with lower AP for genres requiring higher-level semantic information.

  • Benchmark setting: 21 visual genres are used after removing rare and non-visual categories from MovieNet’s 28 genres.The image benchmark uses posters, still frames, products, and production art.
  • Image benchmark: 1.6M photos are used for image-based genre classification, split into 1.1M training, 160K validation, and 321K test images.The selected photos exclude publicity images and other unrelated photo types.
  • Video benchmark: 32K trailers containing at least one target genre support the video-based genre-classification benchmark.The trailer split contains 22.5K training, 3.2K validation, and 6.4K test videos.
  • Evaluation: Genre classification is evaluated as a multi-label problem using mAP, recall@0.5, and precision@0.5.The passage identifies 0.5 as the threshold used for recall and precision.
  • Results: AP is much lower for Biography and Documentary because these genres are determined by higher semantic elements.The passage also notes that animation has a significant characteristic for classification.

C.2 Cinematic Style Analysis

MovieNet evaluates cinematic-style prediction on trailer shots using separate scale and movement categories. The benchmark compares temporal video models and visual-input designs for these style labels.

  • Benchmark setting: 46K shots from 8K trailers form the cinematic-style benchmark, split into 26K training, 7K validation, and 13K testing shots.The benchmark covers cinematic-style prediction at the shot level.
  • Tasks: The benchmark evaluates shot scale and shot movement as distinct cinematic-style prediction tasks.The supplied passages describe baseline comparisons for both scale and movement classification.
  • Baselines: TSN and I3D baselines are implemented with multiple ResNet backbones for cinematic-style prediction.The listed backbones include ResNet-18, ResNet-34, and ResNet-50.
  • Model design: The two-branch models concatenate features before a fully connected layer, using whole and background images for movement and whole and subject images for scale.The classification models are trained with cross-entropy loss for 60 epochs using mini-batch SGD.

C.4 Character Identification

MovieNet frames character identification as large-scale movie retrieval: finding every instance of a character from one portrait. Its challenging domain and scale expose limitations of existing person re-identification datasets while enabling multimodal character analysis.

  • Dataset and task: More than 1.1M character instances spanning 3,000 identities support MovieNet’s character identification benchmark.The benchmark split contains 639.9K, 336.6K, and 364.2K instances across training, validation, and test identities, respectively.
  • Dataset and task: The task searches a movie for all instances of a character using a single portrait query.Portraits downloaded from credited-cast homepages serve as query portraits, and performance is evaluated with mean average precision across queries.
  • Baseline analysis: MovieNet’s character identification setting is larger and more challenging than conventional person re-identification datasets.The authors use cross-dataset ResNet-50 training to examine the domain gap between person and movie character identification.
  • Baseline analysis: Current person re-identification datasets cannot adequately support movie character analysis because of the large domain gap.Market and CUHK03 are compared with MovieNet, alongside LP and PPCC methods.
  • Implementation: The implementation combines face and body features, with a two-step retrieval strategy that expands the query set before body-feature set-to-set retrieval.LP is described as noise-sensitive, while PPCC improves it with a competitive consensus scheme.

C.6 Action Recognition

The supplied passages describe MovieNet’s action-recognition benchmark and related multimodal movie-understanding experiments. Baselines use multilabel action prediction, while retrieval methods progressively combine appearance, subtitles, actions, characters, and cinematic style.

  • Benchmark setup: Action recognition predicts multiple action tags for each video clip using a movie-disjoint train, validation, and test split.The split contains 23,747 training, 7,543 validation, and 9,969 test clips.
  • Benchmark setup: Action recognition is trained with binary cross-entropy and evaluated using mean average precision.TSN uses a ResNet-50 backbone, samples three video segments, and averages their consensus scores.
  • Action-recognition results: SlowFast achieves the best action-recognition performance, while TSN performs worst and SlowFast substantially outperforms the other baselines.Per-class average precision for SlowFast is further analyzed in Fig. C1.
  • Story understanding: The movie-segment retrieval model progressively adds appearance, subtitle, action, and character nodes to a VSE baseline.Video-segment and synopsis-paragraph features are transformed with two-layer multilayer perceptrons before comparison.
  • Story understanding: The retrieval objective compares matching and nonmatching segment–paragraph similarities using a margin-based ranking loss.S(Qj, Pi) denotes the similarity between the jth segment and ith paragraph, while θ denotes model parameters.
  • Story understanding: Cinematic style is incorporated by using style probabilities with node features to produce attention weights for output embeddings.The attention module is designed to emphasize whether a node is important in a particular shot.
Loading 2007.10937v1…