MovieNet: A Holistic Dataset for Movie Understanding

1 Introduction

• The authors motivate movie understanding as a high-complexity challenge for high-level machine intelligence because movies connect richly to the real world.Movies portray characters facing varied situations and behaviors, while their large historical collection offers rich content and multimodal information for data-hungry models.Source: §1 · pp. 1–2
• MovieNet contains 3K hour-long videos, 3.9M photos, 10M text sentences, and 7M items of metadata across multiple modalities.Its data include movies, trailers, photos, subtitles, scripts, and metadata such as genres, cast, directors, and ratings.Source: §1 · p. 2
• MovieNet annotates 1.1M characters, 40K scene boundaries, 65K action/place tags, 12K aligned description sentences, and 92K cinematic-style tags.The annotations cover characters, scenes, actions and places, natural-language descriptions, and cinematic styles such as view scale and camera motion.Source: §1 · p. 2
• MovieNet benchmarks genre analysis, cinematic-style prediction, character analysis, scene understanding, and movie-segment retrieval, with experiments analyzing performance and gaps in comprehensive understanding.The authors also examine how holistic annotations benefit throughout-video analytics.Source: §1 · pp. 2–3
• MovieNet is presented as the first holistic movie-understanding dataset combining large-scale multimodal data with high-quality annotations across different aspects.The authors hope it will support video editing, human-centric situation understanding, and story-based video analytics.Source: §1 · p. 3

2 Related Datasets

• Existing datasets typically target one movie element, have limited scale and annotations, and cannot support comprehensive understanding from middle-level elements to high-level story.Examples include character identification using several TV episodes, action recognition using clips from twelve movies, and scene segmentation using only three movies.Source: §2 · p. 4
• MovieQA has 15K questions for 408 movies, but lacks middle-level annotations and relies on wiki-plot questions that favor textual QA over story-based video understanding.Textual-plot approaches achieve much higher accuracy than approaches using video and subtitles.Source: §2 · p. 4
• LSMDC covers 200 movies with audio descriptions, whose accessibility-focused style differs from natural audience descriptions and whose supply is difficult to scale.In contrast, MovieNet provides multiple textual sources and annotations of middle-level entities.Source: §2 · p. 4
• AVA contains 430 15-minute movie clips labeled with 80 atomic actions, but its imbalanced labels and less story-informative actions limit story understanding.MovieNet instead annotates semantic-level actions for action recognition and story understanding.Source: §2 · p. 4
• MovieGraphs annotates characters, interactions, and attributes in social situations from 51 movies, whereas MovieNet offers richer data sources, supports more aspects of understanding, and is much larger.MovieNet includes photos, subtitles, scripts, and trailers, while MovieGraphs focuses on situation recognition.Source: §2 · p. 4

3 Visit MovieNet: Data and Annotation

• MovieNet brings together diverse movie data and annotations, with the paper highlighting its quality, scale, and richness.The overview illustrates the dataset with Titanic and points to comparisons with other movie-understanding datasets in Tables 1 and 2.Source: §3 · p. 5

3.1 Data in MovieNet

• The dataset includes 1,100 selected movies, 375K metadata items, and 805K genre tags across 28 genres.Movies were chosen to be colored, longer than one hour, and varied in genre, year, and country; metadata includes identifiers, cast, storyline, and other attributes.Source: §3.1 · p. 5
• MovieNet adds aligned subtitles, 60K trailers, around 2K scripts, 11K synopses, and 3.9M photos to its movie data.Trailer links were taken from IMDb and TMDb; scripts were aligned using dialogue and subtitles, and synopses were manually aligned to movies.Source: §3.1 · p. 6

3.2 Annotation in MovieNet

• MovieNet annotates 47K shots for view scale and camera movement, using five view-scale categories and four camera-movement classes.The annotation effort also covers character boxes and identities, scene boundaries, action and place tags, and event descriptions.Source: §3.2 · p. 6
• The character annotations yield 1.1M instances, while manual scene-boundary annotation identifies 42K scenes.The process uses 758K selected key frames, detector-assisted box cleanup, and manual identity labels; credited identities are limited to the top ten cast members, with others labeled separately.Source: §3.2 · p. 7
• MovieNet labels 42K scene-based segments with 19.6K place tags and 45K action tags, and aligns synopses to 4,208 segment-paragraph pairs.Annotators were encouraged to create diverse tags and exclude uninformative actions; three annotators used a coarse-to-fine procedure for synopsis alignment.Source: §3.2 · p. 7

4.1 Genre Analysis

• Genre classification performance drops on MovieNet’s larger, long-tailed dataset, where Drama labels outnumber Sport labels 40 to 1 and genre prediction requires high-level semantics.For long-video analysis, the paper uses trailer-trained shot features and a temporal model to produce genre response curves that can guide trailer generation.Source: §4.1 · p. 9

4.2 Cinematic Style Analysis

• MovieNet’s cinematic-style benchmark covers both view scale and camera motion, and adding saliency-derived subject maps improves prediction performance.The paper presents this as a large-scale dataset for a topic with few prior works and no large-scale datasets.Source: Table 4 · p. 9, §4.2 · p. 10

4.3 Character Recognition

• 95.17% mAP for MovieNet-trained Cascade R-CNN contrasts with 75.95% mAP for character identification, which remains challenging.The character benchmarks contain more than 1.1M instances from 3,087 identities; the analysis also finds domain gaps for models trained on general detection or ReID datasets and the value of aggregating visual cues.Source: Table 6 · p. 10, §4.3 · p. 11

4.4 Scene Analysis

• MovieNet supports both scene segmentation and scene tagging, and results indicate that multimodality and multiple semantic elements improve segmentation while action and place tagging remain challenging.The dataset contains more than 43K scene boundaries and 65K action/place tags and is described as the only one supporting both tasks.Source: §4.4 · pp. 11–12, Table 9 · p. 12, Table 10 · p. 12

4.5 Story Understanding

• Graph matching further boosts movie-segment retrieval by modeling temporal event order and spatial relations among story elements.The framework also combines middle-level entities and modalities, including subtitles, with cinematic-style-guided attention over elements such as action and character.Source: §4.5 · pp. 13–14, Table 11 · p. 14

Discussion

• MovieNet is a holistic dataset containing diverse annotations to support comprehensive movie understanding.Source: §5 · p. 14
• Movie segment retrieval results show that integrating filming art and middle-level entities according to movies’ internal structure helps story understanding.The results demonstrate the effectiveness of holistic annotations.Source: §5 · p. 14

Future Work

• Future work will extend MovieNet beyond its current 1,100 movies, explore more effective approaches, and investigate video-editing topics such as movie deoldify and trailer generation.Source: §5 · p. 14

Data

• The data supplement covers metadata, movie statistics, subtitle collection and alignment, trailer processing, script alignment, synopsis statistics, and movie-photo statistics and examples.It describes data content as well as collection and cleaning procedures.Source: Supplementary Material · p. 15

Annotation

• Annotation workflows cover character bounding boxes and identities, cinematic styles, scene boundaries, action and place tags, and synopsis-to-segment alignment.The supplement describes annotation interfaces and workflows, including semi-automatic image annotation and coarse-to-fine synopsis alignment.Source: Supplementary Material · p. 15
• The supplement also introduces an automatic approach for aligning trailer shots with original movies, supporting tasks such as trailer generation.It presents the trailer-movie alignment annotation in Sec. B.6.Source: Supplementary Material · p. 15

Benchmarks

• MovieNet benchmarks cover multi-label genre classification, cinematic-style scale and movement classification, character detection and identification, and scene-boundary detection.The supplement introduces task settings and implementation details for these benchmarks.Source: Supplementary Material · pp. 15–16
• Other benchmarks address multi-label action and place recognition and story understanding through MovieNet segment retrieval.For these tasks, the supplement introduces baseline models, experimental results, or experimental settings and results.Source: Supplementary Material · p. 16

Toolbox

• A codebase and toolbox provide data-management and processing tools alongside code for the benchmarks.The toolbox’s features are introduced in Sec. D.Source: Supplementary Material · p. 16

• MovieNet contains data from multiple modalities for movie understanding.Source: A Data in MovieNet · p. 16
• MovieNet provides high-quality annotations covering different aspects of movie understanding.Source: A Data in MovieNet · p. 16
• Tab. A1 compares MovieNet’s overall data with that of related datasets.Source: A Data in MovieNet · p. 16

A.1 Meta Data

• MovieNet includes metadata for 375K movies, including all 1,100 video-source movies, with identifiers and descriptive fields supporting access to broader movie information.Metadata fields include title, genre, release date, country, ratings, filmmakers, cast, and plot descriptions.Source: §A.1 · pp. 16, 18, 19

A.2 Movie

• MovieNet’s 1,100 films span diverse years, countries, and genres, and are represented with shot-based visual features, supplementary shot audio, and frame-based features for motion-dependent tasks.Shot-based features use three sampled key frames per shot; frame-based features are provided for tasks such as action recognition.Source: §A.2 · pp. 20–21

A.3 Subtitle

• Manual checks found few remaining problems after subtitle alignment, which filters out subtitles with shifts over 60 seconds and uses audio-to-text matching to estimate shifts.The workflow checks subtitle completeness and language, removes noise, estimates shifts, and then manually reviews automatically aligned subtitles.Source: §A.3 · pp. 21–22

A.4 Trailer

• MovieNet contains 60K trailers from 33K unique movies and distinguishes informational shots from content shots using detected text coverage and image contrast.A shot is classified as an info-shot when its text score exceeds α and its average contrast is below β.Source: §A.4 · pp. 22–23

A.5 Script

• Dynamic programming aligns script dialog with subtitles and transfers subtitle timestamps to storyline snippets, connecting storylines to the movie timeline without human assistance.The alignment formulation uses dialog-subtitle similarity and preserves the order of matched dialog snippets.Source: §A.5 · pp. 24–25

A.6 Synopsis

• MovieNet’s synopsis corpus is described as higher-quality than wiki plots, with richer content and longer descriptions.The section presents a statistics comparison and word-cloud visualization of the synopsis corpus.Source: §A.6 · p. 26, Table A2 · p. 26, Figure A11 · p. 26

A.7 Photo

• MovieNet contains 3.9M photos across seven types, with the appendix reporting the proportion of each type and showing examples.Source: §A.7 · p. 26, Figure A12 · p. 26, Figure A13 · p. 27

B Annotation in MovieNet

• The annotation section introduces a workflow and labeling interface designed to achieve high-quality annotations.Source: B Annotation in MovieNet · p. 27

B.1 Character Bounding Box and Identity

• 1.1M character instances and 1.3M bounding boxes underpin MovieNet’s character annotations, with a 95% mAP detector and semi-automatic identity labeling.Annotators labeled boxes on 758K sampled key frames, and temporal-context cleaning addressed bias and noise from the semi-automatic identity system. Instance counts follow a long-tail distribution, with famous actors receiving many more instances.Source: §B.1 · pp. 27, 29, 30

B.2 Cinematic Styles

• MovieNet cinematic-style annotations use five shot-scale categories and four shot-movement categories, with each task annotated three times.Annotators are trained by cinematic professionals and must pass a test; the categories are simplified to keep annotation affordable while covering most applications.Source: §B.2 · pp. 30–32

B.3 Scene Boundary

• Scene boundaries are selected from shot boundaries, and chunks are independently labeled by three annotators, with disagreements sent to two additional annotators.Movies are divided into overlapping chunks of about 500 to 700 shots to help annotators maintain focus; scene segmentation can remain difficult when visual cues are similar.Source: §B.3 · pp. 33–34

B.4 Action and Place Tags

• 41,259 action clips have 45,033 action tags, while 13,700 place clips have 19,642 place tags in MovieNet.Annotations cover 80 action classes and 90 place classes; generic actions such as standing and walking are excluded to focus on story-relevant actions. Each scene receives multiple place tags, while action tags are assigned to uninterrupted human-action sub-clips.Source: Table B1 · p. 37, §B.4 · p. 37

B.5 Synopsis Alignment

• 4,208 synopsis-paragraph–segment pairs were retained after three annotators labeled each paragraph and only high-consistency alignments were kept.The coarse-to-fine workflow splits each movie into 64 segments, has annotators choose consecutive clips for each paragraph, then refine segment boundaries.Source: §B.5 · pp. 40–41

B.6 Trailer Alignment.

• Trailer shots are aligned by finding the most similar movie subsequence using GIST and ResNet features, then low-scoring alignments are flagged and manually checked.GIST features address frames with modest changes in color, size, lighting, or boundaries, while ResNet features help with harder cropped or carefully edited shots.Source: §B.6 · pp. 42–43

C Experiments

• The 1,100 movies are split 3:1:1, and each task’s train, validation, and test sets contain no overlapping movies.Annotations are split according to the movie split.Source: C Experiments · p. 43

C.1 Genre Classification

• The benchmark uses 21 visual genres, 1.6M selected photos, and 32K trailers for image- and video-based classification.The photos come from poster, still frame, product, and production art types; the photo splits are 1.1M/160K/321K and the trailer splits are 22.5K/3.2K/6.4K for train/validation/test.Source: §C.1 · p. 43
• Genre classification is evaluated as a multi-label task using mAP, recall@0.5, and precision@0.5, with 0.5 as the positive-prediction threshold.Scores above 0.5 are positive and the others negative.Source: §C.1 · p. 43
• Models use BCE Loss, 224×224 inputs, and SGD; the video model trains on eight clips of three frames and averages clip scores at inference.Source: §C.1 · p. 43
• Animation achieves the highest accuracy, while Biography and Documentary have much lower AP because they depend on higher semantic elements.The per-genre results are reported for ResNet-50 in Table C1.Source: Table C1 · p. 44, §C.1 · p. 44

C.2 Cinematic Style Analysis

• The subject-based style method combines video-clip and saliency branches, using whole and background images for movement and whole and subject images for scale classification.R3Net produces each frame’s saliency map; concatenated branch features feed a fully connected layer for prediction.Source: §C.2 · pp. 44–45
• Deeper 2D models improve scale and movement classification, while deeper 3D models show slight movement-performance drops attributed to overfitting.The baselines include TSN with three segments and I3D, using ResNet-18, ResNet34, and ResNet-50 backbones.Source: §C.2 · p. 44
• The cinematic-style benchmark contains 46K shots from 8K trailers, split into 26K training, 7K validation, and 13K test shots.Source: §C.2 · p. 44
• Style models use cross-entropy and train for 60 epochs with batch-size-128 SGD, momentum 0.9, ImageNet initialization, and learning-rate reductions at epochs 20 and 40.The initial learning rate is 0.001.Source: §C.2 · p. 45

C.3 Character Detection

• Character detection uses 1.3M bounding boxes from 758K keyframe images, split into 692K training and 66K test images, and evaluates mAP across IoU thresholds 0.5–0.95 in 0.05 increments.The metric follows COCO.Source: §C.3 · p. 45
• The detection setup compares Faster R-CNN with ResNet-50 across COCO, CalTech, and MovieNet-PDet to examine domain gap, then tests RetinaNet and Cascade R-CNN with ResNeXt-101 and feature pyramid on MovieNet.Source: §C.3 · p. 45

C.4 Character Identification

• Character identification searches for all instances of a character in a movie from one portrait, using a split of 639.9K, 336.6K, and 364.2K instances.The train, validation, and test splits contain 2,088, 821, and 876 identities, respectively; credited-cast homepage portraits serve as queries, and evaluation uses mAP.Source: §C.4 · pp. 45–46
• The authors conclude that Market and CUHK03 cannot support movie character analysis because of the large domain gap from MovieNet.They train ResNet-50 with softmax loss on Market, CUHK03, and MovieNet for comparison, and also compare LP and PPCC.Source: §C.4 · p. 46
• Character identification combines face and body features; its Two-Step method retrieves by face, adds high-confidence instances, then performs body-feature set-to-set retrieval.Face features come from ResNet-101 trained on MS1M, while body features use ResNet-50 trained on MovieNet identity annotations or another ReID dataset. LP is noise-sensitive; PPCC uses a competitive consensus scheme and is described as the current state of the art.Source: §C.4 · p. 46

C.5 Scene Segmentation

• MovieNet’s 42K scene segments are split into 25K, 8.9K, and 8.1K train, validation, and test segments, with no movie overlap.Evaluation uses average precision of detected boundaries per movie and a distance-weighted mIoU; baselines include Grouping, Siamese, and MS-LSTM.Source: §C.5 · p. 47
• MS-LSTM extracts audio, character, action, and place features from each shot for scene segmentation.The audio features separate speech and background sound; character features use detected instances and a MovieNet-trained identity model, action features use TSN pretrained on AVA, and place features use ResNet-50 pretrained on Places.Source: §C.5 · p. 47
• Grouping improved 98% from 0.170 to 0.336, Siamese 34% from 0.268 to 0.358, and MS-LSTM 39% from 0.334 to 0.465.The passage reports performance improvements for models trained on MovieNetSSeg and says jointly using audio, character, action, and place information surpasses any single element.Source: §C.5 · p. 47

C.6 Action Recognition

• Action recognition predicts multiple tags per clip using a movie-disjoint 23,747/7,543/9,969 train/validation/test split and mAP evaluation.The baselines use binary cross-entropy; TSN, I3D, and SlowFast are trained with distinct architectures and schedules.Source: §C.6 · pp. 47–48

C.7 Place Recognition

• For place classification, TSN achieves 8.33 mAP versus 7.66 for I3D.Source: Table C8 · p. 48

C.6 Action Recognition

• SlowFast performs best and TSN worst on MovieNet action classification, with SlowFast outperforming the other baselines by a large margin.The results are reported in Table C7, and Figure C1 presents SlowFast’s per-class AP scores.Source: Table C7 · p. 48, §C.6 · p. 49, Figure C1 · p. 49

C.8 Story Understanding

• Synopsis retrieval searches for a relevant movie segment from a paragraph query, using 2,422/867/919 train/validation/test samples and Recall@K and MedR metrics.The reported extended results are in Table C9; the supplied passages do not give its cell values.Source: §C.8 · p. 50, Table C9 · p. 50
• The VSE retrieval model adds appearance, subtitle, action, and character nodes; it averages embeddings for the first three and uses KuhnMunkres matching for cast features.Cinematic-style probabilities produce attention weights for shot elements, while event-flow and character-interaction modules model spatial and temporal graph relations.Source: §C.8 · p. 51
• Retrieval features combine shot-level ResNet-101 appearance, Word2Vec subtitle and action features, and detected character representations with synopsis text and named-character features.Synopsis action verbs are selected from frequent verbs and filtered for visually observable actions; character portraits are retrieved using IMDb.Source: §C.8 · pp. 51–52

D Toolbox

• MovieNet’s toolbox, released with the dataset and benchmark code, provides crawlers, multimedia preprocessing, data generators, movie-alignment tools, and parsers for data and metadata.Examples include extracting audio, cutting and resizing movies, generating shot-detection data, and visualizing character boxes or movie genres.Source: D Toolbox · p. 52