PaperlayerView exact revision on arXiv

Source-linked AI summary

Movie101: A New Movie Understanding Benchmark

Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, Qin Jin

arXiv:2305.12140v2cs.CVcs.MM

TL;DR

Movie narration benchmarks often simplify the task, while producing narration scripts is costly for visually impaired audiences. Movie101 introduces Chinese movie-narration and grounding tasks with external knowledge; its knowledge-enhanced methods outperform baselines, and MNScore aligns with human evaluation.

  • Problem

    Movie narration requires costly scripts, and existing approaches simplify the task in ways that do not fully reflect real movie-narration needs.

    Source: §1 · pp. 1–2

  • Method

    Movie101 provides a Chinese benchmark with role-aware narration for clips without actor dialogue, Temporal Narration Grounding, and external movie knowledge.

    Source: §1 · p. 2

  • Results

    For MCN and TNG, knowledge-enhanced methods outperform carefully designed baselines, and MNScore aligns with human evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Movie101 supports research on Chinese movie narration and grounding, and experiments validate the importance of external knowledge including genres and roles.

    Source: §1 · p. 2, §6 · p. 8

    Coherence across different clips within a movie is not studied and remains future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

To help the visually impaired enjoy movies, automatic movie narrating systems are expected to narrate accurate, coherent, and role-aware plots when there are no speaking lines of actors. Existing works benchmark this challenge as a normal video captioning task via some simplifications, such as removing role names and evaluating narrations with ngram-based metrics, which makes it difficult for automatic systems to meet the needs of real application scenarios. To narrow this gap, we construct a large-scale Chinese movie benchmark, named Movie101. Closer to real scenarios, the Movie Clip Narrating (MCN) task in our benchmark asks models to generate role-aware narration paragraphs for complete movie clips where no actors are speaking. External knowledge, such as role information and movie genres, is also provided for better movie understanding. Besides, we propose a new metric called Movie Narration Score (MNScore) for movie narrating evaluation, which achieves the best correlation with human evaluation. Our benchmark also supports the Temporal Narration Grounding (TNG) task to investigate clip localization given text descriptions. For both two tasks, our proposed methods well leverage external knowledge and outperform carefully designed baselines. The dataset and codes are released at https://github.com/yuezih/Movie101.

1 Introduction

Movie101 addresses gaps in movie-narration benchmarks with a Chinese dataset, more realistic narration and grounding tasks, and an evaluation metric designed to align with human judgments. Its design targets long, role-aware plots and narration where actors are not speaking.

1 Introduction

  • Audio description is costly to produce, limiting movie access for visually impaired audiences; the paper also motivates a Chinese benchmark for a large Chinese-speaking audience.The introduction reports more than 17 million visually impaired people among Chinese mother-tongue speakers.Source: §1 · pp. 1–2
  • Prior movie-narration datasets include MPII-MD and M-VAD, but their aligned scripts or audio descriptions do not resolve the benchmark gaps addressed here.The earlier datasets are among the works supporting automatic movie narration.Source: §1 · p. 1
  • Earlier benchmarks use short shot-level descriptions, omit concrete role names, and rely on n-gram metrics that can penalize semantically correct wording.These limitations make coherent longer-plot narration and role-to-plot connections difficult to evaluate.Source: §1 · p. 1
  • Movie101 covers 101 Chinese movies, with 30,174 narration clips totaling 92 hours and 14,109 longer clips for movie-clip narration.The longer clips average 20.4 seconds and 80.7 Chinese characters.Source: §1 · p. 2
  • MCN generates role-aware narration for clips without actor dialogue, combining narration clips between dialogues to form longer inputs.The authors note actor-line timestamps can help identify where narration is needed in a new movie.Source: §1 · p. 2
  • Movie101 also supports Temporal Narration Grounding and introduces MNScore, which aligns with human evaluation.The benchmark evaluates existing methods and proposes externally knowledge-enhanced methods for both tasks.Source: §1 · p. 2

2 Related Works

Related work covers movie narration datasets, video-captioning approaches, and temporal sentence grounding. The paper builds on these task lines while targeting role-aware movie narration and grounding across longer movie content.

Datasets

  • M-VAD and MPII-MD provide aligned movie clips and narrations or scripts, and were merged into LSMDC; TVC adds short TV-show clips, while existing datasets are English.M-VAD clips average 6.2 seconds and MPII-MD clips 3.9 seconds.Source: §2 · pp. 2–3

Video Captioning

  • Video captioning progressed from templates to sequence generation, while dense captioning commonly detects proposals before describing them; OVP instead generates paragraphs directly.Identity-aware movie narration labels people as PERSON1 or PERSON2 but does not generate concrete role names [Park et al., 2020].Source: §2 · p. 3

Temporal Sentence Grounding

  • Temporal sentence grounding typically proposes and ranks candidate moments; later work improves modality interaction or detection heads, and this paper adds external knowledge to IA-Net.IA-Net iteratively interacts inter- and intra-modal features, while Li et al. (2022) learn fine-grained alignment across video and query hierarchies.Source: §2 · p. 3

3 Dataset

Movie101 collects movies, narration, dialogue, and metadata, then organizes them into datasets for narration and temporal grounding. It contains 30,174 short clips and includes longer narration clips and movie-scale grounding videos.

3.1 Data Collection

  • The dataset collection gathers all 101 movies then available from Xigua Video’s barrier-free channel and crawls titles, genres, directors, actors, and role information.Role information includes actor names, role names, portraits, and role rankings.Source: §3.1 · p. 3
  • Human annotators correct ASR transcripts, remove non-narration material, and merge narration fragments into clips using timing, length, and punctuation criteria.Fragments less than one second apart are merged, with a 100-character threshold to limit over-merging.Source: §3.1 · p. 4
  • Movie101-N groups narration paragraphs between adjacent actor dialogues, while Movie101-G pairs narration queries with aligned videos for temporal grounding.Lines less than five seconds apart are grouped into dialogues; validation and test sets each select 10 movies of different genres.Source: §3.1 · p. 4

3.2 Dataset Statistics

  • Movie101 covers 101 movies, 41 genres, and 645 roles; comedy, romance, and action are its three most represented genres.Movies can belong to up to four genres.Source: §3.2 · p. 4, Figure 2 · p. 4
  • 30,174 narrated clips average 11.0 seconds and 47.3 Chinese characters; Movie101-N has 14,109 clips averaging 20.4 seconds and 80.7 characters.Movie101-G contains 30,174 clips drawn from 101 movies, with average video length of 6,144 seconds.Source: §3.2 · p. 5
  • Movie101-N clips vary substantially in length, and Table 1 compares Movie101 with other movie-narrating and temporal-grounding datasets.Movie101-N clips and descriptions are longer than those in existing movie-narrating datasets.Source: Table 1 · p. 4, §3.2 · p. 5
  • Movie101 narrations are also characterized by distributions of role-name and action counts per clip.HanLP POS tagging and movie metadata are used to locate role names and action words.Source: Figure 3 · p. 4, §3.2 · p. 5

4 Movie Clip Narrating

Movie Clip Narrating (MCN) asks models to generate plot-related paragraphs from clips, using movie genres and role portraits as external knowledge. The section presents RMN, evaluates narration metrics against human judgments, and reports gains over baselines, especially in role awareness.

4.1 Task Description

  • MCN generates plot-related paragraph descriptions from movie clips and provides role portraits to support accurate descriptions of action subjects.The task also provides genre information because narration styles may vary across movie genres.Source: §4.1 · p. 5

4.2 Proposed Method

  • RMN encodes video, frame-level faces, and genre features, then uses a context-conditioned pointer to select cast role names during narration generation.A dynamic memory bank refines video representations at each decoding step.Source: §4.2 · pp. 5–6

4.3 Evaluation

  • Movie narrations can have multiple valid phrasings despite a single reference, making text matching inadequate for evaluating paragraph quality.The authors compare objective metrics against human rankings of candidate narrations.Source: §4.3 · p. 6
  • MNScore aligns best with human evaluation, while BERTScore outperforms n-gram metrics in narration evaluation accuracy.The metric combines EMScore, BERTScore, and RoleF1.Source: §4.3 · p. 6

4.4 Experiments

  • RMN outperforms Vanilla Transformer and OVP, especially on RoleF1; adding video-face features significantly improves role awareness.The authors attribute the role-name generation gains to the pointer network and the benefits of bridging video content with actor portraits.Source: §4.4 · p. 7, Table 3 · p. 7

5 Temporal Narration Grounding

Temporal Narration Grounding (TNG) locates a clip’s start and end times in a movie from a narration query. The proposed two-stage method retrieves an approximate shot before local grounding, and role-aware RNL outperforms baselines in the reported experiments.

5.1 Task Description

  • TNG predicts a target clip’s starting and ending times in the whole movie from a clip narration query.The task is framed as helping an AI agent locate clips of interest during movie entertainment.Source: §5.1 · p. 7

5.2 Proposed method

  • The two-stage framework first retrieves a coarse global shot anchor, then performs local temporal grounding to determine the precise timestamp.Global retrieval divides movies into 20-second shots and selects the shot most similar to the text query.Source: §5.2 · p. 7
  • The first-stage retrieval model transfers ChineseCLIP to video-text matching and fine-tunes it with contrastive learning on aligned Movie101 shot-narration pairs.Shot frames are encoded separately, with their CLS-token features mean-pooled into a video feature.Source: §5.2 · p. 7
  • RNL grounds the target within a 200-second window around the anchor shot, using face features and role-name portraits alongside video and text representations.The video encoder uses a bidirectional GRU, while role portraits are encoded as visual tokens in the query.Source: §5.2 · p. 8

5.3 Experiments

  • RNL outperforms baselines with role-aware video and text encoding; either face-feature stream beats IA-Net, while both achieve the best local-grounding performance.In combined inference, k-way re-ranking improves Rank@5 but harms Rank@1.Source: Table 5 · p. 7, §5.3 · p. 8, Table 6 · p. 8

6 Conclusion

Movie101 is a Chinese movie-understanding benchmark with Movie Clip Narrating and Temporal Narration Grounding tasks. Its proposed metric and methods leverage external knowledge, while experiments show a remaining gap from expert annotations.

6 Conclusion

  • Movie101 supports realistic Movie Clip Narrating and Temporal Narration Grounding tasks for movie understanding.The conclusion describes MCN as addressing automatic movie description and TNG as more challenging than previous TSG benchmarks.Source: §6 · p. 8
  • MNScore is designed for human-preference-compatible narration evaluation, and experiments validate the importance of genres and roles for movie understanding.Source: §6 · p. 8
  • The proposed models still have a significant gap from expert annotations, so further research is needed to help visually impaired people enjoy movies with AI.Source: §6 · p. 8

Limitations

The task uses narration paragraphs and longer video clips to support within-movie coherence, but coherence across separate clips remains unstudied.

Limitations

  • MCN uses narration paragraphs as ground truth and longer video clips as input to advance within-movie coherence.Source: Limitations · p. 9
  • The work does not study coherence across clips, which it says requires models to comprehend the whole movie and connect different plots.The authors leave this higher-level capability for future investigation.Source: Limitations · p. 9

Ethics Statement

The paper describes its data source and crowdsourcing practices, and reports that narration clips cover most dialogue gaps in the collected movies.

Ethics Statement

  • The collected movies are publicly available on Xigua Video and permitted for crawling under the site's service contract; only movie URL lists will be released due to copyright concerns.The source contains no information naming or uniquely identifying individuals and no offensive content.Source: Ethics Statement · p. 9
  • Twenty Chinese college students were recruited to clean and review narration text, with stated per-movie payments of 50 RMB and 25 RMB, respectively.Workers were informed of the data's future use before annotation began.Source: Ethics Statement · p. 9

A Narration Distribution

  • The 30,174 narration clips fill 95.3% of 15,206 dialogue gaps by count and cover 92.9% of their total duration across the 101 movies.The authors define no-actors-speaking clips as scenes without verbal dialogue, whether actors are visually present or absent.Source: A Narration Distribution · p. 11

B Dataset Quality Description

The dataset uses two annotation stages to clean and review narrations, and a manual sample evaluation reports high text consistency and mostly coherent paragraphs.

B Dataset Quality Description

  • Two worker groups clean the data against guidelines, then check and correct the annotations in a second stage.Paragraph-division heuristics were based on the authors' observation experience.Source: B Dataset Quality Description · p. 11
  • In 300 sampled paragraphs, 96.7% matched original AD textually, while 90% were complete and coherent, 7.7% needed context merging, and 2.3% needed division.The authors conclude the narrations are of good quality for downstream tasks.Source: B Dataset Quality Description · p. 11

C Implementation Details

The implementation details specify how candidate narrations and evaluation metrics are constructed, and report training hyperparameters and computational burden in Table 7.

C Implementation Details

  • Five candidate narrations per sampled clip include outputs from two baselines and RMN, plus two ground-truth variants with role names or nouns and verbs replaced.The baselines are Vanilla Transformer and OVP.Source: C Implementation Details · pp. 11–12
  • Table 7 reports key model-training hyperparameters and computational burden, with each model's results derived from a single run.Other implementation choices include ChineseCLIP-huge for CLIPScore and EMScore, BERT-base-Chinese for BERTScore, averaged 1-gram and 2-gram diversity for DIV, and causal Ernie 3.0 for PPL.Source: Table 7 · p. 12, C Implementation Details · p. 12
  • RoleF1 combines recall of ground-truth roles with precision to account for generated role names that may be hallucinated.CLIPScore uses mean-pooled features from 10 uniformly selected frames and the overall text feature; EMScore uses all selected frame and textual token features.Source: C Implementation Details · p. 12

D Qualitative Result

Qualitative examples show RMN generating role names where the baselines fail, while the proposed two-stage grounding method uses a retrieved anchor shot to help locate target clips.

D.1 Movie Clip Narrating

  • RMN relates roles in video clips to their role names, while Vanilla Transformer and OVP can mention actions but fail to generate correct roles absent from training.The qualitative cases also show MNScore ranking candidates more consistently with human rankings; Figure 6 marks correct and incorrect role names and metric-ranking agreement in green and disagreement in red.Source: §D.1 · p. 12, Figure 6 · p. 13

D.2 Temporal Narration Grounding

  • Global Shot Retrieval finds an anchor shot near the target clip, which helps Local Temporal Grounding locate the final target.In the examples, the prediction intervals are 183–205s, 2102–2122s, and 724–746s, compared with ground-truth intervals of 182–200s, 2095–2121s, and 725–748s, respectively.Source: §D.2 · pp. 12, 14

Related papers

Loading 2305.12140v2…