Source-linked AI summary
Evaluating Multimodal Narrative Understanding of Popular Hollywood Films
David Bamman, Kent K. Chang, Allison Cooper, Juishan Hsu, Reina Kushihashi, Madison Mar, Arnav Podichetty, Rachael Samberg, Ipek Nil Sancak, Yuhan Shao
TL;DR
Stable Hollywood-film benchmarks are difficult to construct because copyright status is uncertain and popularity captures only one form of cultural significance. The paper builds a popularity- and public-domain-based collection, publishes historical Variety box-office data, and evaluates narrative understanding with a multimodal benchmark. Many vision-language models perform near chance, while the strongest audio-visual configuration reaches 61.1% accuracy, below human-level performance.
Problem
Hollywood-film benchmarks must balance copyright stability with cultural significance, while prior evaluations risk measuring memorized movie knowledge rather than narrative understanding from film content.
Method
The authors combine Variety box-office popularity with copyright-registration and renewal research, then construct a multimodal narrative benchmark and filter questions for memorization.
Results
Many vision-language models perform near chance, while audio-visual models reach a maximum accuracy of 61.1%, below human-level performance.
Takeaways & Limitations
The benchmark provides a proving ground for comparing multimodal models on complex narrative questions grounded in popular Classical Hollywood films.
Takeaways & Limitations
Popularity encodes only one form of cultural significance, excluding films that circulated outside mainstream U.S. distribution, while researcher-created questions reflect U.S. university perspectives.
Abstract
from arXiv · showhide
Multimodal language models increasingly show promise for enabling the large-scale computational analysis of film, opening up new avenues for learning about film history and the evolution of narrative techniques. But the creation of stable benchmarks built around Hollywood films is complicated by copyright protections. In this work, we address these concerns directly, by building a new collection of Hollywood films defined by two criteria: box office popularity (where we publish the first large-scale, open collection of weekly box office earnings reported by Variety magazine from 1922-1979); and likely public domain status (by researching copyright registrations and renewals in the US Catalog of Copyright Entries). We build a new multimodal MCQ benchmark on top of this collection that focuses on narrative elements that directly evaluate the abilities of models to inform meaningful research on film narrative; we find that many vision-language models struggle on this task (with many performing at near-chance levels of accuracy), while audio-visual models (including those that use audio in captioning scenes) reach a maximum accuracy of 61.1%, well below human-level performance.
1 Introduction
The paper addresses the difficulty of building culturally significant, copyright-stable Hollywood-film benchmarks by combining popularity-based collection criteria with public-domain research. It then introduces a narrative-understanding benchmark showing substantial challenges for multimodal models.
- Film analysis with NLP, computer vision, and AI has examined pacing, luminosity, race, gender, and comedic timing at scale.
- Public-domain movie benchmarks face uncertain copyright status, while culturally significant films are not necessarily represented by arbitrary public-domain collections.
- The collection measures U.S. cultural significance through box-office popularity and likely public-domain status through US Catalog of Copyright Entries research.
- The paper presents the first systematic database of historical box-office earnings from Variety, spanning 1922–1979.
- The authors introduce a dataset of popular films likely in the public domain and a narrative benchmark covering temporality, plot, character, setting, perspective, representation, symbolism, and objects.
- Many vision-language models perform near chance, while audio-visual models reach at most 61.1% accuracy, below human-level performance.
2 Defining the collection
The collection is defined by historical box-office popularity and carefully investigated likely public-domain status. Variety records are extracted and ranked into a large open dataset, while copyright and underlying-source checks narrow the final film corpus.
- 2.1 Identifying popular movies: Variety provides weekly box-office receipts for individual movies in specific theaters from 1922 onward, supporting historical popularity measurement.
- 2.1 Identifying popular movies: The extraction represents each record as a structured tuple containing location, theater, movie title, and weekly box-office estimate.
- 2.1 Identifying popular movies: Year-specific alias mappings connect terse Variety movie references to IMDb records, enabling yearly rankings by reported box-office totals.
- 2.1 Identifying popular movies: 4,072 manually labeled tuples from 21 Variety issues form a gold-standard evaluation, with rank correlation serving as the primary validity measure.
- 2.1 Identifying popular movies: The resulting open dataset contains 1.4M weekly box-office numbers for over 24,000 movies across 1922–1979.
- 2.1 Identifying popular movies: Films enter the collection when they rank among the top 100 by yearly box office, but this proxy excludes works outside mainstream U.S. distribution.
- 2.2 Investigating public domain status: Likely public-domain status is investigated through registration and renewal records, with unmatched records manually reviewed and films based on renewed copyrighted sources removed.
- 2.2 Investigating public domain status: Applying popularity and likely public-domain criteria, while restricting source content to sound films, produces a dataset of 61 popular movies.
3 Building a benchmark
The benchmark evaluates long-form narrative understanding in popular films through multiple-choice questions grounded in scenes and organized across eight narrative categories. Its construction accounts for copyright complexity, question-writing scale, and expert human performance.
- 3 Building a benchmark: The benchmark targets long-form narrative understanding in films whose copyright status requires attention to publication, registration, and renewal history.Prior to the 1976 Act, protection depended on authorship, publication status, registration date, and renewal.
- 3 Building a benchmark: The task uses four-option multiple-choice questions with one correct answer, organized across eight narrative categories.Categories include temporality, plot, character, setting, perspective, representation, symbolism, and object identification.
- 3 Building a benchmark: 779 questions were initially created across 61 films, averaging 12.8 questions per film.Seven co-authors watched each movie in full and wrote the questions.
- 3 Building a benchmark: 82.0% human accuracy was achieved on 133 questions covering ten movies, with errors partly reflecting ambiguity and difficult information-seeking items.Questions used plain-language scene descriptions rather than explicit timestamps and avoided obviously off-topic distractors.
4 Memorization
Popular-film questions risk measuring memorized online knowledge rather than access to film content. The authors identify this contamination and filter questions using title-and-date prompting to reduce it.
- 4 Memorization: 66.3% accuracy was previously reported for Gemini 2.5 Pro answering movie questions from title and release date alone, versus 50% random performance.This motivates concern that online movie discussions enter language-model pre-training data.
- 4 Memorization: Gemini, Opus, and GPT showed memorization rates of 39.4%, 40.7%, and 38.8%, respectively, under title-and-date prompting.The filtering procedure retained questions for which aggregate performance was approximately 25%, yielding 628 benchmark questions.
5 Experiments
The experiments compare multimodal models and compression paradigms for long-form narrative questions, finding that audio and dialogue access matter more than vision alone. Clip-based captioning approaches native full-video processing for the strongest model, while frame budgets have limited impact.
- 5.1 Setup: The benchmark evaluates closed-book, subtitles-only, end-to-end, Socratic captioning, and agentic retrieval paradigms across long-video, vision-language, and closed-source models.Contexts vary from no movie input to subtitles, sampled frames, captions, or query-conditioned retrieval.
- 5.2 Results: No closed-book model exceeds a 25% lower confidence bound, while subtitles-only accuracy reaches 48.1 for Gemini, 39.2 for GPT, and 36.9 for Claude.These results establish that the questions require movie content and that dialogue provides a strong baseline.
- 5.2 Results: 61.1 ±3.8 is Gemini 3 Flash’s full-video accuracy, compared with 58.9 ±3.8 using clip-based Socratic captions.The confidence intervals overlap, so the two configurations show no meaningful performance gap.
- 5.2 Results: Frame-image retrieval barely exceeds chance, whereas caption retrieval and Socratic performance vary across QA backbones.The frame-retrieval loop omits audio and dialogue, while caption-based methods preserve access to that information.
- 5.2 Results: Frame budgets have limited impact: the largest within-backbone gain is +3.6 pp from N=64 to N=256, with heavily overlapping confidence intervals.The study therefore uses canonical budgets of N=64 for long-video models and N=256 for vision-language models.
- 5.2 Results: Gemini rises from 39.3% with 256-frame visual input to 61.1% with full video including audio, a gain of +21.8 pp.Within Socratic, replacing frame captions with audio-inclusive clip captions also significantly improves all three models.
- 5.2 Results: Per-film performance correlates positively with subtitles-only accuracy, with ρsubtitles ranging from +0.40 to +0.52 and all p ≤0.002.Across closed models, negative ρhits and ρtitle values do not indicate easier performance for films with greater web presence or recognizable imagery.
- 5.2 Results: Gemini’s parametric film knowledge leaves a per-film signal in downstream Claude and GPT-5-mini clip-captioned Socratic performance.Their correlations with Gemini’s closed-book ranking are +0.37 and +0.22, respectively.
6 Conclusion
The paper introduces a stable Classical Hollywood narrative benchmark built from popular films whose copyright status is unlikely to be challenged. Its questions target complex narrative elements and remain difficult for frontier multimodal models.
- 6 Conclusion: The collection combines popularity measured by Variety box-office numbers with likely public-domain status based on release date or copyright renewal history.The benchmark and supporting data and code are publicly available.
- 6 Conclusion: Questions cover complex narrative elements including plot, character, setting, and perspective.The benchmark is designed as a proving ground for comparative model performance in computational film analysis.
- 6 Conclusion: Frontier multimodal language models find the benchmark challenging.The paper positions the benchmark for assessing models used in large-scale computational analysis of film.
7 Limitations
The benchmark’s scope is constrained by its popularity-based collection criterion, U.S.-university question authorship, and focus on Classical Hollywood cinema through 1963.
- 7 Limitations: Popularity captures only one form of cultural significance and omits films circulated outside major U.S. metropolitan distribution.Examples include silent-era “race” films and transnational festival films.
- 7 Limitations: The benchmark’s narrative salience reflects questions created by researchers of varying disciplinary backgrounds at U.S. universities.This authorship context influences which narrative aspects are treated as salient.
- 7 Limitations: The benchmark covers Classical Hollywood cinema through 1963, so performance may not generalize to films outside that period.The authors describe this temporal restriction as a trade-off for defining a collection suitable for stable additional benchmarks.
- 7 Limitations: The public-domain investigation process can be applied to define new collections under alternative criteria.This provides a route for addressing some collection-scope constraints without changing the status-research procedure.
B Box office extraction accuracy
The extraction pipeline converts Variety pages into structured box-office tuples, aggregates them into yearly movie rankings, and evaluates accuracy with rank, movie-set, and exact-tuple metrics.
- The pipeline extracts ⟨city, theater, movie, gross⟩ tuples from Variety pages and maps movie titles to IMDB identifiers.
- It aggregates gross values for each movie and generates rankings over movies by their total gross.
- Spearman rank correlation measures agreement between predicted and gold gross-based rankings for movies appearing in both sets.
- Movie F1 measures whether predicted movie lists invent nonexistent titles or omit titles present in gold annotations.Precision is the fraction of predicted movies appearing in gold annotations, while recall is the fraction of gold movies appearing in predictions.
- Exact-tuple accuracy evaluates whether movie title, theater, city, and gross are all extracted correctly.
- Table 3 averages each metric across Variety issues, giving every issue equal weight.
C Sample registrations
The sample registration materials illustrate how copyright registrations and renewals are documented for assessing films’ likely public-domain status.
- Figure 3 shows The Bells of St. Mary’s registration in the Catalog of Copyright Entries, Cumulative Series 1940–1949.
- The renewal references the original registration date and number, linking the later renewal to the initial record.
- The registration extraction records the film title, registration identifier, and copyright date from the CCE.
- Figure 4 shows the same film’s copyright-registration renewal in the CCE’s 1973 motion-picture volume.
D Memorization
The study filters benchmark questions to reduce the influence of memorized movie knowledge, targeting metadata-only performance near random guessing.
- The authors rank questions by how many frontier models answer them correctly from movie metadata alone and retain questions until average accuracy reaches approximately 25%.
- 39.4%, 40.7%, and 38.8% are the memorization rates for Gemini Pro 3.1, Claude Opus 4.7, and GPT 5.5, respectively.
- 628 benchmark questions remain after discarding 151 questions from the original pool.
E Frame budget sweep
Increasing frame budgets produces small, inconsistent within-model accuracy changes, so the study uses canonical budgets separately for vision-language and long-video models.
- N ∈{64, 128, 256} are the evaluated frame budgets for six end-to-end backbones.
- GPT-5-mini gains +3.6 pp from N=64 to N=256, while Qwen3-VL-8B gains +4.5 pp from N=64 to N=128.
- Within-backbone confidence intervals overlap heavily, and several backbones do not improve monotonically as frame count increases.
- The frame-budget effect is an order of magnitude smaller than the audio-access gap, motivating canonical budgets of N=64 for long-video models and N=256 for vision-language models.
F Movies in Benchmark
The benchmark’s film collection lists Hollywood titles from 1929 through 1963, together with years, genres, directors, and production companies. The entries span multiple genres and production eras.
- Each entry records a title, release year, genres, director, and production company.
- The collection covers films released from 1929 through 1963.
- The listed films represent genres including drama, comedy, romance, crime, adventure, musical, western, war, and film-noir.
- The collection includes films associated with major studios and independent production companies, including Warner Bros., Paramount Pictures, Metro-Goldwyn-Mayer, Universal Pictures, and RKO Radio Pictures.