Source-linked AI summary
TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film
Kaishuu Shinozaki-Conefrey, Olivier Pascaud, Robin Courant, Xi Wang, Dimitris Samaras, Vicky Kalogeiton
TL;DR
Current multimodal evaluations largely test what happens in films rather than why filmmakers present scenes through particular audiovisual choices. TAKE 85 introduces a 398-film benchmark with expert-verified intent questions and controlled modality evaluation. Models recognize perceptual content but struggle with directorial intent, with the best reported model scoring 58.1 on specific questions.
Problem
Existing multimodal benchmarks mainly evaluate what happens on screen and largely omit the communicative role of cinematographic and auditory decisions.
Method
TAKE 85 evaluates directorial intent on 398 films using expert annotations, fixed intent questions, and controlled combinations of visual, audio, and textual inputs.
Results
Models recognize objects, actions, and narrative events but consistently struggle with intentional reasoning; Gemini-3.5-Flash scores 58.1 on specific questions.
Takeaways & Limitations
Directorial intent exposes a previously overlooked dimension of multimodal understanding and a gap between recognizing audiovisual content and understanding its intended meaning.
Abstract
from arXiv · showhide
Films communicate through deliberate creative choices, including lighting, color, composition, editing, dialogue, music, and sound. Humans naturally interpret these signals as directorial intent, yet current multimodal large language models (MLLMs) are evaluated almost exclusively on understanding what happens rather than why it is presented that way. We introduce TAKE 85, the first benchmark for directorial-intent understanding, comprising 398 short films (85 hours) with expert-verified question-answer pairs spanning global and fine-grained visual and audio intent. Through controlled modality ablations, TAKE 85 enables systematic evaluation of multimodal reasoning. Experiments on state-of-the-art MLLMs reveal a substantial gap between perceptual recognition and intentional understanding: while models accurately describe events and narratives, they consistently fail to infer the communicative role of filmmaking decisions. Our results establish directorial intent as a previously overlooked dimension of multimodal understanding: even the strongest model reaches only 58 out of 100, and our ablations show that no input modality is sufficient on its own. All code, Q&As, and models are publicly available from https://github.com/KaiShinozakiConefrey/Take-85
1 Introduction
TAKE 85 targets a gap in multimodal evaluation: models are tested on what happens in films, but rarely on why filmmakers present scenes through particular audiovisual choices. It introduces a benchmark and controlled evaluation framework for measuring directorial-intent understanding across modalities and filmmaking dimensions.
- Existing benchmarks mostly evaluate events, characters, dialogue, and perceptual capabilities rather than the communicative role of cinematographic and auditory decisions.
- Directorial-intent understanding requires jointly reasoning over visual and audio information to infer the communicative purpose behind filmmaking decisions.
- The benchmark decomposes intent into overall, visual, audio, lighting, color, composition, staging, dialogue, music, and sound-design dimensions.
- TAKE 85 is the first benchmark for directorial intent, containing 398 short films with expert annotations spanning global and fine-grained filmmaking decisions.
- Controlled modality combinations enable systematic analysis of how visual, audio, and textual inputs contribute to directorial-intent understanding.
- State-of-the-art MLLMs recognize objects, actions, and narrative events but consistently struggle to infer why scenes were constructed through particular creative decisions.
2 Related work
Prior video benchmarks emphasize factual, plot-level, and cinematographic recognition, while TAKE 85 focuses on the filmmaker’s intended communicative meaning. This places directorial intent beyond conventional descriptions of what appears on screen.
- Video understanding and audio description primarily describe who, what, where, and when, without addressing why a filmmaker chose a particular presentation.
- Existing benchmarks cover factual video question answering, plot comprehension, temporal reasoning, and related explanatory or counterfactual questions.
- TAKE 85 organizes questions across general, visual, and audio categories, providing a benchmark structure for intent-oriented evaluation.
- Even the strongest MLLMs struggle to identify cinematographic technique.
- Cinematography research classifies shot scale, angle, movement, composition, color, and related formal properties, with some work approaching holistic style and directorial intent.
3 TAKE 85: A directorial intent dataset
TAKE 85 is built from 398 short films and 4,776 fixed directorial-intent questions spanning global and fine-grained visual, audio, and general filmmaking decisions. Ground-truth answers combine creator metadata, multimodal film evidence, LLM synthesis, and expert verification.
- TAKE 85 contains 398 short films totaling approximately 85 hours, each paired with plot and director-provided intent summaries.
- Each film has 12 fixed questions, yielding 4,776 Q&A pairs across global and fine-grained intent categories.
- Films were selected from SF20K through metadata filtering, beginning with Omeleto’s curated short-film pool and its hand-written intent metadata.
- The question taxonomy covers general techniques, themes, and narrative structure; visual lighting, color, and composition; and audio music, sound design, and dialogue.
- Ground-truth answers combine visual captions, audio analysis, subtitles, sampled keyframes, and hand-written creator metadata in one LLM call.
4 Evaluation protocol
The evaluation gives every model the same fixed questions and film-derived input channels while withholding creator intent metadata. Free-form answers are scored with holistic and claim-level LLM-judge metrics broken down by intent category.
- Every model answers the same 12 questions using only visual captions, audio analysis, subtitles, and sampled keyframes; creator metadata is withheld.
- Input channels are supplied separately so arbitrary non-empty modality subsets can be withheld without changing the remaining input format.
- Single-pass graded uses one holistic 1–5 judgment of substantive agreement, with final scores rescaled to 0–100.
- Checklist scoring decomposes each reference into atomic claims and reports the fraction of claims supported by the prediction, rescaled to 0–100.
- Scores are reported for overall, visual, and audio intent, plus general, visual, and audio groups of fine-grained questions.
5 TAKE 85 benchmark
TAKE 85 evaluates directorial-intent understanding across standardized questions and audiovisual input settings. Results show that models perform substantially better with full context than with isolated modalities, while performance varies more by directorial device than by modality.
- Benchmark and evaluation: TAKE 85 evaluates ten MLLMs using full-context inputs comprising visual captions, subtitles, audio analysis, and sampled keyframes.The evaluation includes open-weight scaling ladders and Gemini 3.5 Flash as a proprietary reference.
- Full-context results: Gemini-3.5-Flash leads full-context performance at 62.0 simple and 58.1 specific, followed by Qwen3.5-27B at 46.4 and 45.9.The same three models occupy the top three positions under checklist scoring.
- Modality ablations: Single modalities specialize by category: captions-only and frames-only score 0.3 and 0.2 on audio questions, while audio-only scores 0.0 on visual questions.Checklist scoring shows the same collapse across opposite categories.
- Modality ablations: Subtitles provide the largest gain for audio questions, raising audio-only performance from 38.1 / 7.8 to 51.2 / 11.4 across single-pass and checklist scoring.Captions provide a modest gain, while frames provide essentially none.
- Modality ablations: The richest modality combination is not consistently best: frames+captions+subtitles outperform full context on every visual column and two of three general columns.Adding audio reduces all five of those scores, while different combinations lead different audio questions.
- Best and worst questions: Difficulty follows directorial device rather than modality: all 33 sound-design and 20 technique questions fall in the worst set, whereas color, music, lighting, and themes exceed 80% correct in the best set.This ordering holds despite some devices sharing the same input category.
- Best and worst questions: Gemini 3.5 Flash averages 35 and answers 41 of 97 scored worst-set questions acceptably, while six open-weight models average below 10 and answer at most 10.The comparison indicates that the worst questions are not intrinsically unanswerable.
6 Conclusion
TAKE 85 introduces a benchmark for directorial intent and shows that current multimodal models struggle to infer the communicative purpose of filmmaking choices. The failure is systematic across modalities and especially pronounced for sound design and directorial technique.
- Conclusion: TAKE 85 is introduced as the first benchmark for evaluating directorial intent in films.The benchmark targets whether models understand why audiovisual choices are presented, not only what appears on screen.
- Conclusion: No input modality is sufficient on its own, and difficulty follows directorial device rather than modality.This conclusion is supported by the benchmark’s modality ablations and device-level error patterns.
- Conclusion: Sound design and directorial technique defeat every evaluated open-weight model.These devices occupy the hardest region of the benchmark’s best-versus-worst comparison.