Source-linked AI summary
As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does
Zhongrui Gui, Junyu Xie, Tengda Han, Weidi Xie, Andrew Zisserman
TL;DR
Identifying principal characters and their names directly from a film is challenging because names must be linked to people using dialogue and cinematic cues. The paper introduces a dialogue-based naming task, dataset, and models, and reports progress toward proprietary-model performance while noting that neither task is fully solved.
Problem
Linking a dialogue-mentioned name to the correct visible character can require reasoning across shots, gaze, and conversational context.
Source: §1 · p. 2Method
The paper builds annotations for dialogue-based character naming and first appearances, fine-tunes a video-and-dialogue model, and introduces a streaming character-bank pipeline.
Source: §1 · p. 2Results
The two-stage SFT+GRPO approach with color-permutation augmentation yields a +7.84% gain over the naive 8B baseline, and the paper reports that it closes most of the gap to Gemini-3.1-Pro-Preview.
Source: §5.2 · p. 7Takeaways & Limitations
The paper presents character naming and progressive character-bank construction as steps toward online, human-like movie understanding, though neither task is fully solved.
Source: §7 · p. 10Both Gemini and the authors’ model struggle with subtle cues, complex scenes with motion, and over-predicting on-screen characters when the ground-truth character is absent.
Source: §5.2 · p. 8
Abstract
from arXiv · showhide
Films introduce characters incrementally as they progress. We study the problem of identifying principal characters and assigning their names directly from the movie itself, without using external cast lists, actor photographs, or pre-built character banks. We introduce the character naming task: given a name mentioned in dialogue and the associated video context, the model must determine which visible character bears that name. This task is challenging because name resolution often depends on cinematic cues such as shot-reverse-shot structure, self-introduction, direct address, third-party reference, gaze, and temporal continuity. We provide a dataset and annotations for evaluating character naming and first character appearances, develop a video-and-dialogue model for the task, and compare it with proprietary multimodal large language models. We further introduce a progressive framework for building a character bank as the film unfolds. Together, these contributions support online, human-like movie understanding in which character identities are inferred from the film itself.
1 Introduction
The paper studies how to identify principal characters and their names from a film as it unfolds, without relying on external cast information. It introduces dialogue-based character naming, supporting annotations and models, and a progressive character-bank framework.
1 Introduction
- Prior approaches described in the paper build character banks from cast lists and actor photos, whereas this work identifies names from information in the film itself.Those prior banks are used with face or voice recognition to label principal characters.Source: §1 · p. 2
- The paper introduces character naming: given a name in dialogue, identify the person in the associated video context, a task that can require reasoning about editing, gaze, and eyeline.The addressed person may appear in a different shot, or be identified through whom the speaker looks at.Source: §1 · p. 2
- The paper introduces annotations for principal characters’ first appearances and a progressive character bank that records their appearance, voice, and names as the film unfolds.A character may appear well before being named; in Juno, the protagonist appears in the first shot but is not named until 7.2 minutes into the film.Source: §1 · p. 2
- The MCN dataset contains 23,792 multiple-choice character-naming instances across 407 movies and first-appearance annotations for 20 movies.The paper also develops a character-naming model using open-source MLLMs and compares it with proprietary MLLMs.Source: §1 · p. 2
2 Movie Character Naming (MCN) Dataset
The MCN dataset supports two complementary evaluations: matching dialogue-mentioned names to characters in video clips and annotating when principal characters first appear and are first named. Its annotations are built from character tracks and dialogue, with separate statistics and preprocessing for the tasks.
2 Movie Character Naming (MCN) Dataset
- MCN pairs dialogue name mentions with ground-truth name-to-track mappings for Character Naming, and records first visual occurrence and first naming timestamps for First Occurrence.For names referring to characters without an available face track, the Character Naming label is none.Source: §2 · p. 2, §2.1 · p. 3
2.1 Data Collection and Curation
- Character Naming annotations align body boxes with face tracks, retain names also mentioned in dialogue, and label unavailable face tracks as none; the test set is manually inspected and corrected.The figure illustrates first visual occurrence and first naming timestamps alongside ground-truth name-to-face correspondences.Source: §2.1 · p. 3, Figure 2 · p. 3
2.2 Dataset Statistics
- The Character Naming test set has 1,582 manually verified questions from 30 movies, while First Occurrence covers 20 movies, 287 characters, and 574 timestamp annotations.About 70% of Character Naming test questions involve direct address; the other evidence types include self-introduction, third-party reference, and textual cues.Source: §2.2 · p. 3
2.3 Preprocessing – Character-Level Face-Tracks and Associated Speech
- Preprocessing extracts character-level face tracks, links voice clusters to them using synchronization scores, and transcribes linked voices into visually grounded subtitles.Clips with at least one character name in the subtitles are retained for the character-naming step.Source: §2.3 · p. 4
3 Assigning Names to Characters
Character Naming requires joint reasoning over dialogue and video cues to assign a dialogue-mentioned name to a visible face track or to none. The paper adapts MLLMs with visual prompting and a two-stage fine-tuning recipe.
3 Assigning Names to Characters
- Character Naming combines textual cues such as conversational logic and vocative address with visual cues such as eye contact and on-screen presence.A named character may be off-screen, or the relevant person may be indicated by a reaction shot after a name is spoken.Source: §3 · p. 4
3.1 MLLM-Based Character Naming
- Visual prompting assigns each face track a unique color, marks its boxes in that color, and embeds color-tagged, timestamped subtitles so the MLLM can match names to identities.The prompt asks the model to associate a character name from the subtitles with a color.Source: §3.1 · pp. 4–5
3.2 Fine-tuning the MLLM for Character Naming
- The two-stage fine-tuning recipe first uses teacher-generated reasoning traces for supervised fine-tuning, then applies GRPO with rewards for response format and name–face correctness.The first stage teaches the student the expected response format before reinforcement learning.Source: §3.2 · p. 5
- Randomly permuting colors during reinforcement-learning training counters a shortcut in which the model associates a protagonist with a frequently used color.Because the ground-truth name–face mapping is invariant to color permutation, consistent reward across variants requires identity-based reasoning; the paper reports that this accounts for a significant share of gains over the cold-start baseline.Source: §3.2 · p. 5
4 Streaming Pipeline for Character Bank Construction
The pipeline builds a character bank online, updating it shot by shot from character tracks, names, and voices while retaining identities that are not yet named. Bank updates assign tracks and bind names; three safeguards address error accumulation from upstream components.
4 Streaming Pipeline for Character Bank Construction
- At each shot, the bank stores face-track clusters, character names, and voice clusters, retaining identities even before they are named.A fine-tuned MLLM extracts character information from the shot, and an update operator merges it into the previous bank.Source: §4 · p. 5
4.1 Character Bank Update
- Face tracks are assigned to the closest bank cluster when they meet a face-distance threshold; otherwise, voice matching uses a tighter threshold, and unmatched tracks create a new entry.Matched centroids are updated after assignment.Source: §4.1 · pp. 5–6
- The illustrated bank updates name an existing cluster, add face exemplars to a cluster, or create an entry for a previously unseen character.These three operations are shown as the bank evolves across incoming shots.Source: Figure 3 · p. 6
- Each entry’s canonical name is the majority-voted name in its face-track name histogram, allowing aliases and reducing the influence of occasional noisy assignments.The paper gives Juno’s names Mama Bear, Home Skillet, and General MacGuff as an example of aliases.Source: §4.1 · p. 6, §4.2 · p. 6
- Retrospective assignment adds later naming evidence to a previously unnamed cluster’s histogram, allowing its canonical name to update as evidence accumulates.The new name initially enters the histogram with a count of one.Source: §4.1 · p. 6
4.2 Error Correction during Character Bank Construction
- The streaming pipeline counters upstream error accumulation by filtering noisy tracks and query names, then resolving cross-entry name conflicts with one-to-one voting.The three mechanisms operate along the streaming chain without revisiting past observations.Source: §4.2 · p. 6
5 Experiments
The experiments evaluate character naming and streaming character-bank construction, including tests on unseen movies. The proposed models are compared with open-source and proprietary baselines, with the streaming pipeline outperforming proprietary baselines across its reported metrics.
5 Experiments
- The evaluation covers character naming against open- and closed-source models, the full streaming character-bank pipeline, and ablations of fine-tuning, augmentation, and error-correction choices.The character-bank evaluation tracks first visual appearance, first name linkage, and final-bank alignment.Source: §5 · p. 6
5.1 Evaluation Metrics
- Mean Temporal IoU measures overlap between the model’s correct-identification interval and human awareness, penalizing late or incorrect identification and weighting characters by subtitle mention frequency.The visual and name variants begin the human-awareness interval at first visual occurrence or first naming, respectively.Source: §5.1 · p. 7
- Principal Character Recall at K measures recovery of top-K principals with both a phonetically matching name and a face centroid within the identity threshold.Principals are ranked by narrative importance using IMDB, and recall is averaged across movies.Source: §5.1 · p. 7
5.2 Results on Character Naming
- A two-stage SFT+GRPO model with color-permutation augmentation gains +7.84% over the naive 8B baseline and outperforms all open-source alternatives, closing most of the gap to Gemini-3.1-Pro-Preview.The model uses a smaller 4B-Thinking backbone; Table 2 reports overall and known accuracy on the MCN test set.Source: §5.2 · p. 7, Table 2 · p. 7
- The model more accurately handles direct address, uses visual cues such as pointing and handshakes, and improves temporal grounding between subtitles and frames relative to open-source models such as Qwen3.5-9B.A remaining gap to the strongest proprietary model is likely driven by transcription and active-speaker detection errors; both systems also struggle with subtle cues, complex motion, and absent characters.Source: §5.2 · pp. 7–8
- On unseen-movie clips, method rankings remain unchanged: the fine-tuned 4B model leads open-source baselines and trails only proprietary Gemini models, while retaining most of its original accuracy.The clips were selected from films released after the evaluated models’ reported training cut-offs.Source: §5.2 · p. 8, Table 3 · p. 8
5.3 Results on Streaming Character Bank Construction
- The streaming pipeline substantially outperforms both proprietary baselines across mTIoU and PCR, despite processing each film in a single streaming pass while baselines receive full-movie access.The metrics assess character appearance and naming times, and alignment of the final bank with canonical identities.Source: §5.3 · p. 9, Table 4 · p. 9
6 Related Work
Prior movie-understanding work has used cast lists, placeholders, dialogue-based naming, and visual identity cues. This paper instead resolves names from the clip around each mention and builds a streaming character bank grounded in visual exemplars.
Character-Centric Movie Understanding
- Audio-description work often relies on cast lists [24–26, 43, 67, 68] or fan wikis, while other approaches avoid naming with placeholders; dialogue-based methods have also linked names to faces [40].The paper resolves identity from the single clip surrounding a name, targeting feature-length movies where names typically come from one-off dialogue rather than recurring TV casts.Source: §6 · p. 9
Visual Reasoning in Vision-Language Models (MLLMs)
Person Clustering in Long Videos
- Prior clustering methods use face, body, and voice cues, while this pipeline uses face embeddings as its primary signal and voice as fallback, updating and naming clusters as the film unfolds.Visual exemplars retain fine-grained identity cues that textual appearance descriptions can lose under similar appearances, costume changes, or lighting changes.Source: §6 · pp. 9–10
7 Discussion
The paper frames character naming and streaming character-bank construction as steps toward online movie understanding, while noting that neither task is fully solved. The appendices provide additional methods, error-correction details, and experiments.
7 Discussion
- Neither character naming nor streaming character-bank construction is fully solved, even by proprietary models, motivating work on the remaining failure modes.The paper proposes further models or an agentic framework as a next step.Source: §7 · p. 10
Appendix
- The appendices add qualitative results, ablations, implementation settings, dataset details, and discussion of broader impacts.They also describe the GRPO formulation, streaming error correction, and an additional baseline.Source: Appendix · p. 15
A Details on Preprocessing
- Preprocessing extracts and clusters face tracks, then links voice clusters to faces using active-speaker synchronization.The linking stage uses diarization alongside per-frame synchronization scores, with an override for high-confidence face-track evidence.Source: A Details on Preprocessing · p. 15, §A.2 · pp. 15–16
A.1 Face Track Extraction and Clustering
- Face tracks are detected across shots and clustered under temporal-overlap and nearest-neighbor-distance constraints.Tracks with partial temporal overlap cannot be merged, and merged tracks must fall below a distance threshold.Source: §A.1 · p. 15
A.2 Linking Voices to Face Tracks
- Voice clusters are linked to face clusters by aggregating active-speaker scores over subtitle spans, then assigning the highest-scoring match above a threshold.A high per-frame score can override the diarization-based grouping.Source: §A.2 · p. 16
B GRPO Reinforcement Learning Fine-Tuning
- GRPO fine-tuning combines format and name-identity accuracy rewards, normalizes rewards within response groups, and updates the policy with a clipped objective and KL regularization.The KL term controls regularization against a reference policy.Source: B GRPO Reinforcement Learning… · p. 16
C Error Correction for Character Bank Construction
The streaming pipeline uses complementary interventions to limit error propagation as the character bank evolves. They filter unreliable face tracks and off-screen names, then resolve conflicting name assignments across entries.
C Error Correction for Character Bank Construction
- Upstream noise from transcription, active-speaker detection, and identity grounding can propagate through the Markovian bank updates.The interventions target distinct stages of this streaming process.Source: C Error Correction for Character Bank… · p. 16
- An MLLM filters blurred or extreme-profile face tracks before clustering to preserve clean, identifiable clusters.The authors report that retained high-quality track embeddings are stable enough to make clustering errors negligible under their thresholds.Source: C Error Correction for Character Bank… · p. 16
- Qwen3.5-9B filters candidate names likely to refer to someone on screen, using surrounding dialogue because a single utterance may not resolve presence.This addresses models' tendency to assign an off-screen name to a visible face rather than abstain.Source: C Error Correction for Character Bank… · pp. 16–17
- A greedy vote resolves names assigned to multiple entries by retaining the highest-vote assignment and replacing the others with next-best candidates until convergence.The paper presents this as an inter-entry conflict-resolution step.Source: C Error Correction for Character Bank… · p. 17
- Together, the interventions address complementary failure modes and prevent error accumulation as the bank evolves, without access to past observations.Source: C Error Correction for Character Bank… · p. 17
D Implementation Details
Implementation uses specified thresholds and models for preprocessing, character-naming training, and streaming bank construction. The authors state that thresholds were not tuned on the test set and, without a validation split, followed prior work.
D Implementation Details
- Face clustering uses τcluster = 0.4, active-speaker association uses τASD = −0.075, and the per-frame override uses τoverride = 0.1.The diarization model is DiariZen-Large-s80-v2.Source: D Implementation Details · p. 17
- Character naming starts from Qwen3-VL-4B-Thinking, with one SFT epoch on synthetic traces followed by one GRPO epoch on the remaining training set.The vision encoder and multimodal projector remain frozen during SFT; GRPO samples G = 4 responses per prompt.Source: D Implementation Details · p. 17
- Streaming bank construction uses face and voice matching thresholds of τf = 0.55 and τv = 0.25, and samples frames at 2 FPS for the naming MLLM.Subtitle names are extracted with spaCy transformer NER, off-screen mentions are filtered by Qwen3.5-9B, and low-quality faces are filtered before bank entry.Source: D Implementation Details · p. 17
- PCR@K evaluation uses τeval = 0.55 for face matching and Double Metaphone for phonetic comparison with IMDB names.The IMDB reference embedding is an L2-normalized InsightFace embedding from the actor's headshot.Source: D Implementation Details · p. 17
- No threshold was tuned on the test set; without a validation split, thresholds followed prior work, with τeval set equal to τf.The authors say τcluster, τf, and τv follow Brown et al., while τASD and τoverride follow TalkNet.Source: D Implementation Details · p. 17
E Ablation Study
The ablations compare naming fine-tuning and augmentation, error-correction stages, and a weakly supervised baseline; the section also describes how MovieNet annotations are aligned to native-resolution frames. The reported comparisons distinguish naming performance from visual identity localization.
E Ablation Study
- 72.76% overall accuracy after RL for Qwen3-VL-4B-Thinking exceeds the 64.92% of direct SFT on Qwen3VL-8B-Instruct.SFT alone on the 4B Thinking model scores 64.16%.Source: E Ablation Study · p. 17, Table 5 · p. 18
- Color-permutation augmentation raises post-RL overall accuracy from 71.37% to 72.76% and known accuracy from 79.56% to 81.06%.The authors say the gains persist across both SFT and RL training stages.Source: E Ablation Study · p. 18, Table 5 · p. 18
- Removing face-track filtering lowers mTIoUv from 84.90 to 83.30 and mTIoUn from 54.40 to 52.27; removing query-name filtering lowers mTIoUn from 54.40 to 52.16 and PCR@10 from 54.54 to 49.46.The first stage filters noisy face features; the second removes off-screen candidate names that interfere with name binding.Source: Table 6 · p. 18, E Ablation Study · p. 18
- Removing inter-entry name disambiguation lowers mTIoUn from 54.40 to 52.40 while mTIoUv remains unchanged, with PCR@5 decreasing from 63.00 to 62.00.The authors characterize this stage as improving name-assignment consistency without changing localization of first occurrences.Source: Table 6 · p. 18, E Ablation Study · p. 18
F Additional Baseline for Streaming Character Bank Construction
- 33.38 mTIoUn and 34.0 PCR@5 for the weakly supervised baseline trail our pipeline's 54.40 mTIoUn and 63.0 PCR@5 despite full-movie access.The baseline's mTIoUv is comparable at 82.85 against 84.90, as both systems share face tracking and clustering.Source: Table 7 · p. 19, F Additional Baseline for Streaming… · p. 19
G Details on Dataset Collection
- MovieNet annotations are aligned to native-resolution frames using CLIP retrieval, LIS filtering, and perceptual-hash frame recovery.The resulting aligned, rescaled records serve as automatic ground truth for MCN.Source: G Details on Dataset Collection · pp. 19–20
H Dataset Examples
The MCN examples span films with varied genres, eras, and visual styles. They show that name resolution can depend on diverse visual and written cues, not dialogue alone.
H Dataset Examples
- Examples span genres, eras, and visual styles, reflecting varied cinematic conventions for introducing characters.The examples include period crowds, romantic comedy, political drama, and sports biopic settings.Source: H Dataset Examples · p. 20
- Direct address may require shot–reverse-shot, eyeline matching, or linking an off-screen telephone addressee to a face seen elsewhere.The visual realization of a spoken name varies even within direct address.Source: H Dataset Examples · p. 20
- Written names require OCR and visual grounding, so some examples need reasoning across spoken language, text, and images.Examples include names on tags, letters, signs, or screens; these cases cannot be solved from dialogue alone.Source: H Dataset Examples · p. 20
I Qualitative Results for Character Naming
Qualitative examples show the model grounding names through spoken cues and visual context. A comparison with a baseline and failure cases illustrate both its strengths and remaining challenges.
I Qualitative Results for Character Naming
- Examples cover direct address, third-party reference, and ambiguous spoken cues resolved using scene role and surrounding visual context.The model grounds each name to the correct colored box by combining spoken and visual evidence.Source: I Qualitative Results for Character… · p. 20
- The model gives a correct, concise identity mapping where a strong baseline’s longer, repeatedly verified reasoning reaches an incorrect one.The comparison illustrates that longer reasoning traces do not guarantee better grounding.Source: I Qualitative Results for Character… · p. 20
- Remaining errors often involve subtle actions, ambiguous speaker grounding, or noisy subtitles, while the model appears better at using fine-grained visual cues and dialogue context.Gemini-3.1-Pro-Preview also fails in examples involving subtle visual actions and interactions.Source: I Qualitative Results for Character… · p. 20
J Qualitative Results for Streaming Character Bank Construction
The streaming pipeline builds character banks for prominent characters in Juno and Meet the Parents, with errors remaining in difficult scenes and name bindings.
J Qualitative Results for Streaming Character Bank Construction
- In Juno and Meet the Parents, the pipeline builds entries for prominent characters, but complex multi-person scenes and transcription errors can produce incorrect or noisy bindings.The Juno example groups face tracks, predicts canonical names, and preserves aliases; the Meet the Parents example includes noisy variants such as Fonker and Falker for Focker.Source: J Qualitative Results for Streaming… · p. 20, Figure 10 · p. 27, Figure 11 · p. 28
K Experiment Compute Requirements
- Fine-tuning and streaming character-bank construction each use four 48-GB GPUs, while MCN test-set inference uses one 48-GB GPU.The specified GPU types are NVIDIA Ada 6000, A40, or L40S.Source: K Experiment Compute Requirements · p. 21
L Broader Impacts
- The authors identify accurate audio descriptions and speaker-attributed subtitles for accessibility as potential uses, and foresee no significant negative societal impacts.The stated method does not perform open-world face recognition or release an identity gallery from personal video sources.Source: L Broader Impacts · p. 21