PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Anonymous Captions: Grounding Character Identity in Video Captioning and Question Answering

Anas Filali Razzouki, Killian Steunou, Khalil Guetari, Thomas Kling, Mounîm El-Yacoubi, Yannis Tevissen

arXiv:2610.10163v1cs.CV

TL;DR

Identity-agnostic descriptions can capture actions without identifying which characters perform them, a distinction that matters for understanding movie narratives. This paper evaluates identity-linked spatial cues for video captioning and person-centric question answering, and introduces LoRA-fine-tuned BAC models. Face boxes combined with textual coordinates provide the most consistent performance across scales, and BAC-8B reaches 93.20% overall QA accuracy.

  • Problem

    Video-captioning models can describe people’s actions and interactions without determining which characters they are, although narrative understanding depends on attributing those actions and interactions to the correct characters.

    Source: §1 · p. 1

  • Method

    The framework provides known identities with explicit spatial cues across video and evaluates grounding strategies, then LoRA-fine-tunes Qwen models at three scales using identity-aware captions.

    Source: §1 · p. 2, §2.3 · p. 5

  • Results

    Face-based visual grounding combined with textual coordinates gives the most consistent performance across scales; BAC-8B reaches 93.20% overall QA accuracy.

    Source: §3.3 · p. 8, §4 · p. 9

  • Takeaways & Limitations

    Explicit visual grounding improves identity association over textual coordinates alone, and BAC applies this formulation across 2B, 4B, and 8B Qwen models.

    Source: §4 · p. 9

    The work studies identity-aware video understanding only in controlled movie settings with predefined cast identities and movie clips.

    Source: §5 · p. 10

Abstract

from arXiv · show

Linking people's appearance and actions to character identities is essential for understanding video narratives. We present a framework for identity-aware video captioning and person-centric question answering that combines automatic character identification, explicit spatial grounding, and task-specific adaptation. Starting from LSMDC v2 movie clips, our pipeline matches detected faces to actor reference images, tracks characters across frames, and builds inputs with identity-linked bounding boxes. A strong vision-language model generates identity-aware captions and questions, which are manually verified and filtered to create a benchmark of 750 captioned clips and 3,000 person-centric questions. We study five grounding strategies combining textual coordinates with visual face or estimated person boxes across Video-MLLM families at roughly 2B, 4B, and 8B parameters and larger frontier models. Combining visual face boxes with textual coordinates yields the most consistent performance across scales and significantly improves overall performance over coordinates alone. Smaller models tend to over-assign known identities when the queried person is not grounded, while larger models better recognize such UNIDENTIFIED cases. We introduce BAC by LoRA fine-tuning Qwen models at 2B, 4B, and 8B scales on about 32K identity-aware captioned clips. Across all scales, BAC outperforms every other evaluated model family of comparable size. BAC-8B reaches 93.20\% overall QA accuracy, ranking behind only GPT-5.6 Sol among the frontier models evaluated in our study. Overall, explicitly communicating who is where, together with lightweight task-specific adaptation, substantially improves identity-aware video understanding without changing the underlying architecture. We release the benchmark, training data, code, and BAC checkpoints at https://github.com/momentslab/beyond-anonymous-captions.

1 INTRODUCTION

The paper asks how known character identities should be spatially represented so general-purpose Video-MLLMs can reason about them across a video. It combines identity-linked spatial cues with video understanding and evaluates captioning and person-centric question answering.

1 INTRODUCTION

  • Existing post-processing methods add names after caption content is largely determined, so people omitted from the original caption cannot naturally be recovered (Pini et al., 2019; Tevissen et al., 2024).M-VAD Names specifically replaces SOMEONE tags in existing captions with character names.Source: §1 · p. 2
  • Other approaches learn identity consistency jointly with generation or ask the multimodal model to infer reference-to-person correspondence (Raajesh et al., 2024; Ji et al., 2025; Gao et al., 2026).The cited video method addresses reasoning about a target person across time and shot transitions (Gao et al., 2026).Source: §1 · p. 2
  • Region-grounding methods link language to visual regions, but the cited approaches are not designed to represent known character identities for general-purpose Video-MLLMs (Heo et al., 2025; Munasinghe et al., 2025).The methods use boxes or masks for region-specific reasoning, or spatio-temporally consistent segmentation masks.Source: §1 · p. 2
  • The framework supplies identity-linked spatial cues when reliable detections and tracks are available, letting general-purpose Video-MLLMs reason about known identities across video without a dedicated region-aware architecture.The study examines how models handle frames where cues are absent and must associate identities with appearances, actions, locations, and interactions.Source: §1 · p. 2
  • The study builds an LSMDC v2 pipeline, a manually verified benchmark of 750 captioned shots and 3,000 person-centric questions, and comparisons across five grounding representations, model scales, and families.It also introduces BAC at 2B, 4B, and 8B scales and releases approximately 32K synthetically captioned training clips, annotations, code, and checkpoints.Source: §1 · p. 2

2 METHODS

The methods build character-linked tracks and a manually verified evaluation benchmark, then compare identity-grounding representations and model families. They also test whether LoRA fine-tuning and explicit grounding improve identity-aware captioning and person-centric question answering.

2.1.1 CHARACTER IDENTIFICATION

  • Character identification combines cast information and actor reference images with face recognition and temporal tracking to match detected face tracks to movie characters.Track identity is accepted when matching confidence exceeds 0.30 and is propagated with face boxes; shots are split to handle abrupt scene or viewpoint changes.Source: §2.1.1 · p. 3

2.1.2 CAPTION AND QUESTION GENERATION

  • Eight frames are sampled per shot for caption and question generation, then manual review and correction produce a benchmark of 750 captioned shots and 3,000 person-centric questions.The generation stage retains valid, coherent, challenging shots; manual curation removes redundant or ambiguous examples and refines overly simple questions.Source: §2.1.2 · p. 3

2.2 IDENTITY-GROUNDING REPRESENTATIONS

  • Five representations compare textual coordinates, visual face or estimated person boxes, and combinations of visual boxes with textual coordinates for identity–location associations.The evaluated strategies include face coordinates alone, face boxes alone, person boxes alone, and both face-box-plus-coordinate and person-box-plus-coordinate combinations.Source: §2.2 · pp. 3–4

2.3 EXPERIMENTAL PROTOCOL

  • Experiments compare the five representations across Qwen3 scales, grounding across multiple Video-MLLM families, and BAC adaptation; QA prompts allow UNIDENTIFIED when no provided identity matches the queried person.Captioning prompts request character-linked identifiers, while QA is performed one person-centric question at a time.Source: §2.3 · p. 5
  • BAC fine-tunes Qwen models at 2B, 4B, and 8B using LoRA on the training split with ≈32K Gemini-generated identity-aware captions, then evaluates them on the manually verified evaluation set.This stage assesses whether task-specific, parameter-efficient adaptation can further improve identity-aware video understanding when combined with explicit identity grounding.Source: §2.3 · p. 5

2.4 EVALUATION METRICS

  • QA is scored by overall, named, unidentified, and clip-level accuracy, while caption evaluation uses a VLM judge for overall quality, identity correctness, and hallucination.The QA set contains 2,074 known-character questions and 926 UNIDENTIFIED cases; caption overall quality is scored on a 0–10 scale.Source: §2.4 · p. 5

3 RESULTS

Visual identity grounding improves identity-aware QA and captioning, especially for smaller models, while BAC fine-tuning strengthens Qwen3 across scales. BAC-8B outperforms several much larger models but remains below GPT-5.6 Sol.

3.1 EFFECT OF IDENTITY GROUNDING STRATEGIES

  • Visual grounding raises QA Overall over FTC alone from 53.87% to 68.90% at 2B, 62.57% to 79.73% at 4B, and 59.77% to 88.53% at 8B.FTC is significantly worse than every visually grounded variant at all model sizes (p < 0.001); among grounded variants, the differences shrink with scale.Source: §3.1 · pp. 5–6, Table 1 · p. 6
  • FVB+FTC provides the most consistent captioning and QA performance across scales, while captioning differences among visually grounded strategies shrink as model size increases.FTC remains significantly worse than every visually grounded variant for captioning at 4B and 8B (p < 0.001).Source: §3.1 · p. 6, Table 1 · p. 6
  • At 2B, PVB+PTC reaches 91.8% QA accuracy for small faces (< 4,000 px2), versus 80.0% for FVB+FTC; FVB+FTC leads for medium and large faces.For medium faces the scores are 92.8% vs. 91.3%, and for large faces 96.8% vs. 91.1%, respectively.Source: §3.1 · p. 6, Table 2 · p. 8
  • At 8B, FVB+FTC achieves nearly balanced Named and UNIDENTIFIED accuracy (88.04% and 86.93%), whereas combined visual–textual strategies favor Named cases at smaller scales.Visual-only grounding generally reduces that imbalance at the cost of lower Named accuracy.Source: §3.1 · p. 6

3.2 BENCHMARKING ACROSS MODEL FAMILIES

  • Among base models, Qwen3 has the highest QA accuracy within each size group and a statistically significant advantage over the strongest competing base model at 2B, 4B, and 8–9B.At 8–9B, Ovis2.5's higher caption identity accuracy than Qwen3 (93.73% vs. 91.60%) is not statistically significant (p = 0.089).Source: §3.2 · p. 8, Table 3 · p. 9
  • BAC raises Qwen3 QA accuracy from 68.90% to 80.33% at 2B, 79.73% to 90.13% at 4B, and 87.70% to 93.20% at 8B.All three improvements are statistically significant (p < 0.001); BAC also improves caption quality and identity accuracy at all three scales while maintaining a low hallucination rate.Source: §3.2 · p. 8
  • BAC-2B slightly exceeds the strongest 4B base model in QA (80.33% vs. 79.73%), while BAC-4B surpasses all evaluated 8–9B base models (90.13% vs. 87.70%).The relative improvement decreases as model size grows.Source: §3.2 · p. 8

3.3 COMPARISON WITH FRONTIER VLMS

  • BAC-8B reaches 93.20% overall QA accuracy, below GPT-5.6 Sol but above Qwen3-VL-235B-A22B, Gemini 2.5 Flash, and Claude Sonnet 4.6.Its Named and UNIDENTIFIED accuracies are 93.88% and 91.68%, respectively.Source: §3.3 · p. 8, Table 4 · p. 9

4 CONCLUSION

The paper combines automatic character identification and spatial grounding for identity-aware captioning and person-centric question answering, then adapts Qwen models through LoRA fine-tuning. Face boxes combined with textual coordinates perform most consistently across scales, and BAC-8B achieves 93.20% QA accuracy.

4 CONCLUSION

  • 93.20% QA accuracy is achieved by BAC-8B, which remains competitive with substantially larger frontier Video-MLLMs.Source: §4 · p. 9
  • Face-based visual grounding combined with textual coordinates provides the most consistent performance across scales and substantially improves identity association over textual coordinates alone.Source: §4 · p. 9
  • The framework combines automatic character identification with explicit spatial grounding for video captioning and person-centric question answering.The manually verified LSMDC v2 benchmark contains 750 captioned clips and 3,000 questions, and the study compares five grounding representations across model scales and families.Source: §4 · p. 9
  • BAC uses LoRA-based fine-tuning of Qwen models at 2B, 4B, and 8B scales.The approach communicates who is where without changing the underlying Video-MLLM architecture.Source: §4 · p. 9

5 ETHICAL CONSIDERATIONS

The work studies identity-aware video understanding only in controlled movie settings with predefined cast identities. It notes privacy risks from possible real-world adaptations and urges safeguards for future applications.

5 ETHICAL CONSIDERATIONS

  • The evaluation is limited to movie understanding with predefined cast identities and movie clips, not unconstrained real-world environments.Source: §5 · p. 10
  • Adapting face recognition, temporal tracking, and spatial grounding to real-world footage could raise privacy, surveillance, and unauthorized-profiling concerns.Source: §5 · p. 10
  • The authors frame the work as character-centric understanding and do not evaluate or advocate surveillance or monitoring applications.They encourage consent, privacy protections, and restrictions on biometric identity information in future applications.Source: §5 · p. 10

A ESTIMATING A PERSON BOUNDING BOX FROM A FACE DETECTION

The method estimates a full-person box from each detected face by scaling face dimensions and positioning the box around the face. It is a heuristic based on approximate human proportions, with fixed parameters across experiments.

A ESTIMATING A PERSON BOUNDING BOX FROM A FACE DETECTION

B EFFECT OF FACE AND PERSON SIZE

The relative effectiveness of face-box and person-box grounding depends partly on target face size. Person boxes help with very small faces, while face boxes can be more reliable for large, informative faces.

B EFFECT OF FACE AND PERSON SIZE

Model FVB+FTC PVB+PTC

The examples contrast face- and person-level grounding across model sizes: person-level boxes succeed on a small-face target, while face-level boxes succeed on a large-face action question.

Model FVB+FTC PVB+PTC

  • For the 2,222 px2 face, PVB+PTC identifies Adrianne Palicki at all three model sizes, while FVB+FTC predicts Lucas Black at 2B and 4B and UNIDENTIFIED at 8B.The target face is 52 px high.Source: Figure 3 · p. 13, Model FVB+FTC PVB+PTC · p. 13
  • The large-face comparison uses an action question about the person opening their eyes with a frightened expression, whose ground-truth identity is Joaquin.The target face is 760.5 px high.Source: Model FVB+FTC PVB+PTC · p. 14, Figure 4 · p. 14
  • For the 391,897 px2 face, FVB+FTC identifies Joaquin at all three sizes, while PVB+PTC predicts Abigail at 2B and identifies Joaquin at 4B and 8B.The example asks who opens their eyes with a frightened expression.Source: Figure 4 · p. 14, Model FVB+FTC PVB+PTC · p. 14

C BAC FINE-TUNING AND MODEL SCALING

BAC fine-tunes Qwen3-VL models at 2B, 4B, and 8B using LoRA, with the vision encoder and multimodal projector frozen. The selected configuration uses the lowest-validation-loss checkpoint at each scale.

C BAC FINE-TUNING AND MODEL SCALING

  • BAC-2B, BAC-4B, and BAC-8B adapt Qwen3-VL with LoRA on language-model projection layers, leaving the vision encoder and multimodal projector frozen.The models are fine-tuned on the training data described in Section 2.1.2.Source: C BAC FINE-TUNING AND MODEL SCALING · p. 15
  • The selected LoRA setting uses a 5 × 10−5 learning rate, rank 16, scaling factor 32, and dropout 0.05, adding 17.4M, 33.0M, and 43.6M trainable parameters at 2B, 4B, and 8B.The configuration came from a five-setting, one-epoch hyperparameter sweep.Source: C BAC FINE-TUNING AND MODEL SCALING · p. 15
  • Figure 5 reports training loss, evaluation loss, and learning-rate schedules for all three scales, marking the checkpoint with the best evaluation loss.Inference uses the lowest-validation-loss checkpoint for each variant.Source: Figure 5 · p. 15, C BAC FINE-TUNING AND MODEL SCALING · p. 15

D CATEGORY-WISE PERSON-CENTRIC QA PERFORMANCE

BAC-8B leads other 8–9B models in every person-centric QA category, while GPT-5.6 Sol thinking leads the frontier models. BAC-8B’s category performance varies, with position its weakest area.

D CATEGORY-WISE PERSON-CENTRIC QA PERFORMANCE

  • BAC-8B maintains strong performance across all question types, supporting the reported benefit of task-specific adaptation across identity-aware reasoning categories.Source: D CATEGORY-WISE PERSON-CENTRIC QA… · p. 16
  • BAC-8B has the highest accuracy in every category among 8–9B models and consistently outperforms Qwen3-VL 8B, while GPT-5.6 Sol thinking leads across frontier categories.Figure 6 orders categories by GPT-5.6 Sol thinking performance in the frontier comparison and by BAC-8B performance among 8–9B models.Source: D CATEGORY-WISE PERSON-CENTRIC QA… · p. 16, Figure 6 · p. 16
  • BAC-8B’s strongest absolute results are on appearance, object, and posture questions, and its lowest performance is on position questions.BAC-4B follows a similar category pattern, while BAC-2B varies substantially more across categories.Source: D CATEGORY-WISE PERSON-CENTRIC QA… · p. 16

E ADDITIONAL QUALITATIVE RESULTS

The additional examples show varied identity-question outcomes across BAC, Qwen3-VL, and frontier models, including both correctly identified and UNIDENTIFIED targets.

E ADDITIONAL QUALITATIVE RESULTS

  • In the hospital scene, every BAC and Qwen3-VL size answers both the posture and action questions correctly, while frontier-model answers include errors and UNIDENTIFIED responses.The questions ask who is lying in a hospital bed and who gathers her dress while walking past it.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 17
  • For the Paul Bettany appearance question, all BAC and Qwen3-VL sizes identify Paul; frontier models vary, with GPT-5.6 Sol thinking also answering correctly.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 18
  • On the Bridget Moynahan posture question, BAC-2B and BAC-8B answer correctly but BAC-4B does not; all three Qwen3-VL sizes are correct.Frontier-model responses vary on this example.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 18
  • For the vehicle interaction, all BAC sizes identify Charles S. Dutton, while Qwen3-VL 4B answers UNIDENTIFIED and its 2B and 8B variants answer correctly.Frontier responses range from correct to UNIDENTIFIED or incorrect.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 18
  • BAC and Qwen3-VL answer the Anjali Jay appearance and Adrianne Palicki posture questions correctly at all three sizes; frontier responses vary across the examples.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 19
  • On the Martin Lawrence interaction, all BAC and Qwen3-VL sizes identify Martin correctly, while frontier models include both correct and UNIDENTIFIED answers.The remaining examples include mixed BAC and base-model outcomes for Elliot Page, Jessica Lucas, and questions whose ground-truth answer is UNIDENTIFIED.Source: E ADDITIONAL QUALITATIVE RESULTS · pp. 19–20
Loading 2610.10163v1…