PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG

Dong-Hee Kim, Seonwoo Choi, Changbeen Kim, Jungmyung Wi, Juyeon Ko, Youngju Choi, Il Hyeon Mun, Hyunwoo J. Kim, Donghyun Kim

arXiv:2608.28699v1cs.CV

TL;DR

Long-form movie understanding needs retrieval units that preserve narrative evidence, but existing scene segmentation methods do not consistently outperform uniform temporal chunking. The paper audits this mismatch, introduces the narrative-centric NarraScene dataset, and finds that narrative-grounded segments improve downstream movie RAG, while the hand-labeled evaluation remains an oracle upper-bound analysis for future automatic methods.

  • Problem

    Existing scene segmentation methods have not been established as effective retrieval units for movie RAG, which requires story-level reasoning over characters, events, and narrative arcs spanning hours of content.

  • Method

    The paper evaluates segmentation methods across movie-understanding tasks, audits annotation targets, and introduces NarraScene with physical, character, and narrative change levels requiring a narrative shift at every valid boundary.

  • Results

    Existing methods consistently fail to outperform uniform temporal splitting, while hand-labeled narrative scenes consistently outperform uniform chunking on downstream claim verification.

  • Takeaways & Limitations

    Useful movie-RAG scene boundaries should recover narrative evidence units rather than merely detect visually salient transitions, motivating narrative-grounded segmentation benchmarks.

  • Takeaways & Limitations

    The hand-labeled narrative-scene result is an oracle analysis and practical upper bound, not evidence that current automatic segmentation methods already solve the problem.

Abstract

from arXiv · show

Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs). Sparse frame sampling fails to capture fine-grained visual details, while dense sampling quickly exceeds context length limits. Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval units. In this paper, we investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content. Scene segmentation, a long-studied problem that partitions movies into semantically coherent units, is a natural candidate for defining such retrieval units. We reexamine whether existing methods actually serve this role through comprehensive evaluation on downstream movie understanding tasks, and find that they consistently fail to outperform naive uniform temporal chunking. Our audit of the most standard scene segmentation benchmarks reveals why: current annotations prioritize visually salient transitions over narrative event structure. Motivated by this mismatch, we introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change, where every valid boundary requires a narrative-level shift. When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks, suggesting that the central challenge for scene segmentation in movie RAG is not detecting boundaries, but identifying the narrative event units that matter for movie understanding.

1 Introduction

Long-form movie understanding requires retrieval units that preserve story-level evidence, but existing scene segmentation boundaries do not reliably serve this role. The paper therefore examines narrative-centric segmentation for movie RAG and introduces NarraScene as a benchmark targeting narrative event changes.

  • Sparse frame sampling loses fine-grained visual and temporal details, while longer visual-token sequences remain difficult for MLLMs to attend to and reason over.
  • RAG retrieves targeted video segments for downstream reasoning, making segmentation boundaries foundational because fragmented or semantically mismatched chunks can exclude needed evidence.
  • Movie understanding demands story-level reasoning over characters, events, and narrative arcs, yet it remains unclear whether MovieNet-based boundaries align with retrieval units required for movie RAG.
  • Across temporal grounding, multiple-choice question answering, and claim verification, existing scene segmentation methods consistently fail to outperform naive temporal uniform splitting.
  • NarraScene defines boundaries through physical, character, and narrative change, requiring every valid boundary to include a narrative shift.
  • NarraScene contains 2,371 narrative boundaries and 2,424 annotated scenes from 53 full-length movies, and hand-labeled narrative scenes consistently outperform uniform chunking on MF2 claim verification.

2 When Scene Segmentation Meets Modern Movie RAG

The paper tests whether scene segmentation provides better retrieval units than uniform temporal chunks in movie RAG. Across temporal grounding, question answering, and fact verification, existing methods show no consistent advantage, motivating evaluation focused on narrative coherence.

  • Experimental Setup: The evaluation covers temporal grounding, movie question answering, and claim verification, each requiring evidence selection from full-length movies.The tasks use MAD, MovieStory101, and MF2, respectively.
  • Experimental Setup: Movie RAG evaluates segmentation methods by partitioning full-length movies, retrieving top-K query-relevant chunks, and supplying them as sole visual context to fixed downstream models.The pipeline isolates segmentation quality by fixing retrieval and downstream models across conditions.
  • Downstream Evaluation: Uniform temporal splitting remains a strong baseline, while legacy scene segmentation methods consistently underperform on temporal grounding.This pattern is reported across the TimeLens and Vidi grounding models.
  • Downstream Evaluation: On movie question answering, retrieved scene units are compared across multiple downstream models while varying the segmentation method.The benchmark uses Qwen3-4B, Qwen3-8B, InternVL3.5 8B, and InternVL3.5 14B.
  • Downstream Evaluation: For fact verification, retrieval is preferable to whole-movie input, but scene-based chunking is not consistently superior to uniform splitting.The comparison also distinguishes chunk-length effects from semantic boundary alignment.
  • Downstream Evaluation: Existing trained and classical scene segmentation methods fail to show a consistent advantage over uniform splitting, while oracle three-minute clips substantially outperform retrieved top-1 chunks.Manual examination found that all 63 sampled oracle clips contained coherent narrative scenes, whereas predicted segments often lacked narrative coherence.

3 Why Existing Scene Segmentation Falls Short for Movie RAG

Existing scene segmentation does not provide effective movie-RAG retrieval units because its benchmark boundaries emphasize visual transitions over narrative event changes. The paper therefore defines valid boundaries around event structure, requiring a narrative-level shift.

  • Across temporal grounding, question answering, and fact verification, existing segmentation methods often fail to outperform simple temporal fixed-length splitting.
  • The paper distinguishes meaningful narrative event changes from visually salient surface transitions when defining scene boundaries.
  • Event Segmentation Theory motivates tracking changes in time, place, participants, causes, and goals rather than treating cuts alone as event boundaries.
  • The taxonomy includes physical change, character or entity change, and shifts in the core goal, interaction, or conversational focus.
  • Every valid narrative scene boundary must include a Level 3 transition, while Level 1 and Level 2 changes may co-occur or be absent.
  • The audited MovieNet-SSeg subset is dominated by Level 1 physical changes, with only a small fraction of boundaries satisfying Level 3 narrative change.
  • MovieNet-SSeg remains valid for its original task, but its visually emphasized target differs from the narrative boundaries needed for movie RAG.

4 NarraScene: Narrative-Centric Scene Segmentation Dataset

NarraScene operationalizes narrative-centric scene segmentation for full-length movies through an EST-grounded taxonomy and multi-stage annotation. As an oracle retrieval target, its hand-labeled boundaries improve movie claim verification over fixed-length and automatic alternatives.

  • NarraScene applies the EST-grounded definition in which valid boundaries must include Level 3 narrative change.
  • The dataset covers 53 full-length movies averaging 88.3 minutes and contains 2,371 narrative scenes.
  • Annotation uses independent boundary proposal, structured cross-verification, and final consensus adjudication.
  • Most verification decisions confirm or relabel boundaries, while only a small minority delete, move, or insert them; every accepted boundary retains a Level 3 shift.
  • The claim-verification evaluation compares fixed-length uniform chunks with semantic oracle boundaries under the same retrieval pipeline using pairwise accuracy.
  • NarraScene achieves the best pairwise accuracy in all eight model–retrieval settings and consistently outperforms fixed-length baselines and automatic GenreDur segmentation.
  • These gains are an oracle upper bound, not evidence that current automatic scene segmentation methods already recover narrative boundaries.

5 Additional Discussion and Analyses

The additional analyses show how narrative coherence affects retrieval evidence and separate retrieval errors from chunk-construction quality. Qualitative and controlled retrieval results explain why uniform chunking can be competitive while still missing the best semantic units.

  • Qualitative Analysis: Fragmenting a narratively coherent exchange can leave the retriever with partial evidence, causing the downstream verifier to miss a key statement and answer incorrectly.
  • Qualitative Analysis: Uniform windows may overlap relevant exchanges and remain competitive, but narrative scenes preserve the full conversational event with its motivation and supporting quote.
  • Qualitative Analysis: The qualitative case identifies a transfer gap: the decisive evidence is a goal-relevant conversational reveal rather than a visually salient transition.
  • Retrieval Quality: The retrieval-quality study replaces actual top-1 chunks with chunks having the largest overlap with annotated three-minute evidence clips.
  • Retrieval Quality: 41.6 to 60.9: best-overlap replacement improves QA accuracy for uniform chunking, while 36.2 to 50.4 improves accuracy for GenreDur.
  • Retrieval Quality: The higher best-overlap accuracy for uniform chunking indicates that chunk quality remains important even under favorable retrieval.
  • Discussion: Movie-RAG evaluation should consider both retrieval quality and chunk quality rather than final QA accuracy alone.

6 Conclusion

The paper concludes that legacy scene segmentation targets are poorly matched to movie-RAG narrative reasoning, while narrative-centric boundaries provide a more relevant retrieval-unit target. NarraScene supports this conclusion through EST-grounded annotations and downstream evaluation.

  • Existing scene segmentation methods do not consistently outperform uniform splitting across downstream movie understanding tasks.
  • The reported gap is associated with annotations aligned more closely to visually salient physical transitions than to narrative event change.
  • NarraScene defines boundaries through physical, character, and narrative change while requiring every valid boundary to include a narrative-level shift.
  • The paper frames cognitively grounded narrative boundaries as a stronger substrate for retrieval, localization, and reasoning than fixed temporal chunks or visually defined segments.

B Detailed Downstream Task Experimental Setup

The experiments hold the retrieval backbone and core preprocessing choices fixed across downstream tasks, so segmentation methods are compared under a common protocol.

  • The same scene candidates produced by each segmentation method are reused as the retrieval pool.
  • Retrieval uses Qwen3-VL-Embedding-2B with fixed 1 FPS preprocessing, ℓ2-normalized embeddings, and dot-product similarity.
  • Compared downstream models receive identical retrieval outputs within each task, isolating segmentation-method differences.

B.1 MovieStory101: Movie Question Answering

MovieStory101 uses retrieved scene clips as evidence for multiple-choice answering, while the MAD setup retrieves scenes before independent temporal localization.

  • MovieStory101: Movie Question Answering: MovieStory101 questions are answered with a two-stage retrieve-then-answer protocol using scene clips as evidence.Each candidate scene is encoded and ranked against the question before answering.
  • MovieStory101: Movie Question Answering: Scenes are ranked by dot product between ℓ2-normalized text and video embeddings, and the top 5 are retained for independent QA-model evaluation.The QA model receives each retrieved scene separately and produces a bracketed option answer.
  • MovieStory101: Movie Question Answering: Top-K accuracy measures whether at least one of the top-K retrieved scenes yields the correct option, for K ∈{1, 3, 5}.Evaluations are run with both 4B and 8B QA models on the same retrieval JSON.
  • MAD: Movie temporal grounding: MAD retrieves the top 5 scenes for each caption, then independently localizes the described event within each candidate clip.TimeLens-8B predicts absolute seconds, whereas Vidi-7B predicts normalized percentages remapped to seconds.

B.3 MF2: Movie Claim Verification

MF2 evaluates scene retrieval and claim verification using paired true and false statements, transcript-augmented visual inputs, and pairwise-consistent scoring.

  • MF2: Movie Claim Verification: Each MF2 instance contains a true claim and a false claim about one movie, retrieved independently as text queries.
  • MF2: Movie Claim Verification: The final retrieval score for each scene is the maximum similarity to the independently encoded true and false claims.ASR transcripts are appended to visual inputs, and the same ASR pipeline is used for every segmentation method.
  • MF2: Movie Claim Verification: The verifier evaluates each claim independently and uses an OR ensemble over the top-K retrieved scenes.A claim receives a TRUE or FALSE verdict if at least one retrieved scene produces that verdict.
  • MF2: Movie Claim Verification: If video decoding fails, the MF2 verifier falls back to a transcript-only prompt while preserving the same answer format.
  • MF2: Movie Claim Verification: Pairwise accuracy is counted only when one specific retrieved scene supports TRUE for the true statement and FALSE for the false statement.Complementary verdicts across different scenes are not combined.
  • MF2: Movie Claim Verification: The MF2 verifier runs independently for each retrieved scene and combines top-K candidates with an OR ensemble.The prompt uses the retrieved scene’s ASR transcript and the claim under evaluation.

C Film-wise MovieNet Audit Details

The MovieNet audit reports film-wise cognitive labels, while NarraScene annotation uses repeated review and consensus to establish boundary locations and level labels.

  • Film-wise MovieNet Audit Details: The MovieNet audit re-annotated ten stratified MovieNet-318 test films and reported percentages with raw boundary counts for each cognitive label.
  • Film-wise MovieNet Audit Details: Across films, the audit found a strong concentration of Level 1 boundaries and relatively low Level 3 coverage.The aggregate pattern was not driven by a single outlier title.
  • NarraScene Annotation Pipeline: NarraScene’s first stage has one annotator watch the full movie, propose timestamps, assign narrative levels, and record memos for difficult cases.Full-movie viewing preserves global story context such as causal continuity, character goals, and long-range callbacks.
  • NarraScene Annotation Pipeline: A second annotator cross-checks proposed boundaries and may confirm, relabel, delete, move, or insert them.Ambiguous cases are marked hard for later consensus review.
  • NarraScene Annotation Pipeline: All disagreements are resolved by four-way consensus after jointly re-watching disputed segments and reconciling narrative rationales.The final decision covers both boundary location and level label.
  • NarraScene Annotation Pipeline: The annotation pipeline records detailed boundary-label statistics for the MF2-aligned portion of NarraScene.

E Additional NarraScene Statistics

The appendix details NarraScene’s MF2-aligned statistics and shows that its annotations target semantic boundaries. On NarraScene ground truth, uniform splitting outperforms MovieNet-trained segmentation methods, while qualitative examples show chunking errors affect downstream predictions.

  • Dataset statistics: The MF2-aligned statistics cover movie collection scale, duration, annotated boundaries, resulting scenes, and scene-duration distributions.Figure E visualizes scene durations and boundary-label combinations.
  • Boundary labels: 100% of annotated boundaries include Level 3 semantics, confirming that semantic scene transitions dominate this subset.
  • Segmentation performance: 49.3 mIoU is achieved by the 2-minute heuristic, outperforming PySceneDetect, BaSSL, and GenreDur on NarraScene ground truth.The result indicates that MovieNet-oriented methods do not transfer well to NarraScene’s event-centric structure.
  • Downstream effects: Qualitative examples show that retrieval-chunk errors propagate directly to answer prediction or temporal localization.

H Limitations and Social Impacts

NarraScene’s statistics indicate broad variation in narrative unit lengths and dominant semantic boundary labels, while the authors caution that the benchmark does not represent all narrative cinema. Qualitative examples illustrate that chunking choices affect retrieval quality and downstream predictions.

  • Limitations and social impact: NarraScene does not capture the full diversity of narrative cinema across genre, editing conventions, and production context.The authors frame it as a research benchmark rather than a definitive model of narrative structure.
  • Dataset scope: NarraScene spans a broad range of scene durations, with a mean of 1.97 minutes and a median of 1.65 minutes.
  • Dataset scope: Level 3 semantics dominate the boundary-label combinations, emphasizing semantic transitions beyond low-level visual discontinuities.
  • Qualitative evidence: Uniform chunking retrieves better-matched evidence than scene-based baselines in a MovieStory101 question-answering example.Scene-based distractor clips lead to incorrect answers or failure.
  • Qualitative evidence: Uniform chunking more tightly covers the target moment than scene-based baselines in a MAD temporal-grounding example.The scene-based candidates produce weaker temporal localization.
Loading 2608.28699v1…