NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

Yuheng Huang; Jianlang Chen; Jiayang Song; Hua Qi; Aza Kai; Vincent Markert; Edison Marrese-Taylor; Jianjun Zhao; Lei Ma

Source-linked AI summary

NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

Yuheng Huang, Jianlang Chen, Jiayang Song, Hua Qi, Aza Kai, Vincent Markert, Edison Marrese-Taylor, Jianjun Zhao, Lei Ma

arXiv:2608.13210v1Aug 13, 2026cs.CVcs.AIcs.MM

TL;DR

Existing benchmarks rarely jointly test evolving narrative and culturally implicit meaning in high-context, non-English long-form video. NARU addresses this gap with a hierarchical annotation-to-QA benchmark for Japanese media, and evaluations reveal substantial limitations in both long-range narrative integration and culturally grounded reasoning.

Problem
Existing benchmarks rarely evaluate evolving narrative and culturally implicit social meaning jointly, particularly in high-context, non-English media.
Method
NARU uses a hierarchical memory-based annotation-to-QA pipeline with structured narrative and cultural annotations, iterative shortcut removal, and two-stage verification by 68 native Japanese annotators.
Results
Evaluations show substantial limitations in long-range narrative integration and culturally grounded reasoning, with current open-source models lagging significantly behind top commercial models across both areas.
Takeaways & Limitations
NARU provides a systematic testing ground for developing MLLMs that reliably interpret long-form, high-context video.

Abstract

from arXiv · show

Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimensions. To construct the benchmark at this scale, we propose a hierarchical memory-based annotation pipeline that transforms raw video into structured event, narrative, and cultural annotations, then generates questions via task-oriented synthesis and iterative shortcut removal. The construction process includes two native-speaker verification stages involving 68 annotators. Evaluations across eight model configurations reveal substantial limitations in both long-range narrative integration and culturally grounded reasoning. By exposing these persistent gaps, NARU offers a systematic testing ground for developing MLLMs capable of reliably interpreting long-form, high-context video.

I. Introduction

NARU targets context-rich Japanese long-form video, where models must integrate evolving narratives with culturally implicit social meaning. It introduces a large benchmark and hierarchical annotation pipeline to evaluate these capabilities at scale.

Context-rich long-form video requires tracking entities, events, causal dependencies, and character development while interpreting moments whose significance depends on accumulated social context.
Existing benchmarks evaluate long-range, narrative, or multilingual capabilities largely separately rather than jointly examining sustained narrative and culturally implicit understanding.
NARU evaluates Narrative Intelligence and Native Cultural Understanding in Japanese videos lasting 30 to 240 minutes within a high-context cultural framework.
A hierarchical annotation-to-QA pipeline decomposes videos, preserves cross-segment narrative continuity, and produces multilevel event, narrative, and social annotations for question synthesis.
1,481 multiple-choice questions ground evaluation in 155 Japanese long-form videos totaling 146.8 hours across extreme long-form settings.
68 native Japanese annotators verify the final benchmark items for video grounding and cultural fidelity, while evaluations establish baselines across narrative and culturally implicit reasoning dimensions.

II. Related Work

Related benchmarks broaden video evaluation across temporal coverage, narrative comprehension, social reasoning, and languages, but they leave a gap in jointly modeling evolving narrative and culturally implicit meaning.

Long-form benchmarks evaluate evidence retrieval, temporal grounding, and reasoning over extended videos, but temporal coverage alone does not establish evolving narrative or social-context interpretation.
Story-focused benchmarks address hierarchical character reasoning, story-centric temporal reasoning, entity persistence, distributed evidence, and long-range storyline comprehension.
Social and cultural benchmarks cover multimodal social reasoning, geographic knowledge, and multilingual video evaluation, yet rarely examine culturally implicit meanings across an extended narrative arc.

III. NARU Benchmark Construction

NARU construction addresses the cost and context-window limits of manual and automated annotation through hierarchical memory-based processing and a taxonomy spanning narrative and cultural interpretation.

III. NARU Benchmark Construction: Manual annotation of extremely long Japanese videos is costly and cognitively demanding, while automated MLLM pipelines can hallucinate or lose narrative consistency across hours of content.
III. NARU Benchmark Construction: A hierarchical memory-based pipeline combines short-term chunk processing with long-term narrative history to analyze and annotate multimodal information across extended timelines.
III. NARU Benchmark Construction: NARU separates narrative intelligence, which tracks evolving events, entities, and concepts, from cultural understanding of implicit socially situated meanings beyond literal audiovisual signals.
Capability Taxonomy: The narrative taxonomy covers entity evolution, sequential flow, plot or conflict progression, and thematic development, distinguishing local from global coherence.
Capability Taxonomy: The cultural taxonomy spans interactional signalling, shared situational understanding, speaker intent, cultural grounding, and affective or interpersonal dynamics.

B. Dataset Construction

Dataset construction begins with broad retrieval and Japanese-language filtering, then retains sufficiently long videos whose duration reflects meaningful temporal progression rather than redundancy.

Retrieval across 16 YouTube upload categories produced over 100,000 unique videos, and ASR-based language identification narrowed these to 51,643 Japanese-language videos.
Applying a minimum duration of 30 minutes yielded 8,018 candidate videos for content screening and diversity-aware selection.
Quality filtering required temporally ordered changes in characters, events, scenes, topics, or interpersonal dynamics to ensure long duration reflected meaningful progression.

C. MLLM-Driven QA Generation

NARU generates questions from hierarchical video annotations that preserve temporal, narrative, and cultural context, then iteratively removes text-only shortcuts through blind solving, diagnosis, and revision.

Annotation: The pipeline converts long-form video into chunk- and segment-level evidence before adding task-oriented narrative and cultural annotations.Chunking uses approximately five-minute processing units, while semantic segmentation identifies coherent chapter-level units independent of those boundaries.
Annotation: Narrative annotations organize characters and events into coherent threads with consequential events and causal progression for four narrative categories.
Question Generation: Candidate four-option questions pair each capability category with supporting evidence, relevant entities, events, segments, and audio cues.Distractors are constrained to be plausible, incorrect with respect to the evidence, and comparable in granularity, polarity, length, and syntax.
Debiasing: The Solver-Critic Loop tests questions without video, diagnoses shortcuts such as tone bias or elimination, and revises vulnerable stems, answers, or distractors.Iteration continues until blind-solving success approaches random chance or reaches a predefined iteration budget.

D. Quality Control

NARU applies two native-speaker verification stages to check answerability, correctness, cultural faithfulness, and distractor quality, producing a final benchmark of 1,481 items.

Two-stage validation: 68 native Japanese annotators verified NARU questions in two stages after generation and shortcut-triggered refinement.The first stage used 40 annotators and the second used 28.
Initial QA verification: 1,500 candidate questions yielded 1,483 items after 178 invalid-question flags, including 161 repairs and 17 removals.Initial verification addressed ambiguous questions, overlapping options, missing correct answers, and incorrect labels.
Post-refinement verification: The second review accepted 949 refined items unchanged, revised 532, and removed two before finalization.Revisions included 436 answer corrections, 108 distractor edits, and 35 question rewrites.
Final benchmark: The final benchmark contains 1,481 items after annotators checked validity, answer correctness, specificity, and distractor plausibility.The second-stage review compared refined items with their originals and source videos.

IV. Evaluation

The evaluation compares open- and closed-source MLLMs on NARU through full-benchmark multiple-choice testing, with analysis across nine capability categories and temporal-capacity conditions.

Evaluation design: NARU evaluation uses full-benchmark multiple-choice testing to compare open-source and closed-source MLLMs.The setup is intended to characterize both overall performance and conditions where long-form, culturally grounded understanding succeeds or breaks down.
Capability analysis: The comparison exposes performance variation across NARU’s nine defined capability categories.The evaluation begins with a controlled comparison across models.
Temporal capacity: The evaluation systematically considers temporal capacity because long-video understanding depends heavily on models’ ability to process extended video.The supplied passage introduces temporal-capacity analysis but does not specify its full procedure.

A. Experimental Setup

Experiments evaluate eight proprietary and open-source MLLMs under long-video inference settings, revealing a strong capability gap between model groups and between narrative and cultural reasoning.

Models: Eight MLLMs are evaluated: three proprietary Gemini models and five open-source models with long-video understanding capabilities.The proprietary systems are Gemini-3-Flash, Gemini-3-Pro, and Gemini-2.5-Flash; open-source systems include Qwen, MiniCPM, and InternVL variants.
Inference setting: For MCQ evaluation, every model answers all four-choice NARU questions using each model family’s recommended input configuration.Gemini videos are uniformly sampled at fps=0.25, while open-source models use maximum uniform sparse-frame sampling.
Overall results: 76.2% accuracy makes Gemini-3-Flash the strongest full-benchmark model, ahead of Gemini-3-Pro at 70.0% and Gemini-2.5-Flash at 51.4%.Open-source models occupy a lower 29.6–39.8% accuracy regime, led by Qwen3.5-9B.
Narrative reasoning: Approximately 11 percentage points separate Gemini models’ narrative and cultural accuracy, while open-source models show virtually no difference between the dimensions.Tracking sequential structure (N.2) is universally the easiest narrative task.
Cultural reasoning: 57.4% is Gemini-3-Flash’s accuracy on C.3 Subtext Interpretation despite a 68.2% cultural average, making C.3 exceptionally difficult even for frontier models.Open-source models perform best on C.2 Kuuki wo Yomu, while pragmatic reasoning remains challenging across systems.

C. Effect of Temporal Evidence

Increasing sampled frames generally improves long-form video accuracy, especially for narrative understanding, but gains vary substantially across model families and task domains.

Experimental setup: Controlled evaluation varies sampled frames from 8 to 128 on the same 500-question subset to test whether broader temporal coverage helps recover distributed evidence.The sweep uses frame budgets f ∈ {8, 16, 32, 64, 128}.
Overall accuracy: All three Gemini models improve substantially with more frames, while open-weight models improve less consistently.Gemini-3-Pro rises from approximately 64% to 71%, Gemini-3-Flash from 53% to 64%, and Gemini-2.5-Flash from 39% to 51%.
Overall accuracy: Qwen3VL-8B shows the largest open-weight gain, increasing from about 31% to 38%, while Qwen2.5VL-7B, MiniCPMo-2.6, and InternVL3.5 remain near 30%.These results indicate that supporting larger frame budgets does not necessarily yield comparable sequence-level evidence integration.
Narrative versus cultural accuracy: Narrative accuracy gains range from 1.4 to 20.5 percentage points, whereas cultural accuracy ranges from a 3.9-point decline to a 10.0-point gain.For every model, increasing the frame budget benefits narrative performance more than cultural performance.
Model scaling dynamics: Gemini-3-Pro leads Gemini-3-Flash across controlled frame budgets, but its margin shrinks from 11 points at 8 frames to 7 points at 128 frames.In the full-benchmark 0.25 FPS setting, the ordering reverses: Flash reaches 76.2% versus Pro at 70.0%.

D. Open-Ended Evaluation

The open-ended evaluation removes answer choices and measures direct generation using atomic-fact recall, revealing capability patterns that differ from multiple-choice results. Models retain their performance hierarchy, but sequential narrative reconstruction becomes especially difficult without structural scaffolding, while cultural responses more often recover partial facts.

Evaluation Setup: Open-ended evaluation converts a 500-question diagnostic subset into free-form questions and scores responses by atomic-fact recall rather than lexical overlap or exact match.GPT-5.5 decomposes reference answers into atomic facts and measures the fraction covered by each response.
Evaluation Setup: Human annotators accept the automated judge’s scores in 90.0% of verdicts, while majority voting accepts 48 of 50 judgments (96.0%).Three annotators independently assessed 50 judged responses, producing 150 verdicts.
Results: 0.66–0.78 atomic-fact recall for Gemini versus 0.21–0.56 for open-source models preserves the multiple-choice performance hierarchy.The result holds after removing answer choices from the evaluation.
Results: N.2 becomes the weakest narrative category for seven of eight models in open-ended generation, despite being the strongest narrative subcategory in multiple-choice evaluation.Candidate options may scaffold temporal organization, whereas open-ended responses require independent retrieval, synthesis, and chronological reconstruction.
Results: 78.9% of cultural responses recover at least one reference fact, compared with 66.3% of narrative responses, reversing the relative difficulty seen in multiple-choice evaluation.Cultural prompts often provide interactional context, while narrative questions more often require reconstructing distributed events without prompting.

V. Conclusion

NARU combines a large Japanese long-form video benchmark with hierarchical annotation, shortcut removal, and native-speaker verification. Evaluations across model configurations show substantial variation in both performance and behavior.

V. Conclusion: NARU contains 1,481 questions from 155 Japanese long-form videos totaling 146.8 hours across four narrative and five cultural dimensions.The benchmark is constructed for long-form narrative and cultural understanding.
V. Conclusion: The benchmark uses a hierarchical annotation-to-QA pipeline, iterative shortcut removal, and two-stage verification by 68 native Japanese annotators.These procedures support question construction at the benchmark’s temporal scale.
V. Conclusion: Evaluation across model configurations shows substantial variation in both performance and behavior.The conclusion reports this pattern without reducing it to a single metric or model.