PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

Junfu Pu, Yuxin Chen, Teng Wang, Ying Shan

arXiv:2604.11102v1cs.CVcs.MM

TL;DR

Long-form cinematic videos remain difficult for MLLMs to convert into detailed, temporally grounded scripts. The paper introduces a benchmark, hierarchical evaluation framework, and 8B-parameter audio-visual OmniScript model trained with progressive reasoning and temporally segmented reinforcement learning. OmniScript outperforms larger open-source models and remains competitive with proprietary systems on event understanding and temporal grounding.

  • Problem

    Long-form cinematic video-to-script generation lacks suitable data and robust evaluation while requiring detailed temporal and multimodal narrative understanding.

  • Method

    The paper builds a human-annotated benchmark and hierarchical evaluation framework, then trains OmniScript through progressive chain-of-thought supervision and GRPO with temporally segmented rewards.

  • Results

    With 8B parameters, OmniScript consistently outperforms much larger open-source models on event-level understanding and temporal grounding while remaining competitive with proprietary models.

  • Takeaways & Limitations

    OmniScript provides a benchmark, baseline, and training insights for audio-visual narrative video understanding.

  • Takeaways & Limitations

    OmniScript can process videos whose duration is less than 5 minutes.

Abstract

from arXiv · show

Current multimodal large language models (MLLMs) have demonstrated remarkable capabilities in short-form video understanding, yet translating long-form cinematic videos into detailed, temporally grounded scripts remains a significant challenge. This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To facilitate this, we construct a first-of-its-kind human-annotated benchmark and propose a temporally-aware hierarchical evaluation framework. Furthermore, we present OmniScript, an 8B-parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension. OmniScript is trained via a progressive pipeline that leverages chain-of-thought supervised fine-tuning for plot and character reasoning, followed by reinforcement learning using temporally segmented rewards. Extensive experiments demonstrate that despite its parameter efficiency, OmniScript significantly outperforms larger open-source models and achieves performance comparable to state-of-the-art proprietary models, including Gemini 3-Pro, in both temporal localization and multi-field semantic accuracy.

1 Introduction

The paper frames long-form cinematic video-to-script generation as an unexplored task requiring detailed temporal and multimodal narrative understanding. It addresses data, evaluation, and computational challenges with a benchmark, evaluation framework, and audio-visual model.

  • Research Gap: Full-length movie or episode transcription into a detailed, structured script remains largely unexplored beyond short-form MLLM video tasks.The target output requires scene-by-scene descriptions of settings, actions, dialogue, emotions, and audio-related content.
  • Challenges: Long-form script generation is difficult because annotations are labor-intensive, outputs are open-ended and temporally structured, and robust evaluation metrics remain unresolved.The challenge includes parsing multi-scene structures and character interactions while measuring accuracy, coherence, and completeness.
  • Challenges: A two-minute clip requires approximately 4,000 tokens, making inference cost and time a practical bottleneck as video duration increases.The paper reports that token requirements can explode for longer videos.
  • Contributions: The paper contributes a diverse movie and TV understanding dataset, a human-annotated V2S benchmark with dedicated metrics, and an omni-modal script-generation model.The model uses audio-visual pre-training, chain-of-thought supervised fine-tuning, and GRPO-based reinforcement learning with open-ended script-quality rewards.
  • Results: OmniScript achieves performance comparable to state-of-the-art closed-source models, including Gemini 3-Pro, on long-form video script generation tasks.The reported experiments characterize the approach as superior while emphasizing comparability with proprietary systems.

2 Related Work

Prior work separately addresses plot reasoning, temporal event localization, and audio-visual integration, but often misses fine-grained, coherent, temporally grounded scripts. OmniScript combines these requirements through a decoupled hierarchical representation.

  • Prior Cinematic Understanding: Earlier cinematic understanding progressed from short-clip descriptions to plot reasoning and structural annotations, but macro-plot and structural approaches leave complementary gaps.Macro-plot analyses lack fine-grained temporal grounding, whereas structural annotations do not necessarily produce readable, coherent narratives.
  • Prior Cinematic Understanding: Existing role-aware narration methods predominantly use isolated, dialog-free clips and therefore do not model full-length film intricacies.The passage identifies Movie101 as an example of this limitation.
  • Holistic Script Generation: Holistic Script Generation transcribes full-length videos into temporally anchored, hierarchically structured scripts with decoupled actions, expressions, dialogues, and audio cues.This annotation format is contrasted with coarse paragraph summaries that entangle multimodal cues.
  • Dense and Omni-Modal Video Captioning: Dense captioning localizes salient events but produces sparse summaries, while audio-visual captioning integrates modalities without explicit temporal grounding.OmniScript addresses both limitations using a Scene →Event →Field hierarchy with fine-grained timestamped elements.
  • Dense and Omni-Modal Video Captioning: Unlike TimeChat-Captioner’s coarse scene-level summaries, OmniScript explicitly isolates character actions, intents, dialogues, and other atomic fields at dense timestamps.The stated design goal is to reduce ambiguity in prior captioning paradigms.

3 Video-to-Script Generation

The V2S task maps untrimmed cinematic videos to temporally grounded hierarchical scripts, while a memory-augmented annotation pipeline preserves character and narrative consistency. The output combines metadata, scenes, ordered events, and multimodal fields.

  • Problem Definition: Video-to-script generation transcribes long-form movie or TV video into a temporally grounded, hierarchically structured output containing metadata and scene-event scripts.The formal target is ˆY = {ˆM, ˆS}, where ˆM is video-level metadata and ˆS contains scene-event scripts.
  • Output Schema: The output schema has meta-, scene-, and event-level structure, covering global attributes, scene context, timestamps, character identity, and multimodal event content.Event fields include dialogue, action, expression, and audio cues such as sound events or background music.
  • Output Schema: Each event is represented with a timestamp, character or environment identity, and dialogue, action, expression, or audio-cue fields.The notation defines c as character or Environment and d/a/x/u as dialogue/action/expression/audio cue.
  • Objective: The learning objective jointly targets temporal localization, character-consistent semantic parsing, and multimodal narrative faithfulness.These objectives correspond to when events happen, who does or says what, and how events are expressed through language, behavior, expression, and sound.
  • Annotation Pipeline: The annotation pipeline uses historical character profiles to guide plot reasoning, updates memory across segments, and feeds plot descriptions with raw audio-visual content into a script annotator.The process partitions videos into semantically cohesive segments, uses Gemini-based reasoning and annotation, and constructs Video →Thinking →Script data for model training.

4 The V2S Benchmark: Metrics and Dataset

The V2S benchmark addresses the difficulty of evaluating hierarchical, temporally grounded scripts by combining semantic event alignment, character resolution, field-level scoring, and temporal-boundary evaluation. It is built around 10 full-length cinematic works with dense scene and event annotations.

  • Evaluation Framework: The evaluation framework separates semantic fidelity from temporal localization through four sequential stages.Stages cover text-based event matching, character mapping, field-level semantic evaluation, and temporal-boundary evaluation.
  • Event Alignment: Composite semantic similarity and dynamic programming align predicted and ground-truth events while tolerating minor temporal noise.Candidate matches allow one-to-many and many-to-one mappings, use dialogue and action similarity, impose a 30.0-second proximity constraint, and maximize globally ordered assignments.
  • Character Resolution: LLM-assisted character mapping resolves open-vocabulary identities before event evaluation to prevent cascading mismatch errors.The process categorizes names, creates an initial bipartite graph, then applies a fallback score based on temporal overlap and lexical similarity under type constraints.
  • Field and Temporal Evaluation: Aligned event groups are evaluated across character, action, dialogue, expression, and audio fields, while temporal scoring penalizes misses and hallucinations.Unaligned ground-truth groups degrade recall, and unaligned predicted groups degrade precision; temporal quality is evaluated independently with tIoU Hit Rate.
  • Dataset: 10 full-length cinematic works totaling 19.9 hours form a benchmark spanning anime, action, suspense, and drama.The benchmark contains 1.4k scenes and over 16.8k structural events, averaging 14.1 events per minute.

5 OmniScript

OmniScript combines temporally aligned audio-visual inputs with hierarchical reasoning and structured script generation, then extends processing to longer videos through alternative context strategies.

  • Architecture: OmniScript aligns visual and audio embeddings at each timestamp to preserve synchrony for dialogue, narration, environmental sounds, and music.A Whisper encoder supplies audio features, which are paired one-to-one with visual embeddings.
  • Architecture: AV-DeepStack injects paired audio-visual tokens across multiple transformer layers, conditioning the language stream on both modalities.The architecture builds on Qwen3-VL and extends its visual-textual fusion to audio-visual-language inputs.
  • Architecture: The model first reasons about plot progression and character relationships before generating temporally grounded event fields and scene context.The structured output includes location, environment, character, action, expression, dialogue, and audio-aware cues.
  • Progressive Training and Alignment: OmniScript uses progressive modality alignment, multimodal pretraining, CoT supervised fine-tuning, and reinforcement learning with temporally segmented rewards.Training stages address audio integration, cinematic-domain adaptation, schema adherence, narrative coherence, and fine-grained event accuracy.
  • Progressive Training and Alignment: The reinforcement-learning reward performs event-level one-to-one matching across the video timeline to penalize fine-grained recall and precision errors.This localized evaluation is designed to avoid global semantic similarity masking short-duration event mistakes.
  • Long Video Processing: For videos shorter than 5 minutes, OmniScript processes the input directly; longer-video handling is explored through context extension and two-stage generation.The two-stage strategy segments the plot, generates scripts independently for each clip, and merges the outputs.

6 Experiments

Experiments evaluate OmniScript on 5-minute and substantially longer videos using semantic, temporal, fine-grained, and multi-dimensional metrics. The results show strong parameter efficiency, improved robustness to long contexts with TSG, and benefits from CoT and reinforcement learning.

  • Experimental Setup: OmniScript uses 8B parameters while achieving strong event-content and temporal-localization performance under the unified benchmark protocol.Evaluation reports F1 scores for event and scene fields, their Overall mean, and tIoU@0.1 for temporal localization.
  • 5-Minute Videos: 37.0 Overall and 69.0 tIoU@0.1: OmniScript surpasses Qwen3VL-235B-A22B by +4.0 Overall and +7.0 tIoU@0.1 at event level.It also reports stronger dialogue understanding and temporal localization than the cited proprietary models.
  • 5-Minute Videos: 52.6 scene-level Overall and 73.6 tIoU@0.1: OmniScript achieves better temporal boundary quality than Qwen3VL-235/22B at much smaller scale.Scene tIoU@0.1 is 74.6 for OmniScript versus 69.6 for TimeChat-Captioner†, while environment and mood remain challenging.
  • Long-Form Video Understanding: At 30 minutes, standard global-context models undergo severe volumetric collapse across almost all 14 evaluated axes because of long-range dependency errors and context dilution.At 10 and 20 minutes, OmniScript-LCE remains competitive with Gemini-2.5-pro before the context cliff emerges.
  • Long-Form Video Understanding: OmniScript-TSG maintains nearly unchanged multi-dimensional performance across extreme durations, whereas competing models shrink substantially at 30 and 40 minutes.The reported explanation is that segment-level conditioning provides localized constraints and anchors narrative consistency.
  • Ablation Studies: CoT raises Overall from 35.3% to 37.0% and Dialogue F1 from 68.2% to 71.0%, while RL further raises Overall to 37.7%.The segmented reward is reported to outperform the standard global reward by penalizing event-level precision and recall errors.

7 Conclusion

OmniScript is presented as an audio-visual language model for script-oriented understanding of long-form narrative videos. With 8B parameters, it outperforms much larger open-source models on event understanding and temporal grounding while remaining competitive on scene-level metrics.

  • Conclusion: OmniScript targets event and scene semantic parsing together with reliable temporal localization for long-form narrative videos.The paper presents comprehensive manually annotated video-script benchmarks featuring long, complex cinematic videos.
  • Conclusion: With only 8B parameters, OmniScript consistently outperforms much larger open-source models on event-level understanding and temporal grounding.It remains competitive on scene-level metrics and is described as balancing quality and efficiency.
  • Conclusion: The paper identifies better fine-grained multimodal perception as important for subtle attributes and boundary-sensitive localization.The subtitle-masking analysis highlights robustness differences between model families and the value of genuine audio-visual reasoning beyond text shortcuts.
  • Conclusion: The work contributes a baseline, benchmark, and training insights for future omni-modal narrative video understanding.The qualitative outputs organize videos into scene metadata and timestamped events with aligned narrative and audio-visual fields.

A Prompt Details

The prompts convert video segments into structured JSON by tracking scenes, characters, plots, relationships, and temporally grounded script events. The subsequent-segment workflow adds prior context to preserve continuity across the video.

  • Character-Centric Plot Reasoning: Character-centric prompts record segment duration, location, time period, continuity notes, character profiles, appearances, name updates, plot events, and relationships.The schema includes identifiers, names, aliases, appearance, voice, personality, timestamps, actions, and evidence for newly discovered names.
  • Initialization: The first-segment prompt identifies characters, summarizes the plot, and outputs results in a strict JSON format.It establishes the initial segment context before later segments are processed.
  • Character-Centric Plot Reasoning: Subsequent-segment processing injects previous character context and plot summaries to match identities, update profiles, and maintain narrative continuity.The prompt also records scene descriptions, plot development, and relationship updates for each segment.
  • Fine-Grained Script Generation: The script-generation prompt extracts chronological events while storing dialogue and action as independent fields.Each event must contain at least one of dialogue, action, expression, or audio_cue, while concurrent behaviors may share one event object.
  • Fine-Grained Script Generation: The prompt analyzes subtext, distinguishes dialogue types, preserves character identity, and emphasizes not omitting dialogue in long videos.Character IDs are maintained using visual features and voice characteristics.
  • High-Point Analysis: High-point mining identifies emotionally stimulating segments by combining visual, audio, and text evidence.Scene transitions occur when location, time, or environment type changes.
  • Scene and Event Schema: Scene records encode scene ID, location, interior or exterior type, environment, time, mood, and timestamped character events.Events include specific actions and facial expressions, with ambient sounds attributed to the environment.

B.1 Trainingset Statistics

The OmniScript training set combines long-form TV dramas with short vertical dramas, producing broad coverage across titles and cinematic genres.

  • Training-Set Composition: 45k clips span 781 unique titles across 10 cinematic genres in the OmniScript training set.The sources comprise synopsis-grounded long-form TV-drama clips and short vertical dramas from streaming platforms.
  • Genre Distribution: Period and romance dramas each represent roughly a quarter of the training data, followed by comedy at 11.6%, drama at 9.9%, and mystery at 9.0%.
  • Training-Set Composition: The training set’s complementary sources provide exposure to diverse narrative content and genre variation.The distribution is described across both long-form television and vertical short-drama formats.

B.2 Benchmark Statistics

The benchmark uses diverse 5-minute clips with a long-tail genre distribution and unusually dense temporal annotations, testing generalization across cinematic styles.

  • Genre Distribution: Mystery accounts for 19.5% and romance for 17.5% of benchmark clips, the largest genre proportions.
  • Genre Distribution: The benchmark also includes Action 10.0%, Drama 9.5%, Animation 8.0%, Youth 8.0%, Comedy 7.5%, War 7.5%, Horror 6.5%, and Period 6.0%.This long-tail distribution spans multiple cinematic styles.
  • Temporal Density and Structure: Its high annotation density departs from sparse captioning datasets by representing timestamped, character-linked events with dialogue, action, expression, or audio cues.The benchmark is designed to test generalization across differing lighting, composition, and narrative pacing.
Loading 2604.11102v1…