PaperlayerView exact revision on arXiv

Source-linked AI summary

Captain Cinema: Towards Short Movie Generation

Junfei Xiao, Ceyuan Yang, Lvmin Zhang, Shengqu Cai, Yang Zhao, Yuwei Guo, Gordon Wetzstein, Maneesh Agrawala, Alan Yuille, Lu Jiang

arXiv:2507.18634v1cs.CV

TL;DR

Generating coherent narratives over extended video durations remains challenging. Captain Cinema plans narrative keyframes, synthesizes video conditioned on them, and trains MM-DiTs for long-context cinematic data. It reports strong long-form narrative video performance and visually consistent short films that exceed existing durations.

  • Problem

    Video-generation advances mainly address visual fidelity and local temporal coherence, leaving coherent, engaging narratives over extended durations challenging.

    Source: §1 · p. 1

  • Method

    Captain Cinema plans narrative keyframes top-down, synthesizes conditioned video bottom-up, and uses GoldenMem and progressive long-context MM-DiT fine-tuning.

    Source: §1 · p. 2

  • Results

    Captain Cinema reports strong long-form narrative video performance and visually consistent short films exceeding existing durations; at 48 pairs, it retains >93% of its 8-pair consistency.

    Source: §1 · p. 2, §4.1 · p. 9

  • Takeaways & Limitations

    The authors describe Captain Cinema as a concrete step toward fully automated, story-driven movie generation.

    Source: §5 · p. 11

    Frame-level and video-level modules are trained separately, narratives rely on human or LLM-supplied text, and generalization is constrained by scarce high-quality feature-length movie data.

    Source: §5 · p. 11

Abstract

from arXiv · show

We present Captain Cinema, a generation framework for short movie generation. Given a detailed textual description of a movie storyline, our approach firstly generates a sequence of keyframes that outline the entire narrative, which ensures long-range coherence in both the storyline and visual appearance (e.g., scenes and characters). We refer to this step as top-down keyframe planning. These keyframes then serve as conditioning signals for a video synthesis model, which supports long context learning, to produce the spatio-temporal dynamics between them. This step is referred to as bottom-up video synthesis. To support stable and efficient generation of multi-scene long narrative cinematic works, we introduce an interleaved training strategy for Multimodal Diffusion Transformers (MM-DiT), specifically adapted for long-context video data. Our model is trained on a specially curated cinematic dataset consisting of interleaved data pairs. Our experiments demonstrate that Captain Cinema performs favorably in the automated creation of visually coherent and narrative consistent short movies in high quality and efficiency. Project page: https://thecinema.ai

1 Introduction

Recent video generation has advanced visual fidelity and local temporal coherence, but coherent storytelling over extended durations remains challenging. Captain Cinema combines keyframe planning, conditioned video synthesis, and long-context training to generate narrative films.

1 Introduction

  • Video generation has advanced visual fidelity and local temporal coherence, while coherent, engaging narratives over extended durations remain a challenge.Longer videos also pose challenges of long-range dependencies, storyline discontinuities, and visual drift.Source: §1 · p. 1
  • A top-down planner outlines the narrative with keyframes, while a bottom-up synthesizer generates conditioned motion; GoldenMem compresses past context to preserve character and scene consistency across acts.The framework is designed to balance global plot structure with local visual fidelity.Source: §1 · p. 2
  • A specialized data-processing pipeline and progressive long-context tuning strategies target stable, efficient MM-DiT fine-tuning on large-scale, long-form cinematic data.These techniques address multi-scene video-generation challenges.Source: §1 · p. 2
  • Captain Cinema achieves strong long-form narrative video performance and creates visually consistent short films that significantly exceed the duration of existing works.Extensive experiments and ablation studies support these reported outcomes.Source: §1 · p. 2

2 Related Works

Prior work addresses short clips, single-turn or single-shot interleaved generation, narrative image sets, and visual-token compression. Captain Cinema distinguishes itself by targeting multi-shot, long-form narrative video with interleaved keyframe planning, conditioned synthesis, and compressed visual context.

Text-to-Video Generation

  • Text-to-video models commonly generate 5–10 s clips, while existing approaches lack mechanisms for narrative or visual consistency over longer horizons.The cited approaches include diffusion-based models [23] and autoregressive alternatives [14] [15] [16] [17].Source: §2 · p. 3

Interleaved Image–Text Modeling & Conditioning

  • Most interleaved systems generate content once, and multi-keyframe-conditioning approaches remain single-shot; Captain Cinema models cross-shot coherence for long-form generation.VideoAuteur [38] uses an interleaved VLM director, while LCT [39] fine-tunes MM-DiTs for long interleaved video generation.Source: §2 · p. 3

Narrative Visual Generation

  • Where prior narrative visual-generation work often targets semantically consistent image sets [47–49], Captain Cinema targets coherent narrative videos through keyframe synthesis and multi-frame-conditioned completion.The paper reports state-of-the-art results in long narrative movie generation.Source: §2 · p. 3

Visual Generation with Token Compression

  • Prior token-compression methods include context packing, adaptive image tokenization, compressed VAEs, and hierarchical video representations; discrete keyframe-interval memory is less discussed.The paper identifies memory in discrete space as important for processing interleaved frame contexts in long-form narrative visuals.Source: §2 · p. 3

3 Method

Captain Cinema learns from interleaved data sampled across whole movies, then plans narrative keyframes and synthesizes the shots between them. Its planning stage combines compressed visual memory, semantic retrieval, progressive training, and hybrid attention; the video generator conditions on keyframes and a global caption.

3.1 Learning Long-Range Context in Movies

  • The keyframe model learns from interleaved image-text pairs and the video model from interleaved video-text pairs sampled across entire movies, supporting generalization across scenes and cinematic contexts.The figure describes frames annotated with detailed visual descriptions and major-character names.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Top-Down Interleaved Keyframe Planning

  • Hybrid attention uses block-diagonal local masks in early double-stream blocks and global attention in later single-stream blocks to aggregate context for coherent keyframe generation.An upper-triangular global mask supports causal autoregressive generation.Source: §3.2 · p. 4
  • GoldenMem stores the current frame at full resolution and earlier frames at progressively coarser scales, keeping a (k+1)-frame history at a fixed small overhead.The inverse-Fibonacci scales make total conditioning cost a constant factor above the single-frame cost.Source: §3.2 · pp. 4–5
  • Semantic-similarity retrieval accommodates non-linear narratives, and CLIP text-to-image retrieval yields higher coverage than T5 text-to-text across retrieval depths.The approach conditions each new frame on embeddings from CLIP and T5 rather than relying strictly on temporal order.Source: §3.2 · pp. 5–6
  • Progressive finetuning grows interleaved context from single-image warm-up through 8, 16, and 32 pairs, while dynamic-stride sampling provides thousands of times more valid sequences than consecutive sampling.The stated comparison uses a 25% overlap rate for dynamic-stride sampling.Source: §3.2 · p. 6

3.3 Bottom-Up Keyframe-Conditioned Video Generation

  • The bottom-up generator conditions on a tiled global caption and visual embeddings of keyframes, which are each the first frame of a shot, to generate the remaining shot frames.Multi-key conditioning anchors appearance at shot boundaries and supports motion dynamics across shots.Source: §3.3 · p. 6

4 Experiment

Captain Cinema combines top-down keyframe planning with bottom-up video synthesis, and the reported evaluations find favorable performance across most metrics and user-study ratings. Stress tests, ablations, and generalization examples examine long-context robustness and the system’s creative capabilities.

4.1 Main Results

  • The qualitative example uses Gemini-generated shot descriptions to guide keyframe generation, then conditions video synthesis on text–keyframe pairs to produce a multi-scene film.The figure presents 24 representative shots, described as demonstrating narrative coherence, character fidelity, and visual style.Source: §4.1 · p. 6, Figure 6 · p. 7
  • At 48 interleaved pairs, Captain Cinema retains >93% of its 8-pair consistency, while LCT quality degrades sharply as context grows.The stress test evaluates context windows from 8 to 48 pairs.Source: Table 2 · p. 8, §4.1 · pp. 8–9
  • Captain Cinema performs favorably across most metrics, with user studies corroborating video-quality and semantic-relevance ratings; its clearest advantages are in temporal dynamics.The evaluation compares against LCT and IC-LoRA combined with I2V.Source: §4.1 · pp. 7–8

4.2 Ablation Studies

  • For FLUX-like distillation models, progressive finetuning with a modest warmup avoids the collapse of direct finetuning and artifacts associated with a late checkpoint.The reported warmup is step 40,000; the late-checkpoint condition is step 80,000.Source: §4.2 · p. 9, Figure 8 · p. 9
  • Combining top-down keyframes with bottom-up video synthesis is reported to reduce artifacts and yield consistent environment dynamics and smoother camera motion.Examples include burning ruins with rising smoke and stable character identities.Source: §4.2 · p. 9, Figure 9 · p. 9
  • GoldenMem reduces computation with minor visual-quality and consistency degradation, and supports expanding the context window from 32 to 48 without out-of-memory errors.The table compares baseline and GoldenMem settings for the same visual context.Source: §4.2 · p. 9, Table 3 · p. 10
  • Moderate conditioning noise supports character consistency, while small noise levels produce the best facial preservation in long-context keyframe generation.The reported ranges are 1–500 steps for character consistency and 1–250 steps for best facial preservation.Source: Figure 10 · p. 10, §4.2 · p. 10

4.3 Generalization Abilities

  • Generalization examples include generating novel scenes from familiar story elements and swapping Bruce Wayne and Alfred Pennyworth into Interstellar’s science-fiction setting.The cross-movie example is described as showing separation between character identity and scene context.Source: §4.3 · pp. 10–11

5 Limitations & Conclusion

The authors identify constraints in training, narrative autonomy, and data availability, while concluding that Captain Cinema is a step toward story-driven movie generation. They also discuss potential applications and risks of long-form video generation.

5 Limitations & Conclusion

  • The frame-level and video-level modules are trained separately because of memory and infrastructure constraints, rather than through end-to-end image-to-video training.Generated frames are used as conditioning inputs for video synthesis.Source: §5 · p. 11
  • The model relies on narratives supplied by human authors or large language models and cannot yet devise them autonomously.Source: §5 · p. 11
  • Scarcity of high-quality feature-length movie datasets curtails generalization, motivating larger corpora and more real-world validation.Source: §5 · p. 11
  • The authors describe the work as a concrete step toward fully automated story-driven movie generation and note risks including misinformation, non-consensual media, intellectual-property violations, and computational impact.Proposed mitigations include gated releases, watermarking, documentation, usage policies, audits, and stakeholder collaboration.Source: §5 · p. 11

A Implementation Details

The appendix reports separate training configurations for keyframe generation and visual-conditioned video generation. Both models use AdamW and bfloat16, with distinct resolutions, batch sizes, learning rates, and schedules.

A Implementation Details

  • The interleaved keyframe generator uses 400 × 800 resolution, batch size 32, bfloat16, AdamW, a 2 × 10^-4 peak learning rate, cosine decay, and 2,500 training steps.Training context pairs range from 2 to 8; inference uses 8 pairs.Source: A Implementation Details · p. 16, Table 4 · p. 16
  • The visual-conditioned video generator uses 480 × 480 video data, batch size 64, bfloat16, AdamW, a 1 × 10^-5 peak learning rate, constant decay, and 20,000 training steps.The passage gives the processed video area as 230,400 pixels.Source: A Implementation Details · p. 16, Table 4 · p. 16
  • The table specifies AdamW for both models, with β1=0.9 and ϵ=10^-8; β2 is 0.999 for keyframe generation and 0.95 for video generation.Source: A Implementation Details · p. 16, Table 4 · p. 16

B Data Annotation Prompt

The annotation prompt directs a film-director assistant to describe frame composition and narrative details, while applying rules for identifying characters and writing concise, specific descriptions.

B Data Annotation Prompt

  • The prompt asks the annotator to describe visual composition, lighting, camera angles, character actions and positioning, and setting and atmosphere.Source: B Data Annotation Prompt · p. 16
  • Character names are used only when identification is certain; otherwise, the annotator describes appearance, role, or action without angle brackets.For dual-identity roles, the prompt says to choose the name matching the on-screen persona.Source: B Data Annotation Prompt · p. 16
  • Descriptions should be concise yet thorough, covering technical and narrative specifics that make the frame distinctive.Source: B Data Annotation Prompt · p. 16

C Additional Qualitative Results for Keyframe Generation.

The appendix presents non-cherry-picked qualitative keyframe-generation results. The captions describe 32-frame generations and claim superior generation quality with character and scene consistency.

C Additional Qualitative Results for Keyframe Generation.

D Long-Context Stress Test Details

The stress test evaluates long-range consistency in ground-truth and generated keyframe sequences using Gemini 2.5 Flash. It scores six aspects, with trends shown in Fig. 7 and context-length results reported in Tab. 2.

D Long-Context Stress Test Details

SYSTEM_PROMPT

The system prompt asks an expert visual analyst to score movie-keyframe sequences across six aspects. Scores use a 0–5 scale, with criteria defined for each aspect.

SYSTEM_PROMPT

  • The analyst evaluates character consistency, scene consistency, visual quality, aesthetics, diversity, and narrative coherence.Source: SYSTEM_PROMPT · p. 21
  • Each aspect receives a 0–5 score and a one-sentence justification, with scores ranging from unacceptable to excellent.Source: SYSTEM_PROMPT · p. 21
  • The rubric judges consistency by stable appearance and settings, and narrative coherence by logical progression without abrupt shifts.Source: SYSTEM_PROMPT · p. 21
  • The requested JSON output contains a score and justification for each of the six aspects.Source: SYSTEM_PROMPT · p. 21

E Out-of-domain Identity Preservation Generalizability

The out-of-domain test conditions generation on noisy visual context from unseen people and prompts from their remaining captions. The authors report that the keyframe generator maintains identity while following appearance instructions.

E Out-of-domain Identity Preservation Generalizability

Related papers

Loading 2507.18634v1…