PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Grid Post-Training for Long-Form Multi-Shot Video Generation

Jiawei Mao, Haoqin Tu, Hardy Chen, Yuhan Wang, Keyang Xu, Jieru Mei, Hongliang Fei, Ruogu Fang, Wei Shao, Cihang Xie, Yuyin Zhou

arXiv:2609.06373v1cs.CV

TL;DR

Long-form multi-shot video generation must preserve motion within shots while maintaining consistent narratives across shots, a challenge existing generators face when shots are packed along one temporal axis. MovieGrid jointly generates temporally ordered chunks on a spatial grid using structured training and character-aware conditioning. It produces 6.05 times more shots under the same token budget and reports state-of-the-art intra-shot and inter-shot consistency, while scaling to longer videos through single or successive generations.

  • Problem

    Long-form multi-shot generation requires coherent within-shot motion and consistency across characters, environments, and narrative progression, but existing models struggle to satisfy both requirements.

  • Method

    MovieGrid jointly generates temporally ordered short video chunks arranged in a spatial grid, supported by MGLV, Noise-Free Random-Grid Training, Grid Embedding, Story Prompts, and Grid Boundary Loss.

  • Results

    6.05 times more shots are generated by MovieGrid than plain Temporal Packing in a 1,616-frame video under the same token budget, with state-of-the-art intra-shot and inter-shot consistency.

  • Takeaways & Limitations

    MovieGrid scales long-form multi-shot generation from 1,616 to 6,464 frames under a fixed token budget and can extend generation across successive grid videos.

Abstract

from arXiv · show

Generating long-form multi-shot videos requires coherent within-shot motion and visually consistent narratives across shots. Existing video generators favor continuous motion and struggle to present complete shot sets when an entire narrative is packed along one temporal axis. We propose MovieGrid, a Multi-Grid Post-Training paradigm that decomposes a long video into shorter, temporally ordered chunks and arranges them on a spatial grid for joint modeling. This design reduces the number of shots handled by each temporal axis while enabling global information exchange across chunks. We construct the Multi-Grid Long Video (MGLV) dataset from 1,000 long-form videos using source video collection, hierarchical segmentation, grid video construction, and character-aware story annotation, producing 54K grid videos paired with story prompts. Our Noise-Free Random-Grid Training retains a random subset of chunks as clean visual context for denoising the remaining chunks. Grid Embedding encodes grid structure, character-aware Story Prompts link recurring entities, and Grid Boundary Loss stabilizes layouts. Under the same token budget, MovieGrid generates 6.05 times more shots than Temporal Packing in a 1,616-frame video. On a benchmark spanning five real-world categories, it achieves state-of-the-art intra-shot consistency (0.9131 versus 0.8086 for HoloCine) and inter-shot consistency (0.5914 versus 0.5384 for StoryMem). MovieGrid can further scale video length with minimal compromise through single or multiple generations.

1 UC Santa Cruz 2 University of Florida 3 Vanderbilt University 4 Google

MovieGrid generates coherent multi-shot sequences across five visual styles, preserving subjects, environments, motion, and fine-grained scene details across shots.

  • MovieGrid maintains consistent subjects, environments, and overall visual appearance across temporally ordered shots in 3D CGI, anime, stop-motion, realistic, and cinematic styles.The examples include changes in viewpoint, shot scale, and scene composition.
  • The enlarged sequence demonstrates temporally coherent within-shot motion, stable continuous motion, and consistent fine-grained scene details over time.

1. Introduction

Long-form multi-shot generation must preserve motion within shots while maintaining characters, environments, and narrative progression across increasingly distant shots. MovieGrid addresses this by jointly generating temporally ordered chunks arranged on a spatial grid, achieving stronger shot realization and consistency under the reported evaluations.

  • Long-form multi-shot generation requires coherent within-shot motion and consistent characters, environments, and narrative progression across shots.These requirements become harder as videos grow longer and contain more visually diverse contexts.
  • Existing methods include autoregressive extension, keyframe-based interpolation, and holistic joint generation, each addressing long-horizon generation through different temporal or structural strategies.Autoregressive methods preserve local continuity but face error accumulation and increasing memory costs with longer histories.
  • MovieGrid decomposes long narratives into temporally ordered short chunks and distributes them across shorter temporal axes to reduce the shots handled by each axis.The design targets the bias toward continuous motion over presenting a complete multi-shot narrative.
  • MovieGrid spatially arranges consecutive chunks in a unified grid for joint generation, enabling cross-chunk information exchange and coordination of characters, environments, and narrative progression.Each chunk retains a local temporal axis, and generated chunks are unpacked in temporal order.
  • 6.05 times more shots are generated by MovieGrid than plain Temporal Packing in a 1,616-frame video under the same token budget.
  • MovieGrid achieves state-of-the-art intra-shot and inter-shot consistency for subjects and backgrounds while maintaining continuous within-shot motion across diverse visual styles.The reported comparisons include intra-shot subject and background consistency against HoloCine and inter-shot subject and background consistency against StoryMem.

2. Related Work

Prior work extends video generation through autoregressive, storyboard-based, and grid-structured approaches, but MovieGrid uses an ordered spatial grid of temporally evolving chunks as a joint representation for long-form narratives.

  • Autoregressive approaches propagate preceding frames, feature caches, or visual memories, but recursive conditioning can accumulate errors and visual drift.
  • Storyboard-based methods use keyframes as visual anchors before expanding them into individual video segments.
  • Grid-structured representations have been used for image generation, visual in-context learning, image editing, and video-frame arrangement.Grid Diffusion Models and GriDiT arrange video frames into 2D grids, while VIC concatenates clips for conditional completion.
  • MovieGrid represents a long-form multi-shot narrative as an ordered grid of temporally evolving video chunks, with each region retaining a local temporal axis.All chunks are jointly generated and unpacked in temporal order.

3. MGLV Dataset

MGLV converts long-form source videos into temporally ordered grid videos with character-aware story supervision. Fixed segmentation creates compact chunks, while MovieGrid’s training and conditioning components support joint generation across the grid.

  • MGLV comprises 54,281 grid videos derived from 1,000 long-form source videos through collection, hierarchical segmentation, grid construction, and character-aware annotation.
  • The source videos span cinematic, realistic, anime, cartoon, stop-motion, and 3D CGI styles and are manually filtered for quality, duplication, suitability, and irrelevant segments.
  • Each source video is resampled to 30 FPS, partitioned into 1,296-frame subvideos, and divided into 16 non-overlapping 81-frame chunks using fixed temporal intervals.Chunks do not necessarily correspond to single shots, avoiding shot-boundary detection overhead and errors.
  • Each 1,296-frame subvideo becomes an 81-frame 4×4 grid video representing all 1,296 original frames in chronological grid order.The average shot load decreases from 25.99 shots per subvideo to 1.76 shots per chunk, a 14.8× reduction along each modeled chunk.
  • Qwen3-VL produces timestamped character records and interval captions that are aggregated into temporally ordered Story Prompts with grid and recurring-entity tokens.
  • Noise-Free Random-Grid Training keeps selected chunks clean while noising the others, and MovieGrid uses Grid Embedding, Story Prompts, and Grid Boundary Loss for joint grid generation.The overall training objective combines flow matching with boundary supervision for grid structure.

4. Method

MovieGrid converts temporally ordered video chunks into a spatial grid for joint generation, combining structured grid conditioning, random clean-chunk context, and boundary-aware supervision. This transfers frames from the temporal axis to spatial layout while preserving ordering and supporting later grid-video extension.

  • Multi-Grid Representation: MovieGrid partitions a video into ordered chunks, tiles them spatially, and reconstructs the full video by unpacking and concatenating grids in index order.The grid represents N × T frames using T temporal steps without discarding frames.
  • Training Procedure: Noise-Free Random-Grid Training keeps randomly selected chunks clean while applying diffusion noise to the remaining chunks for joint denoising.The method activates this conditioning with probability pvis = 0.3 and samples 1–8 noise-free chunks when activated.
  • Inference Extension: At inference, selected grids from a previous generation can provide noise-free visual conditions for generating remaining grids in a subsequent grid video.This enables extension across successive grid videos without additional training.
  • Grid Conditioning: Grid Embedding combines grid identity, geometry, and intra-grid position to distinguish tokens across grids and encode their local coordinates.Geometry encodes normalized center coordinates and dimensions, while intra-grid position encodes each token’s normalized location.
  • Training Objective: MovieGrid jointly optimizes flow matching and grid boundary losses, with boundary supervision focused on latent interfaces between adjacent grids.The boundary loss uses a fixed mask and λGB = 0.1, encouraging stable separation without constraining interior visual content.

5. Experiments

Experiments show that MovieGrid improves long-form multi-shot consistency, shot realization, grid structure, length scaling, and long-range identity preservation across controlled comparisons and ablations.

  • Main Results: MovieGrid achieves state-of-the-art intra-shot and inter-shot consistency for subjects and backgrounds, while remaining competitive in aesthetic quality, dynamic degree, and semantic alignment.It scores 0.8970 and 0.9291 for intra-shot subject and background consistency, and 0.6139 and 0.5689 for inter-shot consistency.
  • Main Results: Relative to the controlled VIC-style baseline, MovieGrid improves intra-shot consistency from 0.3362 to 0.9131 and inter-shot consistency from 0.2189 to 0.5914.The matched comparison controls the backbone, grid scale, LoRA configuration, training data, and training settings.
  • Scaling Video Length: Scaling from 16 to 64 grids quadruples video length under a fixed token budget, but incurs a 13.45% consistency drop and a 2.97% semantic-alignment drop.The reported consistency changes are from 75.23% to 61.78%, while semantic alignment drops to 16.62%.
  • Grid Packing Strategy: Under the same token budget, MovieGrid detects 8.17 shots versus 1.35 for Temporal Packing and raises Ordered Story-Shot Recall from 36.61% to 83.07%.Temporal Packing increasingly favors continuous motion as the target shot count grows, whereas MovieGrid distributes chunks across shorter temporal axes.
  • Component Ablations: Removing Noise-Free Random-Grid Training causes the largest ablation degradation, an average 22.14% drop across four metrics, while Grid Embedding, Grid Boundary Loss, and character tags also contribute.The corresponding average drops are 13.26%, 9.23%, and 3.00%, respectively.
  • Long-Range Consistency: MovieGrid preserves recurring character identity, non-human subject identity, and fine-grained background details across distant shots despite intervening content.The qualitative comparisons also show preservation across shot transitions and coherent within-shot motion.

6. Conclusion

MovieGrid reformulates long-form multi-shot generation as joint generation of temporally ordered chunks within a spatial grid. This design supports consistent 1,616-frame videos, 6,464-frame single generations, and further extension across successive generations.

  • Conclusion: MovieGrid jointly models temporally ordered video chunks within a spatial grid, reducing shot transitions assigned to each temporal axis.The framework is built on MGLV and grid-aware post-training designs.
  • Conclusion: Increasing the grid count from 16 to 64 extends a single generation to 6,464 frames under a fixed token budget.The conclusion presents this as a length-scaling capability of the grid formulation.
  • Conclusion: MovieGrid further extends video length across successive generations by conditioning each new grid video on previous video chunks.This continuation strategy is presented as part of the framework’s scalable long-form generation paradigm.

A. Benchmark Detail

The benchmark evaluates long-form multi-shot generation at the story level rather than assuming one prompt or boundary per physical shot. It measures event coverage, temporal progression, and cross-event consistency across diverse domains.

  • Benchmark Construction: Stories are built from detector-free fixed temporal intervals, so one chunk may contain one or multiple physical shots.Corresponding captions are aggregated into character-aware Story Prompts without assuming one-to-one alignment with shots.
  • Evaluation Criteria: The benchmark evaluates whether generated videos realize requested events, preserve their specified order, and maintain recurring characters, objects, and environments.These are defined as event coverage, temporal progression, and cross-event consistency.
  • Benchmark Coverage: The benchmark contains 89 out-of-distribution stories with no narrative overlap with MGLV across five visual domains.The domains are 3D CGI, anime, cinematic, realistic documentary, and stop motion.

B. MGLV Details

MGLV uses character-aware annotation to create temporally ordered Story Prompts with persistent entity identifiers and explicit grid-layout prefixes. The annotation process provides narrative supervision without requiring alignment to physical shots or grids.

  • Character-Aware Annotation: The two-stage annotation pipeline identifies recurring foreground entities, assigns persistent identifiers, and records their visibility over consecutive 10-second intervals.The second stage then annotates scene, action, visual style, and cinematographic attributes for each interval.
  • Story Prompt Construction: Story Prompts associate recurring entities with persistent special tokens, concatenate interval descriptions in temporal order, and use a <grid N> prefix to declare the target grid layout.Here, N denotes the number of video chunks.
  • Annotation Scope: Annotation intervals provide ordered narrative supervision but are not required to align one-to-one with grids or physical shots.Videos are sampled at 0.5 FPS with deterministic decoding and malformed JSON outputs are automatically validated and retried.

C. More Results

MovieGrid-64 demonstrates scalable long-form generation through an 8 × 8 grid of temporally ordered chunks, producing a 6,464-frame multi-shot video. The supporting annotation prompts define global entities and structured segment descriptions for the MGLV dataset.

  • MovieGrid-64 qualitative result: 6,464 frames are produced by unpacking a jointly generated 8 × 8 grid of temporally ordered video chunks.The sequence covers diverse events and viewpoints while preserving recurring subjects and coherent visual context.
  • Dataset annotation: The global entity catalog identifies at most eight foreground, story-relevant subjects and assigns stable identifiers across the video.The annotation prompt uses identifiers such as C01 and C02 for consistent subject tracking.
  • Dataset annotation: Entity annotation groups visually similar crowd members into one entity unless recurring individuals are consistently distinguished.The prompt prioritizes foreground subjects that a human annotator would likely mention in a caption.
  • Dataset annotation: Prompt templates support global entity catalogs, coarse 10-second timelines, and structured captions for individual MGLV subvideo segments.These templates organize long videos before grid construction and captioning.
  • Dataset annotation: Segment captions separate visible setting, action, visual style, and video style into constrained fields without repeating entity identifiers.Scene and action descriptions have explicit word limits and exclude entity identifiers.
Loading 2609.06373v1…