Source-linked AI summary
OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory
Zhaochong An, Menglin Jia, Haonan Qiu, Zijian Zhou, Xiaoke Huang, Zhiheng Liu, Weiming Ren, Kumara Kahatapitiya, Ding Liu, Sen He, Chenyang Zhang, Tao Xiang, Fanny Yang, Serge Belongie, Tian Xie
TL;DR
Existing multi-shot video generation struggles to preserve long-range cross-shot context because fixed temporal windows and single-keyframe conditioning limit narrative modeling. OneStory reformulates generation as autoregressive next-shot synthesis with Frame Selection, Adaptive Conditioner, and referential-caption data, achieving superior narrative coherence across diverse text- and image-conditioned scenarios.
Problem
Existing multi-shot video methods struggle to model long-range cross-shot context because fixed windows discard older shots and keyframe conditioning restricts context to a single image.
Method
OneStory performs autoregressive next-shot synthesis with a pretrained I2V model, selecting semantically relevant prior frames and adaptively patchifying them for direct conditioning.
Results
OneStory achieves superior narrative coherence across diverse and complex narratives in both image-conditioned and text-conditioned generation.
Takeaways & Limitations
OneStory supports coherent, scalable long-form storytelling by combining global yet compact cross-shot context with referential-caption training data.
Takeaways & Limitations
Keyframe-based baselines cannot directly accept image inputs in the I2MSV setting, limiting their inclusion in that comparison.
Abstract
from arXiv · showhide
Storytelling in real-world videos often unfolds through multiple shots -- discontinuous yet semantically connected clips that together convey a coherent narrative. However, existing multi-shot video generation (MSV) methods struggle to effectively model long-range cross-shot context, as they rely on limited temporal windows or single keyframe conditioning, leading to degraded performance under complex narratives. In this work, we propose OneStory, enabling global yet compact cross-shot context modeling for consistent and scalable narrative generation. OneStory reformulates MSV as a next-shot generation task, enabling autoregressive shot synthesis while leveraging pretrained image-to-video (I2V) models for strong visual conditioning. We introduce two key modules: a Frame Selection module that constructs a semantically-relevant global memory based on informative frames from prior shots, and an Adaptive Conditioner that performs importance-guided patchification to generate compact context for direct conditioning. We further curate a high-quality multi-shot dataset with referential captions to mirror real-world storytelling patterns, and design effective training strategies under the next-shot paradigm. Finetuned from a pretrained I2V model on our curated 60K dataset, OneStory achieves state-of-the-art narrative coherence across diverse and complex scenes in both text- and image-conditioned settings, enabling controllable and immersive long-form video storytelling.
1 Introduction
Multi-shot video generation must preserve narrative entities and reason across discontinuous scenes, but existing fixed-window and keyframe-based approaches lose long-range context. OneStory addresses this with autoregressive next-shot synthesis, adaptive memory, a referential-caption dataset, and training strategies for coherent long-form generation.
- Motivation: Multi-shot generation requires persistent entities and selective preservation of invariant and evolving scene attributes across discontinuous shots.The central difficulty is maintaining long-term cross-shot context while supporting changes in time, location, viewpoint, and action.
- Limitations of Existing Methods: Fixed-window methods discard older shots as the window advances, while keyframe pipelines compress cross-shot context to one image and weaken complex storyline adherence.These limitations produce memory loss and inconsistency beyond the window or restrict propagation of narrative cues.
- OneStory: OneStory reformulates multi-shot generation as autoregressive next-shot synthesis built on a pretrained image-to-video model.This formulation provides strong visual conditioning while generating shots sequentially.
- OneStory: Frame Selection retrieves sparse, semantically relevant frames from all prior shots, and Adaptive Conditioner importance-guidedly patchifies them into compact direct conditioning.Together, the modules provide global yet compact cross-shot context without relying on fixed temporal ordering.
- Data and Training: OneStory is trained with a roughly 60K curated multi-shot dataset containing shot-level referential captions and next-shot training strategies.The dataset uses a three-step curation pipeline and is designed to reflect evolving real-world storytelling without a separate global script.
- Results: OneStory achieves superior narrative coherence for minute-scale ten-shot videos across image-conditioned, text-conditioned, and out-of-domain scenarios.The reported examples show consistent characters and environments while following complex, evolving prompts.
2 Related Works
Single-shot video generation has advanced visual fidelity through text-to-video, image-to-video, and diffusion-transformer models, while multi-shot methods primarily model cross-shot context with bounded temporal windows.
- Single-shot Video Generation: Single-shot video generation includes text-to-video and image-to-video models, with diffusion transformers enabling unified spatial-temporal modeling.Large-scale systems extend diffusion architectures with temporal modules or transformer-based designs.
- Multi-shot Video Generation: Multi-shot methods primarily use fixed-window attention to model multiple shots within a bounded temporal context.Representative approaches modify attention masks for caption-shot alignment or augment architectures for longer context encoding.
3 High-quality Multi-shot Video Dataset
The dataset defines high-quality multi-shot videos through thematic consistency and coherent narrative progression rather than unrelated clip concatenation. It uses referential shot-level captions to support flexible evolution across shots.
- Dataset Definition: A high-quality multi-shot video maintains a consistent theme and coherent narrative progression across shots.The definition distinguishes coherent storytelling from concatenated unrelated clips.
- Caption Design: Unlike global-script approaches, the dataset uses shot-level captions with referential narrative flow so subsequent shots can evolve flexibly.This design avoids fixing the entire storyline in a separate global script.
- Construction Pipeline: The curation pipeline detects shot boundaries, retains videos with at least two shots, and generates captions through independent captioning followed by contextual rewriting.Rewriting subsequent captions based on prior frames and captions introduces references such as “the same man.”
- Dataset Outcome: The resulting captions provide narrative continuity and a foundation for training multi-shot generation models.The dataset construction is intended to support coherent progression across shots.
4 Method
OneStory reformulates multi-shot video generation as next-shot synthesis with autoregressive visual conditioning. It selects semantically relevant historical frames and adaptively compresses them into compact context tokens for coherent generation.
- Task Formulation: OneStory reframes multi-shot generation as predicting each final shot from preceding shots, enabling autoregressive synthesis initialized from pretrained I2V models.Training predicts the final shot from prior shots, while inference generates videos sequentially from a memory bank.
- Frame Selection: The Frame Selection module encodes all preceding shots into a global memory and selects frames most relevant to the current shot.Caption-conditioned queries interact with visual memory to score frames, after which the top-Ksel frames form historical context.
- Adaptive Conditioner: OneStory assigns finer patchifiers to highly relevant, potentially non-adjacent frames instead of allocating compression according to temporal order.This importance-guided patchification creates content-driven context tokens from the selected memory.
- Adaptive Conditioner: The Adaptive Conditioner concatenates compact context tokens with current-shot noise tokens, enabling joint attention while balancing context retention and computation.Adjusting patchifiers controls compression, producing relevance-aware conditioning with limited additional computation.
- Training Strategies: Shot inflation converts two-shot sequences into synthetic three-shot training examples, enabling uniform training when the dataset contains relatively few three-shot videos.Synthetic triplets are created by inserting a sampled shot or augmenting the first shot.
5 Experiments
OneStory is evaluated against fixed-window, keyframe-conditioning, and edit-and-extend baselines across text- and image-conditioned multi-shot generation. It achieves stronger narrative and shot-level quality, while ablations and qualitative cases support the roles of adaptive memory, frame selection, and training design.
- 5.1 Experimental Setup: OneStory consistently outperforms all baselines on shot-level quality and narrative consistency in both T2MSV and I2MSV settings.The evaluation covers subject, background, aesthetic, motion, character, environment, and semantic-alignment metrics.
- 5.2 Main Results: Qualitative comparisons show that OneStory follows evolving shot captions while preserving characters and environments across reappearances and compositional scenes.Baselines exhibit viewpoint errors, wrong-character generations, and identity failures in later shots.
- 5.3 Ablation Study: Combining the Adaptive Conditioner and Frame Selection modules yields the best ablation performance, confirming their complementary roles in cross-shot context modeling.Using only the last frame performs weakest because it lacks historical context; the conditioner expands contextual range and frame selection improves relevance.
- 5.3 Ablation Study: Automatic frame selection maintains contextual consistency in sixth-shot generation and dynamic-motion cases, whereas uniform and most-recent sampling fail.The comparison uses the same context budget and includes a first shot with large camera motion requiring precise frame selection.
- 5.3 Ablation Study: Shot Inflation and Decoupled Conditioning improve narrative learning over mixed two- and three-shot training.Shot Inflation supplies richer temporal context, while the decoupled conditioning curriculum stabilizes early optimization.
- 5.3 Ablation Study: A single latent-frame equivalent of context tokens already provides strong inter-shot coherence, while larger token budgets further improve performance.The default configuration uses one latent-frame equivalent of context tokens, supporting compact adaptive memory.
6 Conclusion
OneStory models coherent multi-shot video generation through adaptive memory and a next-shot formulation. Its Frame Selection and Adaptive Conditioner modules provide compact global context for scalable autoregressive story synthesis.
- OneStory achieves superior narrative coherence for complex multi-shot narratives through adaptive memory modeling.The framework is presented as a solution for coherent, scalable, and story-driven video generation.
- The next-shot formulation enables autoregressive shot synthesis while leveraging the visual conditioning capacity of pretrained I2V models.
- Frame Selection identifies semantically relevant frames across prior shots, while Adaptive Conditioner performs importance-guided patchification with direct condition injection.
A Additional Training Details
This section expands the training formulation underlying OneStory, covering unified three-shot training and frame-level pseudo-label construction.
- The additional training details describe unified three-shot training and frame-level pseudo-label construction.These details complement the corresponding training sections of the main paper.
A.1 Unified Three-Shot Training
OneStory standardizes training around three-shot sequences by synthesizing missing shots and predicting the final shot from preceding context and its caption. Frame relevance supervision uses similarity-based and coarse labels to teach the selector which historical frames matter.
- A.1 Unified Three-Shot Training: Unified three-shot training stabilizes optimization by converting two-shot samples into three-shot sequences through synthetic-shot construction.Two-shot sequences are most common, so an additional shot is synthesized before training.
- A.1 Unified Three-Shot Training: Synthetic intermediate shots are created either by inserting a shot from another video or by spatially or chromatically augmenting the first shot.
- A.1 Unified Three-Shot Training: The model predicts the final shot conditioned on the first two shots and the final-shot caption under a rectified-flow diffusion objective.Real triplets use (Sfirst, Ssecond, Slast), while synthetic triplets use an inserted or augmented intermediate shot; Slast remains the target.
- A.2 Frame Relevance Pseudo-Labels: Frame-level pseudo-labels estimate historical-frame relevance using DINOv2 and CLIP similarity for real frames and coarse labels for synthetic frames.Randomly inserted shots receive y_r = -1, whereas augmented-first-shot frames receive y_r = 0.
- A.2 Frame Relevance Pseudo-Labels: A regression loss supervises predicted frame relevance scores, with λ weighting selector supervision in the joint training objective.This encourages informative context selection while maintaining high-fidelity generation.
B Additional Details on Evaluation Benchmark
The benchmark evaluates text- and image-conditioned multi-shot generation under realistic referential narratives. It covers identity preservation, long-range recall amid distractors, and composition of separately introduced characters.
- B Additional Details on Evaluation Benchmark: The benchmark uses referential captions with progressive narrative flow to reflect realistic storytelling conditions.
- B Additional Details on Evaluation Benchmark: Main-subject consistency tests whether characters preserve identity across changing environments and actions.
- B Additional Details on Evaluation Benchmark: Insert-and-recall tests long-range memory by returning to primary subjects after an intervening scene or character distractor.
- B Additional Details on Evaluation Benchmark: Composable generation tests whether characters introduced separately can be integrated into a coherent later scene.
- B Additional Details on Evaluation Benchmark: The benchmark contains 64 six-shot test cases each for T2MSV and I2MSV across diverse subjects, environments, and cross-shot relationships.
C Additional Qualitative Results
Qualitative examples show that OneStory maintains continuity across long-range shots, preserving identities, background details, and reappearing narrative elements. The model also supports compositional scenes where characters from different shots appear together.
- Overview: The qualitative analysis evaluates continuity across shots using examples from the main paper, with additional videos available on the project page.The analysis focuses on identity consistency, background details, and reappearance or composition.
- Identity consistency: OneStory preserves character identity across long-range shots despite changes in viewpoint and actions.The same subject remains consistent across Shots 4–5 under viewpoint changes and Shots 1, 3, and 8 under action changes.
- Background details: Fine-grained background details remain spatially aligned across shots despite substantial cross-shot dynamics.Plants and fences persist from Shot 1 to Shot 7, while red flowers recur across Shots 1, 4, 5, 6, 7, and 9.
- Reappearance and composition: The model recalls characters or environments that disappear and reappear after several intervening shots.Examples include reappearances between Shots 4 and 9 and between Shots 2 and 6.
- Reappearance and composition: OneStory supports composable generation by bringing characters from separate narrative shots together in a later scene.In Shot 7, the woman from Shot 1 and the man from Shot 4 appear together.