PaperlayerView exact revision on arXiv

Source-linked AI summary

DreaMontage: Arbitrary Frame-Guided One-Shot Video Generation

Jiawei Liu, Junqiao Li, Jiangfan Deng, Gen Li, Siyu Zhou, Zetao Fang, Shanshan Lao, Zengde Deng, Jianing Zhu, Tingting Ma, Jiayi Li, Yunqiu Wang, Qian He, Xinglong Wu

arXiv:2512.21252v2cs.CV

TL;DR

One-shot video generation is costly to produce physically, and clip concatenation does not ensure smooth, coherent transitions. DreaMontage combines arbitrary frame conditioning, staged visual training, and segment-wise autoregressive generation; the authors report superior controllability and visual fidelity to state-of-the-art baselines.

  • Problem

    Physical one-shot production is costly, and concatenating clips generated with first-last-frame conditioning does not guarantee smooth, coherent transitions.

    Source: §1 · p. 1

  • Method

    DreaMontage combines intermediate frame conditioning with Adaptive Tuning, Visual Expression SFT, Tailored DPO, and Segment-wise Auto-Regressive generation.

    Source: §1 · p. 2

  • Results

    The authors report superior controllability and visual fidelity compared with state-of-the-art baselines.

    Source: §5 · p. 12

  • Takeaways & Limitations

    The framework is presented as a way to turn disparate visual assets into a unified sequence with visual fidelity and temporal coherence.

    Source: §1 · p. 2

Abstract

from arXiv · show

The "one-shot" technique represents a distinct and sophisticated aesthetic in filmmaking. However, its practical realization is often hindered by prohibitive costs and complex real-world constraints. Although emerging video generation models offer a virtual alternative, existing approaches typically rely on naive clip concatenation, which frequently fails to maintain visual smoothness and temporal coherence. In this paper, we introduce DreaMontage, a comprehensive framework designed for arbitrary frame-guided generation, capable of synthesizing seamless, expressive, and long-duration one-shot videos from diverse user-provided inputs. To achieve this, we address the challenge through three primary dimensions. (i) We integrate a lightweight intermediate-conditioning mechanism into the DiT architecture. By employing an Adaptive Tuning strategy that effectively leverages base training data, we unlock robust arbitrary-frame control capabilities. (ii) To enhance visual fidelity and cinematic expressiveness, we curate a high-quality dataset and implement a Visual Expression SFT stage. In addressing critical issues such as subject motion rationality and transition smoothness, we apply a Tailored DPO scheme, which significantly improves the success rate and usability of the generated content. (iii) To facilitate the production of extended sequences, we design a Segment-wise Auto-Regressive (SAR) inference strategy that operates in a memory-efficient manner. Extensive experiments demonstrate that our approach achieves visually striking and seamlessly coherent one-shot effects while maintaining computational efficiency, empowering users to transform fragmented visual materials into vivid, cohesive one-shot cinematic experiences.

1 Introduction

Physical one-shot filmmaking is costly and constrained, while clip concatenation does not guarantee smooth, coherent transitions. DreaMontage targets arbitrary temporal control and continuity through intermediate conditioning, staged visual training, and segment-wise generation.

1 Introduction

  • Physical one-shot production incurs substantial costs and skill demands, while first-last-frame clip concatenation cannot guarantee smooth, coherent transitions.The task calls for a continuous shot guided by images or clips at temporal positions, with transitions that remain coherent.Source: §1 · pp. 1–2
  • Intermediate conditions are difficult to represent precisely because causal 3D VAE temporal downsampling makes a latent span multiple frames.DreaMontage uses intermediate conditioning and Adaptive Training to provide arbitrary frame control at relatively low cost.Source: §1 · pp. 2–3
  • DreaMontage combines intermediate conditioning, Visual Expression SFT, Tailored DPO, and Segment-wise Auto-Regressive generation for arbitrary frame-guided one-shot video.The framework targets coherent transitions between user-provided images or clips at specified temporal positions.Source: §1 · p. 2
  • High-quality data with SFT and DPO forms a progressive training pipeline for generating vivid, seamless, and coherent one-shot videos.The framework applies DPO to address abrupt cuts and unnatural subject motion.Source: §1 · pp. 2–3
  • Segment-wise Auto-Regressive generation is designed to support long one-shot videos while balancing performance and efficiency.The strategy decouples long-video generation from strict computational and memory constraints while preserving one-shot content integrity.Source: §1 · pp. 2–3

2 Related Work

Video-generation work has expanded beyond image-to-video, but first- and last-frame conditioning remains restrictive. Existing conditioning strategies trade off flexibility and computational cost, motivating more efficient support for intermediate conditions.

2 Related Work

  • First- and last-frame conditioning remains restrictive for controlling motion evolution and intermediate transformations in long-take videos.The related-work discussion frames combinations of images and videos at arbitrary temporal positions as a more flexible conditioning setting.Source: §2 · p. 3
  • Conditioning strategies include timestep-based control and semantic image injection, while some methods encode guidance across the target video’s full length.Full-length guidance encoding is described as computationally costly and redundant; semantic injection increases contextual sequence length and cost.Source: §2 · pp. 3–4
  • DreaMontage extracts condition latents and concatenates them with noise latents channel-wise to handle multiple intermediate conditions.The paper describes this as simpler and more efficient than encoding a conditional video of target length.Source: §2 · p. 4

3 Method

DreaMontage modifies a DiT-based video-generation pipeline for intermediate-frame conditioning, then adds targeted training and segment-wise generation for arbitrary-frame-guided long videos.

3.1.1 Model Design

  • Shared-RoPE adds sequence-wise reference latents at target-frame RoPE positions alongside channel conditioning, applying this strategy only to a video condition’s first frame.Causal VAE latents can span multiple target frames, creating correspondence issues for intermediate-frame conditioning; lightweight tuning largely addresses this mismatch.Source: §3.1.1 · pp. 4–5

3.1.2 Adaptive Tuning

  • Adaptive Tuning filters for single-shot videos with visual variation, aesthetic quality, motion, and pose, then trains with sampled image and video conditions.For video-condition segments, the first frame is re-encoded while subsequent frames are resampled from the latent distribution to approximate causal encoding.Source: §3.1.2 · pp. 5–6

3.2 Visual Expression SFT

  • Visual Expression SFT uses category-balanced data with longer videos, up to 20 seconds, and more seamless scene transitions to improve motion dynamics and instruction-following.The dataset taxonomy targets five underperforming areas: Camera Shots, Visual Effects, Sport, Spatial Perception, and Advanced Transitions.Source: §3.2 · p. 6

3.3 Tailored DPO

  • Tailored DPO uses VLM-selected cut and non-cut pairs and human-screened subject-motion pairs, then optimizes a likelihood margin against an SFT-initialized reference.The objective targets abrupt cuts and implausible subject motion while controlling deviation from the reference model.Source: §3.3 · pp. 6–7

3.4 Segment-wise Auto-Regressive Generation

  • SAR partitions latent videos with a variable-length sliding window, generates segments autoregressively from prior-segment tail latents and local guidance, then fuses overlaps before decoding.The authors state that conditioning on prior-segment tail latents maintains continuity across segment boundaries and that latent-space generation yields smoother transitions than pixel-based approaches.Source: §3.4 · pp. 7–8

4 Experiment

The evaluation demonstrates arbitrary frame-guided generation qualitatively, benchmarks first-last and multi-keyframe settings against existing models, and tests individual optimizations through ablations. It also explores creative workflows supported by mixed-condition generation.

4.1 Experimental Settings

  • The internal test set spans 5–60-second mixed-condition sequences, with derived multi-keyframe and first-last subsets; blinded expert GSB ratings cover four dimensions.The dimensions are visual quality, motion effects, prompt following, and overall preference; GSB is calculated from wins, losses, and ties.Source: §4.1 · pp. 8–9

4.2 One-shot Video Generation

  • Qualitative examples show transitions across disparate concepts, video-guided scene changes and extensions, and a hybrid image-video trajectory.The demonstrations include a train-to-city transition, skiing-to-surfing, a cat extending its bike ride to a horse, and a rider becoming an astronaut.Source: §4.2 · p. 9

4.3 Comparison with existing Methods

  • In multi-keyframe overall preference, DreaMontage leads Vidu Q2 by +15.79% and Pixverse V5 by +28.95%; versus Kling 2.5, it ties visual quality and leads overall preference by +3.97%.Against Vidu Q2 and Pixverse V5, prompt following gains are +23.68%; against Kling 2.5, motion effects and prompt following each gain +4.64%.Source: §4.3 · pp. 9, 11, Figure 7 · p. 11

4.4 Ablation Study

  • SFT improves motion effects by +24.58% and overall preference by +20.34%; Tailored DPO scores +12.59% on abrupt cuts and +13.44% on subject motion, while Shared-RoPE preference reaches +53.55%.DPO training smooths narrative bridges and mitigates anatomical distortions and unnatural movement; Shared-RoPE aligns high-resolution generation with conditioning signals to address flickering and color shifts.Source: §4.4 · p. 11

4.5 Exploration on Applications

  • The application scenarios include storyboard-based trailer creation, autoregressive long-video extension, and repurposing mixed image and video assets for games and advertising.The paper describes these as potential workflows, including pre-visualization and brand-style-faithful motion.Source: §4.5 · p. 12

5 Conclusion

DreaMontage combines arbitrary frame-guided conditioning, memory-efficient segment-wise autoregressive generation, and progressive training. The paper reports improved controllability and visual fidelity over state-of-the-art baselines.

  • DreaMontage combines arbitrary frame-guided DiT conditioning with memory-efficient SAR to move beyond first-last-frame rigidity.Source: §5 · p. 12
  • Adaptive Tuning, Visual Expression SFT, and Tailored DPO form a progressive pipeline aimed at narrative coherence and physical plausibility.Source: §5 · p. 12
  • Evaluations report superior controllability and visual fidelity versus state-of-the-art baselines, alongside a scalable and efficient solution for visual storytelling.Source: §5 · p. 12
Loading 2512.21252v2…