PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative View Stitching

Chonghyuk Song, Michal Stary, Boyuan Chen, George Kopanas, Vincent Sitzmann

arXiv:2510.24718v3cs.CVcs.LG

TL;DR

Video diffusion models have limited context, motivating methods that extend pretrained models beyond their training length. GVS uses parallel stitching with an off-the-shelf Diffusion-Forcing model and Omni Guidance; it generates camera-guided videos described as consistent, faithful to the conditioning trajectory, and stable.

  • Problem

    Video diffusion models have limited context, making methods to extend pretrained models beyond their training context an attractive prospect.

    Source: §1 · p. 1

  • Method

    GVS uses training-free parallel diffusion stitching compatible with off-the-shelf Diffusion-Forcing models, adding Omni Guidance to strengthen past and future conditioning and support loop closing.

    Source: §1 · pp. 1–2

  • Results

    GVS generates camera-guided videos that are consistent, faithful to the conditioning trajectory, and stable.

    Source: §5 · p. 10

  • Takeaways & Limitations

    GVS presents a promising alternative to autoregressive extension for long video generation.

    Source: §5 · p. 10

    Because GVS breaks causality, it cannot be applied online when future camera poses are unavailable, such as in interactive generation or streaming.

    Source: §5 · p. 10

Abstract

from arXiv · show

Autoregressive video diffusion models are capable of long rollouts that are stable and consistent with history, but they are unable to guide the current generation with conditioning from the future. In camera-guided video generation with a predefined camera trajectory, this limitation leads to collisions with the generated scene, after which autoregression quickly collapses. To address this, we propose Generative View Stitching (GVS), which samples the entire sequence in parallel such that the generated scene is faithful to every part of the predefined camera trajectory. Our main contribution is a sampling algorithm that extends prior work on diffusion stitching for robot planning to video generation. While such stitching methods usually require a specially trained model, GVS is compatible with any off-the-shelf video model trained with Diffusion Forcing, a prevalent sequence diffusion framework that we show already provides the affordances necessary for stitching. We then introduce Omni Guidance, a technique that enhances the temporal consistency in stitching by conditioning on both the past and future, and that enables our proposed loop-closing mechanism for delivering long-range coherence. Overall, GVS achieves camera-guided video generation that is stable, collision-free, frame-to-frame consistent, and closes loops for a variety of predefined camera paths, including Oscar Reutersv\"ard's Impossible Staircase. Results are best viewed as videos at https://andrewsonga.github.io/gvs.

1 INTRODUCTION

Existing autoregressive methods extend video beyond short model contexts, but cannot condition current frames on future camera poses, limiting predefined-trajectory generation. GVS instead stitches a sequence in parallel using Diffusion Forcing models, with Omni Guidance supporting temporal and long-range consistency.

1 INTRODUCTION

  • Video models commonly generate only 5–10 seconds, while autoregressive rollouts extend short-horizon models to hundreds of frames and support real-time streaming.Retrieval-based methods can add consistency with respect to the distant past.Source: §1 · p. 1
  • Autoregressive sampling cannot condition current frames on future camera poses, so a predefined trajectory can force the camera through generated walls and cause exposure bias and collapse.The passage identifies one-shot cinematography and autonomous-driving scenario generation as settings where this future conditioning matters.Source: §1 · p. 1
  • Prior diffusion stitching generates sequences in parallel from overlapping segments, but existing approaches lack video temporal consistency or require costly custom training.StochSync lacks the temporal consistency required for video, while CompDiffuser requires a specially trained sequence diffusion model.Source: §1 · p. 2
  • GVS is a training-free stitching method for camera-guided video generation, compatible with off-the-shelf Diffusion Forcing models and designed for stable, collision-free, consistent long-horizon videos.Omni Guidance strengthens conditioning on the past and future, enabling the proposed loop-closing mechanism across predefined camera paths.Source: §1 · p. 2

2 RELATED WORK AND PRELIMINARIES

The section reviews diffusion and long-video approaches, then positions Diffusion Forcing and camera-guided generation. It describes the gap between autoregressive methods that use history and the need to respect future camera poses in offline generation.

2 RELATED WORK AND PRELIMINARIES

  • Diffusion models iteratively corrupt data toward white noise and learn a score function for reversing that process through iterative denoising.The noise schedule is predefined, and the denoising process can inject random noise at each step.Source: §2 · p. 2
  • Long-video approaches retrieve relevant tokens or compress history into hidden states, while test-time stitching can extrapolate models trained with larger contexts without specialized architectures or training paradigms.The section notes that attention layers scale quadratically with token count.Source: §2 · pp. 2–3
  • Diffusion Forcing trains sequence models with independent noise levels per token and permits selective context masking, supporting variable-history conditioning and stable camera-guided autoregressive rollouts.For predefined trajectories, autoregressive sampling can collide with generated content because it does not account for future camera poses.Source: §2 · p. 3
  • Camera-guided video models commonly add pose conditioning to pretrained image or video diffusion models and fine-tune on pose-annotated videos, while long-video extension often uses autoregressive sampling.Online camera control or planning can address collisions, but does not suit offline generation for a predefined trajectory.Source: §2 · p. 3

3 GENERATIVE VIEW STITCHING

GVS extends diffusion stitching to camera-guided video using off-the-shelf Diffusion Forcing models, then adds Omni Guidance and cyclic conditioning to improve temporal consistency and close loops.

3.1 CHALLENGES IN EXTENDING DIFFUSION STITCHING TO VIDEO GENERATION

  • CompDiffuser requires a custom backbone with specialized conditioning, making its use for video infeasible because training such a model would incur unacceptable cost.Source: §3.1 · p. 4

3.2 GENERAL-PURPOSE VIDEO MODELS ALREADY ENABLE STITCHING

  • GVS uses Diffusion Forcing models for training-free stitching, partitioning videos into short chunks and jointly denoising targets with neighboring chunks.Only the denoised target chunks update the stitched sequence; the denoised conditioning chunks are discarded.Source: §3.2 · pp. 4–5

3.3 STOCHASTICITY IS NECESSARY BUT INSUFFICIENT FOR CONSISTENCY √

  • Maximum stochasticity improves temporal consistency but often oversmooths generations, simplifying consistency at the cost of video quality.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 OMNI GUIDANCE ENHANCES CONSISTENCY

  • Omni Guidance steers the joint-distribution score toward the desired conditional distribution by strengthening conditioning on both past and future.It enables partial stochasticity to reduce oversmoothing while maintaining similar consistency.Source: §3.4 · pp. 5–6

3.5 LONG-RANGE CONSISTENCY VIA CYCLIC CONDITIONING

  • Cyclic conditioning alternates temporal and spatial context windows so target chunks receive information from temporal and spatial neighbors during denoising, enabling loop closure.GVS does not achieve global visual return without an explicit loop-closing mechanism, despite its theoretical global receptive field.Source: §3.5 · pp. 6–7

4 EXPERIMENTS

GVS is evaluated against autoregressive sampling and StochSync on camera-guided video generation, with tests of consistency, collision avoidance, stochasticity, and loop closing. Results favor GVS on consistency and collision avoidance while maintaining comparable video quality, and an application demonstrates a continuous loop through an Impossible Staircase variant.

4 EXPERIMENTS

  • The benchmarks use challenging camera trajectories designed to test video length extrapolation, loop closure, and collision avoidance.All methods use the same Diffusion-Forcing video model with an 8-frame context window; consistency, collision avoidance, and video quality are evaluated with specified metrics.Source: §4 · p. 7

4.1 COMPARISON WITH BASELINES

  • GVS outperforms both baselines on temporal consistency (F2FC), long-range consistency (LRC), and collision avoidance (CA), with comparable video quality (IQ, AQ).StochSync's zero collisions come with scene shape-shifting and poor temporal consistency; results average 40 generations.Source: Table 1 · p. 8, §4.1 · pp. 8–9

4.2 ABLATIONS

  • Without Omni Guidance, increasing stochasticity improves temporal consistency but often oversmooths generations and lowers video-quality metrics.Omni Guidance enhances consistency across stochasticity levels, allowing reduced stochasticity to alleviate oversmoothing; at η = 1 on Straight Line, it can hurt consistency.Source: Table 2 · p. 9, §4.2 · p. 9
  • Omni Guidance cannot replace explicit loop closing, but significantly improves long-range consistency when loop closing is activated.Without the explicit mechanism, GVS fails to visually return to the same place, suggesting its effective receptive field is not global.Source: Table 3 · p. 9, §4.2 · p. 9
  • At the default stochasticity level η = 0.9, loop closing oversmooths Panorama 2-loop generations, but reducing stochasticity can alleviate this without compromising consistency.Omni Guidance makes that reduction possible.Source: §4.2 · p. 10

4.3 NEW APPLICATION: THE IMPOSSIBLE STAIRCASE

  • GVS generates a visually continuous loop through an Impossible Staircase variant despite different heights at the conditioning trajectory's endpoints.The showcased navigation video is 120 frames long.Source: §4.3 · p. 10, Figure 7 · p. 10

5 DISCUSSION

GVS supports stable, trajectory-faithful camera-guided videos, but its use and performance are bounded by future-pose availability, backbone capabilities, and manually defined spatial windows.

5 DISCUSSION

  • Because GVS breaks causality, it cannot serve online settings without future camera poses, including interactive video generation and streaming.It also struggles to propagate external-image conditioning, depends on its backbone, fails to loop-close wide-baseline viewpoints absent from training, and uses manually defined spatial windows.Source: §5 · p. 10
  • The authors present GVS as a competitive stitching framework and a promising alternative to autoregressive extension for long video generation.The stated scope is camera-guided generation that is consistent, faithful to the conditioning trajectory, and stable.Source: §5 · p. 10

A EXPERIMENTAL DETAILS

The supplied experimental-details passages identify the GVS algorithm and part of its input-output specification.

A EXPERIMENTAL DETAILS

A.1 IMPLEMENTATION DETAILS

GVS generates stable, collision-free long-horizon videos and uses parallel temporal processing, overlapping context windows, and cyclic conditioning to handle camera trajectories. Its latency advantages come with increased memory use, while staircase loop closure requires adapted camera segments.

A.3 COMPUTE RESOURCES

  • The evaluated backbones share a U-ViT architecture and RealEstate10K training data; the scalable implementation processes context windows one at a time on lower-VRAM GPUs.The three backbones include one Diffusion-Forcing model and two DF-free models; full-sequence diffusion does not support Omni Guidance.Source: §A.1 · p. 15, §A.3 · p. 18

A.1 IMPLEMENTATION DETAILS

  • The method uses overlapping context windows with T_overlap = 4 and T_target chunk = 4: each 8-frame window contains two past-chunk frames, the target chunk, and two future-chunk frames.Sampling uses stochasticity η = 0.9 and guidance scale γ = 1; under this convention, γ = 0 means no guidance, whereas baseline methods use γ = 1.Source: §A.1 · p. 16

A.2 DETAILS ON THE IMPOSSIBLE STAIRCASE (SEC. 4.3)

  • Cyclic conditioning alternates temporal and spatial context windows; for the Impossible Staircase, modified camera segments condition each endpoint on the other.The staircase’s disconnected endpoint camera segments are replaced by a continuous straight line to encourage visual continuity.Source: §A.1 · p. 17, §A.2 · p. 17

B.1 APPLICATION: THE INDEFINITE STAIRCASE

  • 1080 frames along an 18-story staircase were generated stably and without collisions, without closing the trajectory’s endpoints.The result highlights GVS’s long-horizon scaling.Source: §B.1 · p. 18

B.2 INFERENCE COST

  • Inference costs were measured on Straight Line trajectories of 24 to 120 frames using wall time per video and allocated GPU memory.Measurements averaged wall time over 40 samples with batch size B = 1.Source: §B.2 · p. 18
  • At batch size B = T, AR+ has virtually the same average runtime and memory use as GVS, but worse batch-level latency and collisions.GVS parallelizes diffusion windows along the temporal dimension, trading increased memory use for shorter runtimes than AR sampling at the same batch size.Source: §B.2 · p. 19

B.3 ABLATION ON OMNI GUIDANCE AND STOCHASTICITY

Reducing stochasticity can alleviate oversmoothing, and Omni Guidance helps preserve consistency across stochasticity levels. Without Omni Guidance, reducing stochasticity hampers both temporal and long-range consistency.

B.3 ABLATION ON OMNI GUIDANCE AND STOCHASTICITY

  • For Panorama 2-loop, reducing stochasticity from η = 0.9 to η = 0.8 alleviates oversmoothing.The default η = 0.9 produced oversmoothed generations.Source: §B.3 · pp. 19–20
  • Omni Guidance allows stochasticity to be reduced without sacrificing consistency.The reported comparison is between rows 3 and 4 of Table 5.Source: §B.3 · p. 20
  • Without Omni Guidance, reducing stochasticity hampers both frame-to-frame and long-range consistency.The measures named in the passage are F2FC and LRC.Source: §B.3 · p. 20

B.4 ABLATION ON BACKBONE DIFFUSION MODELS

GVS produces stable, collision-free rollouts on DF-free backbones and retains advantages over autoregressive sampling and StochSync on reported comparisons. Omni Guidance is unavailable for the full-sequence backbone, where consistency is slightly worse than on backbones that support it.

B.4 ABLATION ON BACKBONE DIFFUSION MODELS

  • GVS produces stable, collision-free, reasonably consistent long-horizon rollouts on DF-free backbones, but consistency is slightly worse on FS, which lacks Omni Guidance.The BD and FS backbones are the two DF-free models evaluated.Source: §B.4 · p. 20
  • For the FS backbone, AR sampling collapses beyond its first context window, while GVS generates stable, collision-free, reasonably consistent videos.The passage notes that FS does not support conditioning on context frames.Source: §B.4 · p. 20
  • On the BD backbone, Omni Guidance enhances temporal consistency across stochasticity levels, allowing more flexibility to reduce oversmoothing.The BD backbone supports every component of GVS.Source: Table 9 · p. 21, §B.4 · p. 21
  • The results suggest GVS could apply to a wide range of off-the-shelf video diffusion models, regardless of training framework.This is stated as a potential generalization.Source: §B.4 · p. 21

B.5 ABLATION ON SAMPLING TIMESTEPS

Consistency generally improves as sampling timesteps increase from low values, then deteriorates at higher values. The authors use an intermediate setting to balance consistency, scene complexity, and inference costs.

B.5 ABLATION ON SAMPLING TIMESTEPS

  • At low sampling-timestep counts, increasing Ks generally improves frame-to-frame and long-range consistency, but at higher counts consistency deteriorates.The authors attribute the latter trend to increased scene complexity making consistency harder.Source: §B.5 · p. 21
  • Increasing sampling timesteps does not eliminate the need for explicit loop closing.Source: §B.5 · p. 21
  • Ks = 50 is the default, balancing consistency, scene complexity, and inference costs.For GVS, time and space complexity are linear in Ks.Source: §B.5 · p. 21

B.6 ABLATION ON CHUNKING PROFILE

The chunking ablation weighs temporal consistency against scene complexity and inference cost. Additional comparisons report GVS’s qualitative performance and limitations involving online use and external context frames.

B.6 ABLATION ON CHUNKING PROFILE

  • Increasing overlap generally improves temporal consistency but raises scene complexity; Straight Line consistency eventually worsens, so experiments use 4-frame target chunks and 4-frame overlaps.The selected intermediate profile balances consistency, scene complexity, and inference costs.Source: §B.6 · pp. 21–22, Table 12 · p. 22

B.7 ADDITIONAL QUALITATIVE COMPARISONS WITH BASELINES

  • GVS generates stable, temporally consistent, collision-free samples and closes loops, while autoregressive sampling collides and collapses and StochSync shape-shifts the scene.Source: §B.7 · p. 22

C.1 APPLICATION TO ONLINE SETTINGS

  • Because GVS requires future camera poses, it cannot be applied to interactive generation or streaming when those poses are unavailable.Source: §C.1 · p. 22

C.2 EXTERNAL IMAGE CONDITIONING

  • External context frames propagate only through frame τ=4, leaving an awkward transition at τ=6 and τ=8; noise-level modulation is proposed for future investigation.Source: §C.2 · p. 23

C.3 LOOP-CLOSING WIDE-BASELINE VIEWPOINTS

GVS’s loop-closing behavior is constrained by wide-baseline viewpoints, while trajectory-specific spatial-neighbor selection and similar-looking trajectory segments pose further challenges.

C.3 LOOP-CLOSING WIDE-BASELINE VIEWPOINTS

  • Without loop closing, GVS tracks the camera motion but depicts different scenes across the forward and backward segments, including a missing countertop and inconsistent floor color.Source: §C.3 · pp. 23–24, Figure 13 · p. 24
  • With loop closing, both segments depict the same scene from the same viewpoint rather than opposite viewpoints; wide-baseline conditioning is out of distribution for the backbone.Single-window diffusion on the trajectory also fails to track the camera motion; the authors link the limitation to backbone training on small viewpoint shifts.Source: §C.3 · p. 24, Figure 14 · p. 24

C.4 SCALABLE CYCLIC CONDITIONING

  • Cyclic-conditioning spatial neighbors are manually defined per trajectory; selecting them from noisy frames or camera field-of-view overlap is left for future work.Source: §C.4 · p. 24

C.5 STRUCTURALLY SIMILAR CAMERA TRAJECTORY SEGMENTS

  • Limited context and relative-pose conditioning can make structurally similar segments ambiguous, producing ascending steps at the bottom of a downward staircase; context images or text could help resolve this.Source: §C.5 · p. 25
Loading 2510.24718v3…