PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models

Geonung Kim, Janghyeok Han, Sunghyun Cho

arXiv:2509.17985v1cs.GR

TL;DR

VideoFrom3D addresses the costly iteration required by detailed 3D scene production and the difficulty of generating high-fidelity complex-scene videos with video diffusion alone. It combines image-diffusion anchor generation with video-diffusion inbetweening, and reports high-quality, style-consistent videos across diverse scenarios that outperform simple and extended baselines. The approach does not support real-time interactive camera control, and temporal inconsistency can arise from diffusion randomness.

  • Problem

    Existing video diffusion models struggle with complex scenes because they must jointly model visual quality, motion, and temporal consistency.

    Source: §1 · p. 2

  • Method

    VideoFrom3D uses SAG to generate cross-view-consistent anchor views with image diffusion and GGI to interpolate intermediate frames with video diffusion, flow-based camera control, and structural guidance.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 5

  • Results

    The method produces high-fidelity, style-consistent scene videos under diverse and challenging scenarios, outperforming naïve and extended baselines.

    Source: §1 · p. 3, §4.1 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The framework supports rapid design exploration from coarse geometry and can provide generated videos for early-stage development or visualization-only deliverables.

    Source: §1 · p. 2

    VideoFrom3D lacks real-time interactive camera control, may exhibit temporal inconsistency from diffusion randomness, and requires computationally expensive LoRA training.

    Source: §5 · p. 10

Abstract

from arXiv · show

In this paper, we propose VideoFrom3D, a novel framework for synthesizing high-quality 3D scene videos from coarse geometry, a camera trajectory, and a reference image. Our approach streamlines the 3D graphic design workflow, enabling flexible design exploration and rapid production of deliverables. A straightforward approach to synthesizing a video from coarse geometry might condition a video diffusion model on geometric structure. However, existing video diffusion models struggle to generate high-fidelity results for complex scenes due to the difficulty of jointly modeling visual quality, motion, and temporal consistency. To address this, we propose a generative framework that leverages the complementary strengths of image and video diffusion models. Specifically, our framework consists of a Sparse Anchor-view Generation (SAG) and a Geometry-guided Generative Inbetweening (GGI) module. The SAG module generates high-quality, cross-view consistent anchor views using an image diffusion model, aided by Sparse Appearance-guided Sampling. Building on these anchor views, GGI module faithfully interpolates intermediate frames using a video diffusion model, enhanced by flow-based camera control and structural guidance. Notably, both modules operate without any paired dataset of 3D scene models and natural images, which is extremely difficult to obtain. Comprehensive experiments show that our method produces high-quality, style-consistent scene videos under diverse and challenging scenarios, outperforming simple and extended baselines.

1 Introduction

VideoFrom3D targets costly, iterative 3D design by generating scene videos from coarse geometry, camera trajectories, and reference images. It combines image diffusion for detailed, consistent anchor views with video diffusion for coherent interpolation, without paired 3D-natural-image data.

1 Introduction

  • VideoFrom3D synthesizes high-quality scene videos from coarse geometry, camera trajectories, and reference images, supporting rapid design iteration and visualization.The framework reduces dependence on detailed modeling, texturing, and lighting during early exploration.Source: §1 · p. 2
  • Video diffusion models struggle with complex scenes because they must jointly synthesize frame quality, realistic motion, and temporal coherence.Image diffusion produced realistic building details in the cited comparison, whereas video models generated distorted structures despite larger parameter counts.Source: §1 · p. 2
  • The two-stage framework generates multi-view-consistent anchor views with image diffusion, then interpolates temporally coherent frames with video diffusion.SAG uses Sparse Appearance-guided Sampling, while GGI adds flow-based camera control and structural guidance.Source: §1 · p. 2, Figure 3 · p. 3
  • Texturing-based alternatives require detailed geometry and cannot represent dynamic, view-dependent effects such as reflections, flames, or flowing streams.VideoFrom3D instead directly generates video, but does not support real-time navigation or pixel-level cross-view consistency.Source: §1 · pp. 2–3
  • The method avoids paired datasets of 3D scene models and natural images while producing high-fidelity, style-consistent videos under diverse and challenging scenarios.The authors report performance surpassing naïve and extended baselines.Source: §1 · pp. 2–3

2 Related Work

Related work includes geometry-conditioned video and 3D generation methods, as well as few-shot reconstruction approaches. VideoFrom3D is positioned as a scene-scale approach intended to improve visual quality and interpolation robustness.

Geometry-guided Video Generation

  • Geometry-conditioned video methods often produce poor visual quality in complex scenes, while prior 3D generation methods are limited by object-level scope or restricted scene layouts.The cited scene-scale methods are described as blurry or constrained to rectangular rooms.Source: §2 · p. 3

Few-shot 3D Reconstruction

  • Few-shot reconstruction methods can fail under wide keyframe baselines or complex transitions, whereas GGI uses video diffusion and structure-aware conditioning for accurate, smooth interpolation.The comparison also identifies scale ambiguity and temporal flickering in an alternative multi-view diffusion approach.Source: §2 · p. 3

3 Method

VideoFrom3D combines image diffusion for style-consistent anchor views with video diffusion for geometry-guided temporal interpolation. Its pipeline uses structural edges, optical flow, sparse appearance guidance, and video training surrogates to synthesize coherent scene videos.

3 Method

  • VideoFrom3D defines video synthesis from a mesh, camera trajectory, and reference style as a sequence of geometry-faithful, consistently stylized views.Source: §3 · p. 3

3.1 Preprocessing

  • The preprocessing stage extracts projected edge maps for structural guidance and optical flows for cross-view consistency and camera-trajectory control.Edge maps use silhouettes, creases, object boundaries, and intersections; the flow is computed from coordinate-encoded mesh correspondences.Source: §3 · pp. 3–4, §3.1 · p. 4

3.2 Sparse Anchor-view Generation (SAG)

  • SAG uses an image diffusion prior, ControlNet edges, reference-style distribution alignment, and sparse appearance-guided latent replacement to generate consistent anchor views.The warped start view supplies semantic and appearance information in observed regions, while early-timestep replacement avoids transferring distorted details; 25 diffusion steps use replacement for the first 12.Source: §3.2 · pp. 4–5

3.3 Geometry-guided Generative Inbetweening (GGI)

  • GGI conditions a video diffusion model on endpoint anchors, flow-warped noise, and encoded edge guidance to synthesize temporally coherent intermediate frames along the camera trajectory.Warped noise provides approximate motion guidance, while structural edges address its limited granularity and implicit flow encoding.Source: §3.3 · p. 5

3.4 GGI Module Training

  • GGI training uses static-scene videos, optical flow, and depth-derived HED edges to approximate unavailable paired 3D-model and natural-image data while reducing the training–inference domain gap.The synthesized edges omit appearance and emphasize strong structural contours, matching the intended inference-time guidance.Source: §3.4 · pp. 5, 7

4 Experiments

Experiments evaluate VideoFrom3D across diverse scenes, styles, baselines, and component variants. The results support the framework’s visual quality, structural fidelity, cross-view consistency, and temporal coherence, while also exposing latency and remaining structural-condition trade-offs.

4.1 Video Generation Results

  • The method handles object-centric scenes, large camera motions, indoor–outdoor transitions, and artistic styles, including animation and painting-like outputs.A separate example demonstrates smooth transitions between distinct seasonal styles over time.Source: §4.1 · p. 7

4.2 Baseline Comparisons

  • Compared with Depth-I2V, VACE, MVSplat360, LVSM, and SEVA, VideoFrom3D achieves higher visual quality and structural fidelity under challenging conditions.The baselines exhibit insufficient detail, reconstruction artifacts, blurry regions, or failures under challenging trajectories.Source: §4.2 · pp. 8–9, Figure 10 · p. 8
  • VideoFrom3D achieves the best performance across most metrics and ranks second-best in a few benchmark comparisons.The evaluation reports visual fidelity, structural fidelity, visual quality, style similarity, and consistency metrics over 48 generated videos from 16 models.Source: Table 2 · p. 8, §4.2 · p. 9

4.3 Ablation & Analysis

  • Sparse Appearance-guided Sampling preserves anchor-view details such as roofs, windows, and facade patterns that otherwise deviate from the start frame.The comparison evaluates corresponding regions of the input geometry with and without guided sampling.Source: §4.3 · p. 9, Figure 11 · p. 9
  • HED-S structural conditioning preserves structure and avoids detail loss better than no condition or direct RGB-derived HED edges in GGI.Without structural conditioning, distortions occur; direct HED edges lose details, while the simulated condition agrees with the quantitative comparison in Table 3.Source: §4.3 · p. 9, Table 3 · p. 9, Figure 12 · p. 9
  • Generating dense views with SAG alone causes severe flickering and temporal inconsistency, demonstrating the necessity of GGI for consistent video synthesis.The comparison uses temporal profiles of a fixed 160×20 pixel region along the camera trajectory.Source: §4.3 · p. 9, Figure 13 · p. 9
  • HED edges generalize better than Canny edges or depth maps for guiding both coarse and detailed geometry in anchor-view generation.Canny edges produce monotonous results for coarse geometry, while depth guidance tends to ignore weak signals in detailed geometry.Source: §4.3 · p. 10, Figure 14 · p. 10
  • 197 seconds are required to generate a single trajectory after LoRA training on an A100-80GB GPU.The latency is reported for the complete trajectory-generation process.Source: §4.3 · p. 10, Table 4 · p. 10

5 Conclusion

VideoFrom3D combines image and video diffusion through SAG and GGI to synthesize style-consistent, natural, and geometrically faithful scene videos. The method remains limited by non-real-time camera control, possible temporal inconsistency, and the computational cost of LoRA training.

5 Conclusion

  • VideoFrom3D synthesizes high-quality 3D scene videos from coarse geometry, camera trajectories, and reference images by combining SAG and GGI.The framework leverages image diffusion for anchor views and video diffusion for interpolation.Source: §5 · p. 10
  • The method demonstrates effectiveness across diverse and challenging scenarios while producing style-consistent, natural, and geometrically faithful videos.These conclusions are based on extensive evaluations.Source: §5 · p. 10
  • The framework does not support real-time interactive camera control, may exhibit temporal inconsistency, and requires computationally costly LoRA training.These limitations are identified as future directions for improvement.Source: §5 · p. 10

Related papers

Loading 2509.17985v1…