Source-linked AI summary
SceneScape: Text-Driven Consistent Scene Generation
Rafail Fridman, Amit Abecasis, Yoni Kasten, Tali Dekel
TL;DR
Perpetual view generation requires long-term, plausible videos that remain geometrically and temporally consistent as the camera moves. SceneScape combines pretrained text-image and depth models with online test-time optimization and a progressively unified mesh, enabling diverse text-driven scenes without domain-specific training data. Its quality and 3D consistency improve over existing methods, while results can degrade through error accumulation and triangular meshes struggle with dramatic depth discontinuities.
Problem
Perpetual view generation must preserve feasible 3D geometry, including parallax and occlusion, while producing smooth content beyond the current field of view.
Method
SceneScape combines pretrained text-to-image and monocular depth models with test-time optimization and a progressively constructed unified 3D mesh.
Results
SceneScape demonstrates significant improvement in quality and 3D consistency over existing methods while synthesizing long-term videos of diverse scenes.
Takeaways & Limitations
The framework enables zero-shot scene generation without training data from a specific domain and uses a unified representation for feasible geometry and rendering.
Takeaways & Limitations
Quality depends on pretrained generative and geometric priors, may decrease over time through error accumulation, and triangular meshes struggle with dramatic depth discontinuities.
Abstract
from arXiv · showhide
We present a method for text-driven perpetual view generation -- synthesizing long-term videos of various scenes solely, given an input text prompt describing the scene and camera poses. We introduce a novel framework that generates such videos in an online fashion by combining the generative power of a pre-trained text-to-image model with the geometric priors learned by a pre-trained monocular depth prediction model. To tackle the pivotal challenge of achieving 3D consistency, i.e., synthesizing videos that depict geometrically-plausible scenes, we deploy an online test-time training to encourage the predicted depth map of the current frame to be geometrically consistent with the synthesized scene. The depth maps are used to construct a unified mesh representation of the scene, which is progressively constructed along the video generation process. In contrast to previous works, which are applicable only to limited domains, our method generates diverse scenes, such as walkthroughs in spaceships, caves, or ice castles.
1 Introduction
SceneScape addresses text-driven perpetual view generation by creating long-range videos from free-vocabulary scene descriptions and camera poses. Its zero-shot framework combines pretrained generative and geometric priors with a progressively constructed unified 3D representation.
- Motivation: Perpetual view generation must preserve feasible 3D relationships, including parallax and occlusion, while maintaining smooth temporal content.The task also requires strong priors for unseen regions beyond the current field of view.
- Motivation: Earlier methods target domains such as landscapes or bedrooms and require large-scale domain-specific training, limiting their use.SceneScape instead synthesizes scenes from free-vocabulary text and camera poses without training data.
- Approach: SceneScape generates long-range videos online, one frame at a time, using a text-to-image diffusion model for newly revealed content and a depth model for its geometry.Test-time depth finetuning matches predictions to projected depth from previously synthesized content.
- Contributions: The method synthesizes long-term videos of diverse domains solely from text and a camera trajectory.This contribution is presented as the first text-driven perpetual view generation method.
- Contributions: SceneScape performs zero-shot/test-time scene generation without large-scale training on a specific target domain.Its 3D-consistency contribution is a progressively estimated unified 3D scene representation.
- Results: Ablations show that the complete method improves quality and 3D consistency over existing methods.The introduction reports a significant improvement, while detailed metric results are discussed later.
2 Related Work
Prior work explores perpetual view generation, 3D-consistent synthesis, 3D-aware generation, and text-to-video or text-to-3D methods. SceneScape differs by targeting diverse scenes without domain-specific training and by using a mesh as a tool for video generation rather than room-specific reconstruction.
- Perpetual View Generation: Perpetual view methods synthesize scenes from images and camera motion, including indoor scenes and long-range landscape flythroughs.Earlier work also includes image-retrieval-based construction of 2D landscapes along desired camera paths.
- 3D-Consistent View Synthesis: Many 3D-consistent view-synthesis methods use NeRFs or other representations trained on large datasets from specific domains.Alternative representations include multiplane images, feature-bearing point clouds, and textured meshes.
- 3D-Aware Image Generation: 3D-aware image-generation methods perform well on structured domains but diverse scenes remain challenging and often require additional guidance or heavy supervision.Accurate camera trajectories are reliably available mainly for synthetic data in some approaches.
- Text-to-Video Generation: Text-to-video models learn temporal priors from large-scale video datasets but remain behind image models in quality, resolution, and video length.Related methods also tune or directly leverage 2D text-image models for video editing.
- Text-to-3D Generation: Text-to-3D methods use language-vision models, differentiable renderers, CLIP-based losses, or diffusion-model priors to generate or stylize 3D content.Examples include optimizing NeRFs to render images within a pretrained diffusion model's distribution.
- Concurrent Work: Text2Room similarly generates textual 3D scenes but specifically creates room meshes and tailors the camera, prompts, and rendering to rooms.SceneScape uses a mesh representation as a tool for videos of diverse scenes.
3 Method
SceneScape generates a video online from a text prompt and camera trajectory by combining text-image synthesis, monocular depth estimation, test-time finetuning, and a unified triangle mesh. The mesh preserves previously synthesized content while progressively incorporating newly revealed geometry and handling rendering artifacts.
- Inputs: The input is a text prompt P describing the target scene and a camera trajectory of poses Ci, with the goal of synthesizing a long-term video that follows that trajectory.Each camera pose Ci is represented in R3×4.
- Online Generation: The framework generates one frame at a time using a pretrained text-to-image diffusion model and a pretrained monocular depth model.The diffusion model synthesizes newly revealed content, while the depth model estimates its geometry.
- Motivation: Frame-to-frame warping is insufficient because multiple source pixels can map to one target pixel and occluded content is not retained in a unified representation.These issues make color aggregation and future rendering difficult.
- Scene Representation: The method initializes a unified triangle mesh by unprojecting the first generated image and its predicted depth.The mesh stores vertices with 3D location and color, along with faces and edges.
- Project and Inpaint: At each step, the current mesh is projected into the next camera view to produce visible-content masks, masked images, and masked depth maps before inpainting newly revealed regions.The inpainting model uses the projected mesh content and text prompt to synthesize the next frame.
- Depth Consistency: Test-time finetuning aligns monocular depth predictions with existing scene geometry because depth estimates can be inconsistent across nearby frames.The depth model is reverted to its original weights at each generation step to avoid catastrophic forgetting.
- Online Mesh Update: New content is unprojected into 3D and merged into the existing mesh, while previously synthesized content remains even when occluded.The update also adds connecting triangles between old and new content.
- Rendering: The renderer removes stretched triangles near depth discontinuities and masks border regions that can otherwise produce floating artifacts under parallax.These operations target unseen content and undesirable reuse of distant geometry.
4 Results
The method is evaluated on diverse 50-frame scenes using COLMAP-based 3D-consistency metrics, camera-pose metrics, visual-quality measures, and prompt similarity. Ablations and baseline comparisons show that its components support high-quality, geometrically consistent video generation, while outdoor scenes remain constrained by dramatic depth discontinuities.
- Evaluation setup: The evaluation uses diverse photorealistic and imaginary scenes, generating 50-frame videos with fast backward camera motion and rotations.Because the scene is represented as a mesh, experiments focus mainly on indoor scenes; outdoor scenes require careful handling of dramatic depth discontinuities.
- Qualitative results: The full method produces high-quality, geometrically plausible scenes with significant parallax, complex structures, varied lighting, and diverse materials.Sample videos include scenes such as ice environments, and additional videos and depth maps are provided in the supplementary materials.
- Metrics: Depth consistency is measured with COLMAP using SI-RMSE, recovered-pixel percentage, and reprojection error, alongside camera-pose accuracy and visual-quality metrics.Visual quality combines AMT two-alternative forced-choice judgments with CLIP aesthetic scores.
- Ablation study: Removing depth finetuning worsens depth consistency, removing decoder finetuning increases reprojection error through high-frequency temporal inconsistency, and removing the mesh degrades all metrics.The no-mesh variant yields substantially sparser reconstructions.
- Ablation study: The unified mesh plays a major role in visual quality, while judgments against the no-depth-finetuning baseline are ambiguous because its artifacts remain temporally consistent.The AMT evaluation collected 4,000 judgments from 200 participants over 20 videos.
- Baseline comparison: Against VideoFusion and GEN-1, the method outperforms both on all reported 3D-consistency metrics and is preferred in 96% and 69% of human comparisons, respectively.Prompt similarity is higher than VideoFusion and on par with GEN-1; the baseline experiments use different setups.
5 Discussion and Conclusion
The method generates diverse long-term videos through test-time optimization, combining pretrained generative and geometric priors with a progressively updated unified 3D representation. Its quality depends on these priors and can decline through error accumulation, while triangular meshes struggle with dramatic depth discontinuities.
- The framework generates long-term videos of diverse scenes using test-time optimization rather than domain-specific training data.It combines two pretrained models with a unified 3D scene representation.
- A unified 3D representation provides feasible geometry and supports high-quality, efficient rendering.
- Result quality depends on the generative and geometric priors and may decrease over time because of error accumulation.
- The triangular-mesh representation has difficulty modeling dramatic depth discontinuities, such as sky versus ground in outdoor scenes.
A Implementation Details
The implementation uses pretrained diffusion and depth models with per-frame optimization, explicit camera-path settings, and mask preprocessing. A 50-frame video takes approximately 2.5 hours on an NVIDIA TeslaV100 GPU.
- The framework uses Stable Diffusion with DDIM sampling and takes approximately 2.5 hours to synthesize a 50-frame video on an NVIDIA TeslaV100 GPU.Each generated frame uses 50 sampling steps.
- Table 3 reports runtime per frame, with rendering including antialiasing and floating-artifact correction.
- The depth model is finetuned for 300 epochs per frame, while the LDM decoder is finetuned for 100 epochs at each generation step.MiDaS-DPT Large is used for depth prediction; Adam learning rates are 1e-7 and 1e-4, respectively.
- The camera combines translation in the negative depth direction with periodically resampled rotational motion in the x-z plane.The experiments use k = 5 and n = 5, and PyTorch3D renders and updates the unified representation.
- Morphological opening regularizes the inpainting mask before Telea fills the mask difference and the diffusion model inpaints the opened mask.The opening uses a kernel size of 3.
A.1 Mesh update.
Each newly generated image is unprojected into colored 3D vertices using its depth map and camera pose, then triangulated and connected to the existing mesh to maintain a unified scene representation.
- The mesh update unprojects selected pixels from the new image into 3D vertices using their depths, camera pose, and pixel colors.
- Each four-neighbor pixel group defines two adjacent triangle faces in the newly unprojected mesh.
- Additional triangles connect the existing mesh to the newly unprojected part, preventing holes in the combined mesh.
- For projected existing faces, the method selects the triangle vertex closest to the current camera center before adding it to the triangulation scheme.This creates the triangles connecting the old and newly unprojected mesh parts.
A.2 Rendering
Rendering addresses mesh-induced artifacts by antialiasing high-resolution rasterization, removing stretched triangles near depth discontinuities, and expanding border depth information to detect floating artifacts.
- Antialiasing renders the mesh at twice the target resolution, applies Gaussian blur, and resizes it to reduce rasterization artifacts.The artifacts arise when high-resolution mesh content is rasterized into relatively few pixels.
- Stretched triangles are detected near depth discontinuities and filtered using triangle normals before rendering.The depth map is processed with a Sobel filter, and the practical normal threshold is ϵ = -0.05.
- The stretched-triangle filter retains triangles satisfying the camera-relative normal condition (center(tri) − c)^T n < ϵ.Here, c is the camera center and n is a triangle normal.
- To detect floating artifacts, the previous depth map is padded with border depth values to 1.5 times the rendering resolution before warping to the next camera.The resulting mask identifies border content that should be inpainted.
B Baseline Comparison Details
The baseline comparisons adapt each method to camera-controlled indoor video generation, using curated trajectories and prompt settings suited to each baseline. SceneScape is also compared with StableDreamFusion using the same camera trajectory and text prompts.
- VideoFusion: VideoFusion is evaluated on 1,000 16-frame videos at 256 resolution, with prompts augmented to encourage backward camera motion.SceneScape videos are downsampled to 256 resolution for comparison.
- StableDreamFusion: The StableDreamFusion comparison uses diverse indoor prompts, including candy houses, haunted houses, hotels, libraries, castles, museums, spaceships, palaces, caves, and ice castles.The listed prompts include both conventional interiors and imaginative environments.
- GEN-1: GEN-1 is compared using 22 filtered RealEstate10K indoor videos with smooth backward camera motion and 25 subsampled frames.The filtering enforces camera motion aligned with the viewing direction and reverses videos with positive average z displacement.
- GEN-1: For GEN-1, depth ranges are aligned by scaling MiDaS predictions with the ratio of median COLMAP and MiDaS depths.The scaling factor is computed across the video frames.
- StableDreamFusion: StableDreamFusion receives SceneScape’s camera trajectory and simple indoor-scene prompts, but its generated scenes contain blur and unrealistic artifacts.Usual prompt sets were not used because StableDreamFusion failed to converge to meaningful results with them.
C Broader impact
The framework avoids training data but relies on pretrained depth and diffusion models whose biases may affect generated static 3D scenes. The authors therefore restrict prompts to general scenery and objects, avoiding sensitive content such as humans.
- Broader impact: The method requires no training data but inherits potential biases from its pretrained depth prediction and text-to-image diffusion models.The authors discuss these inherited biases in the context of generating static 3D scenes.
- Broader impact: To avoid harmful content, the authors use prompts describing general scenery and objects while excluding sensitive content such as humans.This is an explicit prompt-selection boundary of the framework.