PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Scene Prompting for Scene-Consistent Camera-Controllable Video Generation

JoungBin Lee, Jaewoo Jung, Jisang Han, Takuya Narihira, Kazumi Fukuda, Junyoung Seo, Sunghwan Hong, Yuki Mitsufuji, Seungryong Kim

arXiv:2510.14945v2cs.CV

TL;DR

Camera-controllable video generation from long inputs must preserve scene context and motion while respecting practical computational limits. 3DScenePrompt combines temporal and spatial conditioning with a static-only 3D memory, and reports stronger scene consistency, camera control, and generation quality than existing methods.

  • Problem

    Existing approaches use very short conditioning sequences, limiting their ability to use longer-video scene context while jointly supporting camera control and natural motion.

    Source: §1 · p. 2

  • Method

    3DScenePrompt combines recent temporal frames with spatially relevant context from a static-only 3D scene memory built using dynamic SLAM and dynamic masking.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Across scene consistency, camera controllability, and generation quality, 3DScenePrompt significantly outperforms existing methods; against DFoT, MEt3R evaluation error drops 77% (0.041 vs 0.181).

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    The framework supports arbitrary-length continuations that preserve scene geometry while allowing natural motion evolution.

    Source: §5 · p. 10

    MegaSAM point-cloud processing adds inference-time overhead, and generated-video fidelity depends on dynamic-mask quality.

    Source: D LIMITATION · p. 18

Abstract

from arXiv · show

We present 3DScenePrompt, a framework that generates the next video chunk from arbitrary-length input while enabling precise camera control and preserving scene consistency. Unlike methods conditioned on a single image or a short clip, we employ dual spatio-temporal conditioning that reformulates context-view referencing across the input video. Our approach conditions on both temporally adjacent frames for motion continuity and spatially adjacent content for scene consistency. However, when generating beyond temporal boundaries, directly using spatially adjacent frames would incorrectly preserve dynamic elements from the past. We address this by introducing a 3D scene memory that represents exclusively the static geometry extracted from the entire input video. To construct this memory, we leverage dynamic SLAM with our newly introduced dynamic masking strategy that explicitly separates static scene geometry from moving elements. The static scene representation can then be projected to any target viewpoint, providing geometrically consistent warped views that serve as strong 3D spatial prompts while allowing dynamic regions to evolve naturally from temporal context. This enables our model to maintain long-range spatial coherence and precise camera control without sacrificing computational efficiency or motion realism. Extensive experiments demonstrate that our framework significantly outperforms existing methods in scene consistency, camera controllability, and generation quality. Project page : https://cvlab-kaist.github.io/3DScenePrompt/

1 INTRODUCTION

3DScenePrompt addresses the challenge of generating camera-controlled continuations from long videos while preserving scene context, natural motion, and practical computation. It combines temporal and spatial conditioning with a static-only 3D scene memory, and the paper reports improved scene consistency, camera control, and generation quality over existing methods.

  • The task requires static geometry to persist while dynamic elements evolve from recent states, with 3D understanding and practical computation.Source: §1 · p. 2
  • The framework combines temporally adjacent frames for motion continuity with spatially adjacent frames for scene consistency, unlike single-frame or temporal-only conditioning.Source: §1 · p. 2, Figure 2 · p. 3
  • 3DScenePrompt constructs a scene memory containing only static geometry so spatial prompts do not preserve dynamic content from the past.Source: §1 · p. 2
  • Dynamic SLAM and dynamic masking extract static regions for projection into geometrically consistent views, supporting precise camera control without explicit camera conditioning.Source: §1 · p. 2
  • 3DScenePrompt significantly outperforms existing methods in scene consistency, camera controllability, and video quality for arbitrary-length inputs.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses camera control, temporal continuity, and geometric consistency in different settings, but the paper distinguishes its goal of extending arbitrary-length dynamic videos with efficient long-range spatial coherence. 3DScenePrompt uses dual spatio-temporal conditioning and SLAM-based spatial memory to address that gap.

Single-frame conditioned camera-controllable video generation

  • Single-frame camera-control methods follow trajectories but lack mechanisms to preserve consistency with extended video context.Source: §2 · p. 3

Multi-frame conditioned camera-controllable video generation

  • Multi-frame camera-control methods improve temporal smoothness but consider temporal adjacency, limiting long-video scene consistency under memory constraints.Source: §2 · p. 3

Geometry-grounded video generation

  • Geometry-grounded methods such as Gen3C address dynamic novel-view synthesis within the input time window, whereas future generation requires masking dynamics during 3D construction.Source: §2 · p. 3
  • The framework overview shows recent frames supplying temporal conditioning while representative static regions are lifted to 3D and rendered along the requested camera trajectory.Source: Figure 3 · p. 4

Long-horizon scene-consistent generation

  • Long-horizon approaches either remain within input coverage, assume static worlds, or face memory constraints; dual spatio-temporal SLAM memory targets efficient arbitrary-length generation.Source: §2 · p. 4

3 METHODOLOGY

3DScenePrompt addresses the tension between preserving distant scene geometry and evolving dynamic content naturally. It combines recent temporal context with spatial prompts projected from a static 3D scene memory.

3.1 PROBLEM FORMULATION AND MOTIVATION

  • The task is to generate future frames following a desired camera trajectory while remaining consistent with the scene in an arbitrary-length dynamic video.A short temporal window can fail to preserve long-term spatial consistency when the camera revisits regions outside that window.Source: §3.1 · pp. 4–5, §3.2 · p. 5

3.2 TOWARDS SCENE-CONSISTENT CAMERA-CONTROLLABLE VIDEO GENERATION

  • Temporal and spatial windows jointly condition generation: recent frames support motion continuity, while viewpoint-similar frames support scene consistency without processing the full input sequence.The framework replaces direct use of past spatially adjacent frames with a memory of static geometry, avoiding preservation of past dynamic elements.Source: §3.2 · p. 5

3.3 3D SCENE MEMORY CONSTRUCTION

  • Dynamic SLAM estimates camera poses and reconstructs scene structure, while dynamic masking removes moving content to produce static-only geometry.Pixel-level motion detection can miss full object boundaries or objects that begin moving later; backward tracking aggregates motion across the sequence to guide object-level masks.Source: §3.3 · pp. 5–6

3.4 3D SCENE PROMPTING

  • Static geometry projected from spatially relevant input frames forms target-aligned spatial prompts that guide persistent scene structure and precise camera control.The point cloud aggregates multiple viewpoints and may fill regions occluded by dynamic objects.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

Experiments assess scene and geometric consistency, camera controllability, and video quality, alongside ablations of dynamic masking and spatial context. 3DScenePrompt outperforms the compared methods across the reported evaluations.

4.1 IMPLEMENTATION DETAILS

  • The model extends CogVideoX-I2V-5B to concatenate temporal frames and static-point-cloud projections as conditioning, while leaving the DiT backbone unchanged.Training uses 4K iterations; the implementation uses nine temporal frames and seven spatial views.Source: §4.1 · p. 7

4.2 SCENE-CONSISTENT VIDEO GENERATION

  • 77% lower MEt3R evaluation error (0.041 vs 0.181), with 3DScenePrompt outperforming DFoT across spatial and geometric consistency metrics.The evaluation covers RealEstate10K and DynPose-100K.Source: Table 1 · p. 8, §4.2 · p. 8

4.1 IMPLEMENTATION DETAILS

  • Evaluation uses 1,000 DynPose-100K dynamic videos and, for scene consistency, 1,000 RealEstate10K static videos, with DFoT as the scene-consistency baseline.Camera control and video-quality comparisons also include camera-controllable methods.Source: §4.1 · pp. 7–8

4.2 SCENE-CONSISTENT VIDEO GENERATION

  • Scene consistency is evaluated by matching generated and input frames at revisited viewpoints, using PSNR, SSIM, and LPIPS, plus Met3R for geometric consistency.RealEstate10K uses whole-image evaluation, while DynPose-100K evaluation masks dynamic regions.Source: §4.2 · p. 8

4.3 CAMERA-CONTROLLABLE VIDEO GENERATION

  • 3DScenePrompt consistently outperforms camera-control baselines and achieves the best dynamic-video generation quality across all reported metrics.The camera-control comparison varies temporal conditioning windows; their effect on camera-controllability performance is minimal.Source: Table 2 · p. 8, Table 3 · p. 9, §4.3 · p. 9

4.4 ABLATION STUDIES

  • Without dynamic masking, PSNR drops by approximately 0.8 dB and MEt3R error increases; performance stabilizes around seven spatially adjacent frames.Unmasked moving objects create ghosting artifacts when warped to new viewpoints; more than seven retrieved frames bring minimal improvement.Source: §4.4 · pp. 9–10, Table 4 · p. 9

5 CONCLUSION

The conclusion presents 3DScenePrompt as a framework for generating scene-consistent, camera-controllable video from arbitrary-length inputs.

5 CONCLUSION

  • 3DScenePrompt combines dual spatio-temporal conditioning with a static-only 3D scene memory built using dynamic SLAM and dynamic masking.The framework aims to preserve scene geometry while allowing natural motion evolution.Source: §5 · p. 10
  • The framework supports continuations from arbitrary-length videos while preserving scene geometry and allowing motion to evolve naturally.Source: §5 · p. 10
  • Experiments show superior camera controllability, scene consistency, and generation quality compared with existing methods.Source: §5 · p. 10

NOTATION SUMMARY

The notation summary defines the input and generated videos, camera controls, conditioning frames, and components used to construct the 3D scene memory.

NOTATION SUMMARY

  • Vin denotes the arbitrary-length input video, Vout the generated future video, and T the text prompt used by text-to-video models.Source: NOTATION SUMMARY · p. 14
  • C is the desired camera trajectory, with camera extrinsics Ct in SE(3); Temporal(w) and Spatial(T) denote temporally and spatially adjacent conditioning frames.Source: NOTATION SUMMARY · p. 14
  • The 3D scene memory M comprises camera extrinsics and static point clouds; M_i^obj denotes object-level masks for dynamic regions.Pstatic denotes aggregated point clouds from static regions only.Source: NOTATION SUMMARY · p. 14

A ADDITIONAL EXPERIMENTAL RESULTS

The supplementary experiments examine long-video generation, dynamic-mask and conditioning ablations, memory-backbone flexibility, and inference cost. They report stronger long-video results than previous baselines, improved outcomes with projected spatial views, and lower latency when MegaSAM is replaced with DepthAnything v3.

Long-Video Generation Results

  • Long videos are generated by iteratively conditioning each next chunk on the prior generated video's last frame.The evaluation also compares autoregressive generation, latent interpolation with autoregressive generation, and training-free long-video techniques.Source: A ADDITIONAL EXPERIMENTAL RESULTS · p. 14
  • The method significantly outperforms previous baselines on long-video generation, while extremely long durations still risk temporal error accumulation.The evaluation uses DAVIS and assesses video quality and camera controllability; the authors note that dedicated refinement may be needed to address long-term drift.Source: A ADDITIONAL EXPERIMENTAL RESULTS · pp. 14–15, Table 5 · p. 15

Ablation Study on Dynamic Masking Pipeline

  • The dynamic-mask ablation finds each pipeline stage necessary, progressing from noisy pixel-level flow masks to point tracking with SAM2 propagation across the clip.The stages address moving-object interference, fragmented flow estimates, and newly appearing or displaced objects.Source: A ADDITIONAL EXPERIMENTAL RESULTS · p. 15, Table 6 · p. 15

Ablation study on the number of temporal condition images

  • A temporal window of w = 9 improves motion smoothness and temporal coherence over w = 1 or w = 5, while camera-controllability metrics change negligibly.For 49-frame CogVideoX generation, w = 9 conditions on the last frames and generates 40 new frames.Source: A ADDITIONAL EXPERIMENTAL RESULTS · p. 15, Table 7 · p. 16

Ablation study on the number of projected images

  • Projected spatial views (n = 4 or n = 7) outperform the n = 0 temporal-only baseline across all metrics, which struggles with scene consistency and camera-trajectory adherence.The ablation evaluates scene consistency and camera controllability on DynPose-100K.Source: A ADDITIONAL EXPERIMENTAL RESULTS · p. 16, Table 8 · p. 16

DepthAnything v3 for 3D memory

  • Replacing MegaSAM with DepthAnything v3 improves metrics and reduces inference time without changing the overall pipeline.The authors attribute the improvement to stronger priors and note that the memory component can be swapped at inference time.Source: A ADDITIONAL EXPERIMENTAL RESULTS · p. 16, Table 9 · p. 17

Inference Time

  • With DepthAnything v3, total inference time is approximately 5 minutes and comparable to CameraCtrl and MotionCtrl, versus over 9 minutes with MegaSAM.The SLAM stage falls from approximately 4 minutes with MegaSAM to roughly 10 seconds with DepthAnything v3; diffusion then dominates runtime.Source: A ADDITIONAL EXPERIMENTAL RESULTS · p. 16, Table 10 · p. 17

B TRAINING DATASET CURATION PIPELINE.

The training corpus combines static indoor scenes from RealEstate10K with dynamic videos from OpenVid-1M, using different geometry and masking procedures for each source.

Training dataset curation pipeline

  • The curated dataset contains approximately 30K RealEstate10K videos and 20K OpenVid-1M videos, each at least 100 frames long.Static scenes use VGGT for 3D memory; dynamic scenes use MegaSAM after filtering and dynamic masking.Source: B TRAINING DATASET CURATION PIPELINE · p. 17

C DISCUSSION AND COMPARISON WITH RELATED WORKS

The discussion distinguishes 3DScenePrompt from prior work through explicit static 3D memory, dynamic-object handling, and spatial prompts that guide generation without hard inpainting constraints.

Explicit 3D Memory and World Models

  • Unlike implicit-memory or static-scene approaches, 3DScenePrompt uses explicit 3D memory for real-world dynamic videos and projects static point clouds to target trajectories without increasing computational costs.The discussion contrasts Persistent embodied world model and WorldMem, including WorldMem's Minecraft training focus and key-frame conditioning.Source: C DISCUSSION AND COMPARISON WITH… · p. 17

Discussion and comparison with related works

  • Compared with SPMem, the method uses dynamic masking rather than naïve TSDF fusion and injects conditioning into zero-padding slots without adding a ControlNet-style architecture.A direct performance and efficiency comparison is unavailable because SPMem's public code is unavailable.Source: C DISCUSSION AND COMPARISON WITH… · pp. 17–18

Dynamic Object Removal and Inpainting

  • Unlike hard-constrained video inpainting, spatial prompts act as soft geometric guides, letting diffusion fill holes, refine details, and generate dynamic details over the condition.The prompt preserves geometric layout where projections are reliable while allowing temporal dynamics.Source: C DISCUSSION AND COMPARISON WITH… · p. 18

D LIMITATION

The framework's point-cloud reconstruction adds computational overhead, and generated-video fidelity depends directly on dynamic-mask quality.

Limitation

  • MegaSAM adds inference overhead, while dynamic-mask quality directly affects video fidelity; stronger reconstruction or masking modules may improve the framework.The authors identify DepthAnything3 as an example of a module that could reduce overhead.Source: D LIMITATION · p. 18

Related papers

Loading 2510.14945v2…