PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Camera-Controlled Video Generation of Static Scenes via Sparse Diffusion and 3D Rendering

Jieying Chen, Jeffrey Hu, Joan Lasenby, Ayush Tewari

arXiv:2601.09697v1cs.CV

TL;DR

Diffusion-based video generation is computationally expensive because it synthesizes every frame, limiting real-time applications. SRENDER generates sparse, adaptively selected keyframes, reconstructs a static scene in 3D, and renders the dense video, achieving more than 40× speed-up while maintaining visual quality and temporal consistency.

  • Problem

    Existing camera-conditioned video methods generate every frame with neural networks, making realistic video synthesis computationally expensive and limiting real-time applications.

  • Method

    SRENDER adaptively predicts a keyframe budget, generates sparse camera-conditioned keyframes with diffusion, reconstructs a 3D scene, and renders the dense video.

  • Results

    More than 40× speed-up on DL3DV and more than 20× on RealEstate10k are achieved while maintaining comparable visual quality and temporal consistency.

  • Takeaways & Limitations

    Explicit 3D reconstruction and adaptive sparse generation reduce video-synthesis cost while preserving geometric consistency and visual fidelity.

  • Takeaways & Limitations

    On challenging DL3DV trajectories, generated keyframes can drift beyond roughly 10 seconds, requiring fixed-length temporal chunks for reconstruction.

Abstract

from arXiv · show

Modern video generative models based on diffusion models can produce very realistic clips, but they are computationally inefficient, often requiring minutes of GPU time for just a few seconds of video. This inefficiency poses a critical barrier to deploying generative video in applications that require real-time interactions, such as embodied AI and VR/AR. This paper explores a new strategy for camera-conditioned video generation of static scenes: using diffusion-based generative models to generate a sparse set of keyframes, and then synthesizing the full video through 3D reconstruction and rendering. By lifting keyframes into a 3D representation and rendering intermediate views, our approach amortizes the generation cost across hundreds of frames while enforcing geometric consistency. We further introduce a model that predicts the optimal number of keyframes for a given camera trajectory, allowing the system to adaptively allocate computation. Our final method, SRENDER, uses very sparse keyframes for simple trajectories and denser ones for complex camera motion. This results in video generation that is more than 40 times faster than the diffusion-based baseline in generating 20 seconds of video, while maintaining high visual fidelity and temporal stability, offering a practical path toward efficient and controllable video synthesis.

1. Introduction

SRENDER addresses the computational cost of camera-conditioned video generation by exploiting redundancy in static scenes: diffusion generates sparse keyframes, while 3D reconstruction and rendering produce the dense video. An adaptive predictor varies keyframe density with camera complexity, achieving major speedups while maintaining visual fidelity and temporal consistency.

  • Motivation: Current diffusion- and flowmatching-based video models generate every frame, requiring up to tens of thousands of neural-network evaluations and several minutes of GPU time for ten seconds.This computational cost limits real-time applications including embodied AI, interactive content creation, and AR/VR.
  • Motivation: Existing efficiency methods still use neural networks for every frame, whereas SRENDER exploits redundant views of an underlying 3D scene.The approach focuses on camera-conditioned generation of static scenes to study scene redundancy without object motion or deformation.
  • Method: SRENDER synthesizes sparse keyframes with diffusion models, reconstructs a 3D scene, and renders the full camera-controlled video from that representation.Sparse multi-view observations can support high-quality reconstruction, while physically based rendering produces intermediate frames efficiently.
  • Adaptive allocation: The keyframe-density model adapts computation to camera trajectories, using very few keyframes for smooth or low-parallax motion and denser sampling for complex motion.This allocates generation effort where it is needed while minimizing redundant generation.
  • Results: Across videos up to twenty seconds, SRENDER selects 4–35 keyframes and achieves more than 40× speed-up on DL3DV and more than 20× on RealEstate10k.The method maintains comparable visual quality and temporal consistency while generating at most one-tenth as many frames as standard 30 fps video.

2. Related Work

Prior work improved video generation through diffusion-model architectures, inference acceleration, and 3D representations, but these approaches generally continued neural generation for every frame. SRENDER instead replaces dense frame synthesis with sparse keyframes followed by deterministic 3D reconstruction and rendering.

  • Video generation: Video-generation research has progressed from latent 3D-UNets to scalable transformers, while diffusion forcing and autoregressive methods enable alternative denoising and long-range strategies.Camera-controlled video generation is an active subfield within this broader development.
  • Efficiency: Inference remains expensive despite distillation and caching, because existing approaches still rely on neural networks to generate every video frame.SRENDER is complementary to diffusion-inference improvements because those improvements reduce sparse-keyframe costs, while SRENDER avoids neural generation of intermediate frames.
  • 3D reconstruction: 3D Gaussian Splatting and related feed-forward systems enable high-quality, consistent scene reconstruction from image collections without test-time optimization.Earlier deterministic reconstruction methods produce geometry but cannot represent the full distribution of plausible scenes and therefore are not generative models.
  • SRENDER: SRENDER replaces dense video-frame generation with deterministic feed-forward 3DGS reconstruction and rendering from sparse generated keyframes.The resulting reconstruction and intermediate-frame rendering are much faster than diffusion-based frame synthesis.

3. Keyframe Diffusion and 3D Rendering

SRENDER predicts a keyframe budget from the camera trajectory and scene appearance, generates those views with conditioned diffusion, and converts them into a dense video through deterministic 3D reconstruction and rendering. Its adaptive selection balances computational cost against coverage and reconstruction completeness, with temporal chunking used when long trajectories cause keyframe drift.

  • Adaptive keyframe selection: Dense keyframe sampling raises computation, whereas overly sparse sampling can produce incomplete reconstructions and visible holes in rendered views.The optimal count depends jointly on camera motion, parallax, and scene geometry.
  • Adaptive keyframe selection: The density predictor is a transformer that processes camera-pose tokens plus a DINOv2 scene-appearance token to predict the optimal number of keyframes.Ground-truth densities are automatically derived from point-cloud coverage procedures on RealEstate10k and DL3DV.
  • Pipeline: SRENDER predicts keyframe sparsity from the input image and camera trajectory, samples poses uniformly, generates keyframes, reconstructs a 3D scene, and renders the target video.The pipeline replaces dense diffusion generation with rendering from a reconstructed representation.
  • Adaptive keyframe selection: The selection procedure incrementally adds frames when projected point-cloud coverage falls below a threshold, ensuring collective pixel coverage across the video.After prediction, selected camera poses are sampled uniformly along the trajectory.
  • Keyframe diffusion: The keyframe generator conditions a low-frame-rate history-guided diffusion model on the input frame and camera trajectory, using the first frame as an appearance anchor.Progressive training reduces frame rate gradually to address instability from large viewpoint jumps; a two-stage scheme extends generation beyond the eight-frame context window.
  • 3D reconstruction and temporal chunks: AnySplat reconstructs a 3DGS representation from the generated unposed keyframes, after which dense video is rendered along the camera trajectory.For long, challenging DL3DV trajectories, keyframe drift beyond roughly 10 seconds can blur a single global reconstruction, so SRENDER uses fixed-length ten-second chunks.

4. Experiments

Experiments show that SRENDER delivers comparable or better quality than dense diffusion and interpolation baselines while substantially reducing generation time. Adaptive keyframe selection and temporal chunking improve the efficiency–quality trade-off and long-video reconstruction.

  • Quantitative Results: SRENDER outperforms HG on both FID and FVD across RE10K and DL3DV, indicating no visual-quality degradation from sparse diffusion and 3D rendering.The comparison covers both image and video quality metrics.
  • Quantitative Results: SRENDER achieves superior FID and FVD while being significantly faster than HG and Voyager on the DL3DV 5 fps test set.Voyager is evaluated only at 5 fps because its available implementation cannot generate hundreds of frames.
  • 3D vs. 2D Interpolation: Compared with 2D interpolation, SRENDER produces lower FVD, avoids morphing and warping across large viewpoint changes, and is faster because 3DGS rendering scales well to long sequences.The interpolation comparison uses the same sparse keyframes, approximately 3 seconds apart; only interpolation time is reported for the baselines.
  • Effect of Temporal Chunking: Temporal chunking improves both FID and FVD with comparable computational time by enabling more consistent and less blurry 3D scene reconstruction.The experiment generates 400-frame, 30 fps videos.
  • Keyframe Selection: SRENDER selects a keyframe density that avoids blank regions from undersampling without paying the redundant cost of excessive keyframes.Too few keyframes underdefine the scene, whereas additional keyframes after sufficient coverage provide no significant quality gains.

5. Discussion

SRENDER currently targets static scenes and trades some high-frequency detail for geometric consistency and speed. The authors position sparse view generation, adaptive keyframing, and 3D rendering as foundations for future dynamic-scene extensions.

  • SRENDER currently applies only to static scenes, while the authors identify dynamic-scene extension as future work.
  • Sparse view generation, adaptive keyframing, and 3D rendering are presented as transferable ideas for future dynamic environments.
  • Rendered videos may be smoother or less detailed at high frequencies than diffusion-based baselines.The method avoids noise amplification and view-dependent distortions while maintaining geometric consistency on challenging camera trajectories.
  • The authors expect improvements in 3D reconstruction models to increase SRENDER’s visual fidelity.

6. Conclusion

The paper presents SRENDER as an efficient approach to camera-conditioned video generation that exploits video redundancy. Its experiments report long, geometrically consistent videos at substantially lower computational cost while maintaining or improving visual quality.

  • SRENDER exploits inherent video redundancy to generate long, geometrically consistent videos at a small fraction of existing diffusion-based models’ computational cost.
  • Experiments report dramatic speed-ups while maintaining or improving visual quality relative to strong baselines.

A. Method Details

The adaptive keyframe predictor uses a dense camera trajectory and a reference image to estimate the sparse keyframe count needed for generation. It combines camera and image representations through a transformer and output MLP.

  • The predictor takes a dense camera trajectory and reference image as inputs and outputs the required number of sparse keyframes.
  • Camera rotations are converted to 4D quaternions and combined with translations into 7D camera vectors.
  • A two-layer MLP projects camera vectors to DINOv2’s dimensionality, while DINOv2 supplies the reference image’s global feature token.
  • A transformer processes the camera-token sequence after the reference-image token is appended.

A.1.2. Training

SRENDER trains its keyframe-budget predictor to regress the total number of sparse keyframes rather than exact indices. The objective averages token-level MLP outputs and compares that prediction with the ground-truth keyframe count.

  • The final MLP produces one value per transformer output token, whose average gives the predicted keyframe density.Scaling the output by 0.1 regulates predictions for keyframe counts ranging from 4 to 35 and improves training stability.
  • The training loss compares the averaged MLP output with the ground-truth keyframe number.
  • Training data uses 10-fps RE10K trajectories and 5-fps DL3DV trajectories, with average training-sequence lengths of 134.94 and 339.48 frames, respectively.
  • The predictor regresses total keyframe count and uniformly distributes keyframes along the camera trajectory because exact-index prediction is unstable.
  • The keyframe generation model is a camera-controlled diffusion-forcing transformer trained on sparse keyframes with camera-pose annotations.

A.2.1. Training

The system trains and applies diffusion-based keyframe generation alongside 3D reconstruction, trajectory alignment, and rendering. It supports autoregressive conditioning and chunk-wise alignment for seamless videos.

  • Training: Diffusion-forcing trains the model to denoise frames with independent noise levels within a shared context window.The model minimizes noise prediction loss across frames and noise levels.
  • Training: The keyframe model is initialized from an RE10K checkpoint and fine-tuned on DL3DV with progressively increasing frame gaps.Training begins with consecutive frames and increases adjacent-frame gaps to approximately 4 seconds.
  • Training: Independent frame noise levels allow conditioning on input or previously generated frames, enabling autoregressive sampling and interpolation guidance.Conditional frames are supplied at zero noise level, and guidance can be applied at arbitrary positions.
  • 3D Reconstruction: AnySplat reconstructs a 3D Gaussian scene representation from generated keyframes and predicts associated camera poses for subsequent processing.The pretrained model uses uncalibrated images and predicts Gaussian parameters, depth maps, and camera poses.
  • Trajectory Alignment: An affine transformation aligns the input camera trajectory with the reconstruction coordinate system before rendering the full video.The transformation is estimated from corresponding keyframe poses and applied to the whole trajectory.
  • Trajectory Alignment: Shared keyframes and additional transformations align independently reconstructed temporal chunks so their boundary poses coincide.This alignment is used to produce a seamless video across chunks.

B.1. Datasets

The experiments use RE10K and DL3DV evaluation videos prepared at compatible resolutions and frame rates. Their camera trajectories are processed to support the target generation settings.

  • RE10K: RE10K experiments use 200 test-split videos with at least 200 frames, taking the first 200 frames at 256×256 resolution and 10 fps.The videos come from the RE10K test split provided with History-Guided Video Diffusion.
  • DL3DV: DL3DV videos are resized to 256-pixel height and center-cropped to square frames for compatibility with HG and SRENDER.The dataset provides camera poses at 5 fps, which are interpolated to 30 fps.
  • DL3DV: DL3DV camera-pose interpolation enables camera-controlled generation at a higher frame rate than the dataset’s provided 5-fps poses.The prepared trajectories support 30-fps evaluation settings.

B.2. Detailed Experiment Setup

The detailed setup standardizes baseline and predictor inputs across datasets and compares Voyager under a constrained implementation setting. Image preprocessing also differs for Voyager to match its original rectangular training format.

  • Baseline Setup: For RE10K, HG generates 12 uniformly spaced keyframes across 200-frame, 10-fps trajectories before filling remaining frames from nearby keyframes.The same initial keyframe density is retained across experimental setups.
  • Keyframe Density Prediction: SRENDER’s density predictor receives target-length camera poses for RE10K and DL3DV, with 30-fps DL3DV trajectories first subsampled to 5 fps.This keeps predictor inputs within its supported input distribution.
  • Voyager Setup: Voyager uses full rectangular DL3DV frames for generation but is evaluated on center-cropped square videos.This preprocessing produces slightly higher performance than direct inference on square images.
  • Voyager Setup: Voyager is evaluated only on 100-frame, 5-fps videos because its claimed arbitrary-length generation and world-caching implementation is absent from the official release.Qualitative results are provided separately, while quantitative results appear in the main paper’s Table 2.

B.4. Comparisons with 2D interpolation methods

The comparison evaluates 3D reconstruction-based interpolation against RIFE and FILM using the same generated keyframes. The 3D method performs better because the 2D methods do not account for camera control during complex motion.

  • Experimental Setup: The experiment applies RIFE, FILM, and SRENDER to the same generated keyframes at a fixed keyframe density.Each method interpolates the full video from that shared keyframe set.
  • Results: SRENDER achieves superior qualitative and quantitative results compared with the two 2D interpolation methods.The comparison is reported in Table 3 of the main paper.
  • Results: Because RIFE and FILM do not consider camera control, they can produce naive morphing when camera motion is complex.SRENDER instead uses 3D reconstruction for camera-controlled interpolation.
Loading 2601.09697v1…