PaperlayerView exact revision on arXiv

Source-linked AI summary

Pantheon360: Taming Digital Twin Generation via 3D-Aware 360° Video Diffusion

Ting-Hsuan Chen, Ying-Huan Chen, Tao Tu, Jie-Ying Lee, Cho-Ying Wu, Fangzhou Lin, Hengyuan Zhang, David Paz, Xinyu Huang, Yuliang Guo, Yu-Lun Liu, Yue Wang, Liu Ren

arXiv:2605.25449v3cs.CV

TL;DR

Generating complete digital twins from video remains difficult because perspective generators lack global coverage and precise 3D-aware control. Pantheon360 uses an explicit 3D Cache to guide controllable 360° video diffusion, achieving state-of-the-art performance and strong cross-view and temporal coherence.

  • Problem

    Dynamic digital-twin generation requires 3D-aware controllability and long-term temporal consistency, while existing 360° methods lack precise trajectory control or validation on in-the-wild scenes.

  • Method

    Pantheon360 reconstructs an explicit 3D Cache from sparse 360° inputs to enforce geometric consistency while diffusion synthesizes photorealistic texture along user-defined camera paths.

  • Results

    Pantheon360 significantly outperforms all baselines across all metrics and achieves state-of-the-art performance across 360° video synthesis tasks.

  • Takeaways & Limitations

    Pantheon360 provides precise camera trajectory control and supports downstream 360° interpolation, Street View synthesis, and video stabilization.

  • Takeaways & Limitations

    Explicit control over object-level dynamics remains challenging because the 3D Cache primarily encodes static scene geometry.

Abstract

from arXiv · show

Generating complete digital twins from videos requires precise camera control, global scene coverage, and strict spatial-temporal consistency constraints that remain challenging for perspective video generators due to their limited field of view (FoV). Their narrow FoV forces long or multi-view trajectories, amplifying cross-view inconsistency and temporal drift. We argue that 360° video generation offers a natural solution: panoramic coverage simplifies trajectory design and provides a strong global context for maintaining coherence. We introduce Pantheon360: Taming Digital Twin Generation via 3D-Aware 360° Video Diffusion, a controllable 360° video generation framework that synthesizes high-fidelity videos from sparse 360° inputs. The key idea is an explicit 3D Cache, reconstructed from the input, which serves as a geometric scaffold for any user-defined camera path. This allows the diffusion model to focus on photorealistic texture refinement while the 3D Cache enforces global geometric consistency. Experiments show that Pantheon360 achieves superior visual quality and unmatched geometric coherence, enabling reliable and flexible 360° scene generation for downstream simulation and digital-twin applications.

1. Introduction

Dynamic digital twins require photorealistic, spatially consistent video generation, but perspective models struggle with limited field of view and unseen regions. Pantheon360 addresses this through 360° context and an explicit 3D Cache that enforces geometric consistency while diffusion synthesizes texture.

  • Motivation: Perspective video generation is unsuitable for complete digital twins because limited field of view forces models to hallucinate unseen regions during long or multitrajectory exploration.This causes redundant conditioning and severe spatial and temporal inconsistencies.
  • Motivation: 360° video captures the entire scene from t = 0, simplifying trajectory representation and improving consistency, but introduces equirectangular distortion and difficult geometric control.These challenges motivate a geometry-aware generation framework.
  • Pantheon360: Pantheon360 uses an explicit 3D Cache, built with geometric priors from 3D foundation models, to assign geometric reasoning to a scene-specific 3D representation.The design lets diffusion focus on photorealistic texture synthesis rather than reconstructing scene geometry.
  • Pantheon360: The framework renders the 3D point cloud along the exact user-defined camera trajectory Ctarget to produce a geometry-only video Vgeo, then conditions diffusion on Vgeo and input semantic features.The 3D Cache enforces global geometric consistency while the diffusion model handles photorealistic synthesis.
  • Results and applications: Pantheon360 enables exact camera trajectory control for in-the-wild 360° videos and achieves state-of-the-art performance against perspective and 360° baselines.The paper also demonstrates 360° interpolation, Google Maps Street View stitching, and video stabilization applications.

2. Related Work

Prior camera-controllable video methods use parametric or geometric mechanisms, but typically target limited-FoV perspective videos. Pantheon360 extends the 3D-cache paradigm to 360° generation, addressing panoramic continuity and control limitations while distinguishing generative synthesis from 360° reconstruction.

  • Camera-Controllable Video Generation: Existing camera-controllable video methods broadly use parametric or geometric approaches to encode camera information.Parametric methods include rotation matrices, translation vectors, and Plücker coordinate embeddings.
  • Camera-Controllable Video Generation: Limited-FoV perspective designs constrain complete scene observation, motivating 3D-cache methods for holistic 360° scene understanding.Pantheon360 extends the 3D-cache approach by grounding generation in geometric structure and leveraging complete 360° inputs.
  • 360° Video Generation: 360° video generation must handle equirectangular distortion and seamless panoramic continuity.Prior work includes text-to-360°, image-to-360° synthesis, and scene inpainting, but these methods generally lack complex or precise camera control.
  • 360° Video Generation: Existing 360° methods generally lack mechanisms for complex or precise camera control, while perspective-to-360° methods address a different conversion task.More recent models tackle direct 360° control, but the supplied passage states that they still fall short; GenEx is identified as a notable 360° world model.
  • 360° Reconstruction Models: 360° reconstruction models reproduce input views and interpolate seen regions, but they are not generative models for large occluded or entirely unseen areas.The paper positions its method in contrast to reconstruction models that cannot creatively hallucinate plausible unseen content.

3. Method

Pantheon360 generates controllable equirectangular 360° videos from sparse inputs by combining an explicit 3D Cache with semantic image conditioning in a fine-tuned latent video diffusion model. The 3D Cache is rendered along user-defined camera trajectories to provide a geometric scaffold for globally consistent generation.

  • Framework: Pantheon360 builds on a pre-trained SVD latent video diffusion model with explicit 3D scene conditioning for controllable 360° video synthesis.The generator is a fine-tuned SVD U-Net using the pre-trained SVD VAE encoder and decoder.
  • Geometric Conditioning: The Cache is rendered along a user-defined target trajectory as a geometry-only ERP video, encoded into a latent scaffold, and concatenated with noised latents during diffusion.This geometric latent provides the model with an explicit 3D-aware condition at each denoising step.
  • 3D Cache Reconstruction: Sparse 360° frames are cropped into perspective views and reconstructed into a 3D point-cloud Cache that models the scene’s spherical geometry.The reconstruction can use PI3, VGGT, or other methods that produce the required point-cloud representation.
  • Semantic Conditioning: Semantic conditioning uses CLIP features from 8 perspective crops of the first 360° frame, sampled every 45° of yaw, and injects them through cross-attention.Perspective crops are used because CLIP features are more robust there than on distorted equirectangular images.
  • Training Objective: The diffusion objective injects geometric-scaffold and semantic features into denoising to guide generation toward geometrically consistent and photorealistic 360° videos.The model denoises the noisy ground-truth latent representation using the geometric latent vequi and concatenated image features cimg.
  • Data Annotation: Because 360-1M lacks camera poses and 3D geometry, training pairs are annotated on the fly by processing ground-truth videos with ViPE to estimate trajectories and SLAM-based 3D Caches.The resulting pairs are formed from the ground-truth video, estimated camera-pose trajectory, and ViPE-generated SLAM representation.

4. Experiments

Pantheon360 is evaluated across single-view, sparse-view, novel-view, world-model, and ablation settings, consistently demonstrating strong trajectory control, geometric coherence, and temporal stability. Its explicit 3D Cache and latent fusion support accurate synthesis, interpolation, chaining, and stabilization.

  • Single 360° view-to-video generation: On Web360, Pantheon360 significantly outperforms ViewCrafter, TrajectoryCrafter, and GEN3C across all metrics using 8-crop perspective adaptations.Its full panoramic field of view improves cross-view consistency and scene understanding.
  • Sparse 360° views-to-video generation: On Habitat sparse-view generation, Pantheon360 achieves the best performance across all metrics, including MET3R: 0.3026 vs. 0.4522 for GEN3C.The 3D Cache provides geometric guidance for precise trajectory control while preserving photorealistic synthesis quality.
  • Sparse-view novel view synthesis: With only two 360° views, Pantheon360 produces novel-view trajectories with correct geometric structure, unlike PanoSplatt3R’s visible distortions and inconsistencies.The setting targets continuous video synthesis from sparse Google Maps Street View panoramas.
  • Comparison with 360° world models: Against GenEX on forward-motion Street View trajectories, Pantheon360 maintains consistent quality and accurately follows the prescribed trajectory while GenEX degrades rapidly.The comparison demonstrates superior temporal stability and geometric accuracy.
  • Ablation study: Dual+Latent Fusion achieves the best overall ablation performance, reaching 28.95 PSNR and 7.44 IE, while latent fusion mitigates geometric inconsistencies.Dual anchor conditioning improves convergence to 27.86 PSNR compared with 20.92 PSNR for the Single model.
  • Applications: Pantheon360 supports indefinite trajectory extension by chaining anchor frames and stabilizes perturbed Habitat videos with temporally coherent, geometrically consistent 360° rendering.Stabilization reconstructs a 3D Cache and re-renders scene geometry along a smoothed trajectory.

5. Conclusion · A. Overview

Pantheon360 enables controllable 360° video generation through an explicit 3D Cache that separates geometric reasoning from photorealistic synthesis, producing temporally consistent and cross-view coherent videos. The conclusion identifies limited object-level dynamics control, while the supplementary overview documents implementation, data, experiments, and interactive demonstrations.

  • 5. Conclusion: Pantheon360 provides precise camera trajectory control through an explicit 3D Cache for controllable 360° video generation.The framework decouples geometric reasoning from photorealistic synthesis.
  • 5. Conclusion: The approach generates temporally consistent videos with superior cross-view coherence by separating geometry from photorealistic synthesis.Experiments demonstrate state-of-the-art performance.
  • 5. Conclusion: The framework supports practical applications including Street View synthesis and video stabilization.These applications are showcased in the conclusion and supplementary demonstrations.
  • 5. Conclusion: Explicit control over object-level dynamics remains challenging despite the model’s ability to handle dynamic objects through learned motion priors.The 3D Cache primarily represents static scene geometry, while dynamic motion depends on the diffusion model’s learned priors.
  • 5. Conclusion: Future work could add explicit motion representations to enable fine-grained control over object dynamics.This limitation follows from the 3D Cache’s focus on static geometry and reliance on learned motion priors.
  • A. Overview: The supplementary material includes implementation and inference configurations, 3D Cache reconstruction settings, and dual-anchor latent fusion details.It also describes training-data curation, quality filtering, automatic trajectory annotation, and additional experimental results including a 3D Cache ablation.
  • A. Overview: An interactive HTML interface provides supplementary videos covering Google Street View synthesis, extended trajectory generation, and video stabilization.The supplementary material also includes qualitative comparisons with other state-of-the-art methods.

B. Implementation Details … B.3. Dual-Anchor Latent Fusion

Pantheon360 uses separate single-anchor and dual-anchor video diffusion models, a Pi3-based 3D Cache reconstruction pipeline, and bidirectional latent fusion for interpolation. Training and inference use fixed equirectangular settings, while dual-anchor conditioning averages forward and backward denoising predictions without noise-injection refinement.

  • B.1. Training and Inference Details: The model initializes from Argus, itself initialized from Stable Video Diffusion-I2V-XL, with separate single-anchor and dual-anchor models.The single-anchor model uses the first frame, whereas the dual-anchor model uses start and end frames for interpolation.
  • B.1. Training and Inference Details: Both models train at 512 × 1024 equirectangular resolution for 50,000 iterations using T = 25 frames, AdamW, a 1 × 10−5 learning rate, and batch size 16.Training requires approximately 5 days on 4 A100 GPUs and uses FP16 mixed precision with gradient checkpointing; inference uses 25 denoising steps.
  • B.2. 3D Cache Reconstruction: Pi3 serves as the primary 3D reconstruction foundation, receiving perspective views extracted from each 360° equirectangular frame.Equi2Pers uses 90° horizontal field of view and samples pitch angles of 0° and 60°.
  • B.2. 3D Cache Reconstruction: Yaw sampling at 45° intervals with 50% overlap produces 16 perspective views per 360° frame: 8 horizontal and 8 floor views.Ceiling views are excluded to avoid sky regions, while the two pitch levels provide complete 360° coverage.
  • B.2. 3D Cache Reconstruction: Pi3 predicts dense point clouds and confidence scores, which are merged in a common camera-pose coordinate frame after filtering.Confidence filtering uses a threshold of 0.25 after converting raw scores to probabilities with a sigmoid function.
  • B.3. Dual-Anchor Latent Fusion: The dual-anchor model uses Time Reversal Fusion to blend start- and end-frame information, especially when sparse inputs make direct geometric conditioning discontinuous.Forward rendering follows the start-to-end trajectory, while backward rendering reverses the trajectory from the end frame toward the start frame.
  • B.3. Dual-Anchor Latent Fusion: At each denoising timestep, separate forward and backward passes use their respective anchor features and geometric scaffolds, then average the predictions as xt−1 = 1/2(xt−1,s + xt−1,e).The geometric videos are encoded into latent space by the VAE encoder, and bidirectional fusion combines geometry while maintaining temporal coherence.
  • B.3. Dual-Anchor Latent Fusion: The method omits noise-injection refinement by setting t0 = 0 and M = 0, relying on bidirectional geometric conditioning for faster inference and effective interpolation.The passage attributes this choice to ablation results in Table 3 of the main paper.

C. Data Curation and Preparation · C.1. Quality Filtering Pipeline · D. Additional Experimental Results

The training data begins with approximately 100,000 high-quality clips curated from 360-1M and undergoes additional filtering for trajectory-controlled generation. Filtering removes mislabeled or visually contaminated videos and excludes clips with unreliable trajectory annotations.

  • C. Data Curation and Preparation: Approximately 100,000 high-quality clips are selected from the 360-1M dataset as the starting training subset.The subset was prepared by [62].
  • C.1. Quality Filtering Pipeline: Additional comprehensive filtering is applied to ensure data quality for trajectory-controlled video generation.The filtering supplements the initial curated subset.
  • C.1. Quality Filtering Pipeline: Format validation removes videos with incorrect labels before subsequent quality checks.The passage identifies mislabeled videos as a filtering target.
  • C.1. Quality Filtering Pipeline: Static-region analysis examines top and bottom frame areas across height ratios from 1% to 80% to detect overlays or borders.The analysis targets static UI elements, borders, and watermarks.
  • C.1. Quality Filtering Pipeline: Videos with MSE less than 1.0 in either top or bottom 20% height region are rejected as likely containing static UI elements or watermarks.The threshold is applied to the 20% height region.
  • C.1. Quality Filtering Pipeline: After ViPE [29] trajectory annotation, clips are filtered when the method fails to produce reliable results.Failure cases include insufficient camera baseline and too few SLAM-related observations.

D.1. 3D Cache Ablation Study

The ablation study evaluates the importance of geometric conditioning Vgeo by progressively removing points from the 3D Cache on Web360 and Habitat. Removing Vgeo consistently degrades performance, weakens geometric correctness, and makes precise trajectory control infeasible.

  • Ablation setup: The study progressively drops 3D Cache points to quantify the importance of geometric conditioning Vgeo on Web360 and Habitat.Results are reported in Table 4 for Web360 and Table 5 for Habitat.
  • Web360 results: Performance degrades consistently as more points are removed from the 3D Cache on Web360.This trend is reported in the Web360 ablation results.
  • Habitat results: Removing Vgeo causes significant performance degradation across all metrics on Habitat.The Habitat results are consistent with the Web360 ablation trend.
  • Effect of removing Vgeo: Without Vgeo, the model reduces to standard image-to-video generation and can still produce visually plausible results.However, geometric correctness cannot be fully guaranteed without geometric conditioning.
  • Effect of removing Vgeo: Removing Vgeo makes precise trajectory control infeasible because camera control explicitly relies on the rendered point cloud as a condition.The rendered point cloud provides the condition used for camera control.

D.2. Runtime and Memory Analysis · D.3. 3D Consistency Validation via Point Cloud Reconstruction

The runtime analysis identifies diffusion denoising as the dominant inference bottleneck while showing that Pantheon360 runs on one GPU. Point-cloud reconstruction further validates that generated videos preserve 3D consistency and complete unseen scene regions.

  • D.2. Runtime and Memory Analysis: The runtime and memory analysis evaluates Google Map data at 1024 × 512 resolution on a single A100 GPU.
  • D.2. Runtime and Memory Analysis: Approximately 80% of total inference time is consumed by diffusion denoising, making it the primary runtime bottleneck.
  • D.2. Runtime and Memory Analysis: Pantheon360 runs entirely on a single GPU during inference.
  • D.2. Runtime and Memory Analysis: Its modular framework is compatible with faster diffusion models that could reduce inference time in future work.
  • D.3. 3D Consistency Validation via Point Cloud Reconstruction: 3D consistency is validated by reconstructing point clouds from both the reference image and generated video using Pi3 [73].
  • D.3. 3D Consistency Validation via Point Cloud Reconstruction: The reference-image reconstruction contains only visible input-view geometry, whereas the generated-video reconstruction is significantly more complete and hallucinates unseen regions.

D.4. Robustness Analysis and Failure Cases · D.5. Closed-Loop Trajectory Validation

Pantheon360 remains robust to moderate 3D Cache imperfections by inpainting and refining incomplete regions, but fails in crowded dynamic scenes and inputs with stitching artifacts. Closed-loop validation shows temporally consistent revisitation, enabled by persistent geometric grounding from the 3D Cache.

  • D.4. Robustness Analysis and Failure Cases: Moderate reconstruction errors are handled by the video diffusion prior, which inpaints and refines holes or inaccuracies in the rendered 3D Cache.The errors may arise from dynamic objects or low-light conditions.
  • D.4. Robustness Analysis and Failure Cases: Crowded dynamic scenes with many moving objects cause motion-blur artifacts, constituting one identified failure case.This limitation is illustrated in Figure 12.
  • D.4. Robustness Analysis and Failure Cases: Generated videos produce more complete 3D point-cloud reconstructions by hallucinating occluded regions while preserving consistency with the original scene.The reconstructions use Pi3 from the reference image and generated video.
  • D.4. Robustness Analysis and Failure Cases: Input 360° images with stitching artifacts cause errors that propagate through the generation pipeline.Addressing these limitations requires advances in 4D reconstruction or higher-quality input capture.
  • D.5. Closed-Loop Trajectory Validation: On a closed-loop trajectory from Google Map data, sparse 360° panoramas yield temporally consistent video when the trajectory revisits its starting region.The result is shown in Figure 13.
  • D.5. Closed-Loop Trajectory Validation: Persistent geometric grounding from the 3D Cache enables consistent video generation throughout the closed-loop trajectory.The 3D Cache supports successful revisitation without temporal inconsistencies.
Loading 2605.25449v3…