PaperlayerView exact revision on arXiv

Source-linked AI summary

Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering

Hanxin Zhu, Jianqi Ma, Cong Wang, Zhihao Hu, Junyao Gao, Tianyu He, Junyan Lin, Jiayin Xiao, Jiateng Shou, Sen Liang, Weidong Zhang, Zhibo Chen, Jun Zhang

arXiv:2610.14685v1cs.CV

TL;DR

Independent camera-trajectory synthesis lacks the cross-view consistency needed for coherent, navigable 4D worlds. Demiurge autoregressively rerenders videos using an evolving 4D point-cloud memory, and reports state-of-the-art results alongside robust large-parallax and geometry-faithful reconstruction capabilities.

  • Problem

    Independent novel-view video trajectories leave cross-view geometry, appearance, and dynamics unconstrained, limiting coherent, traversable 4D world construction.

    Source: §1 · p. 1

  • Method

    Demiurge uses an evolving 4D point-cloud memory to condition novel-view generation and consolidate recovered geometry and appearance across autoregressive rerendering steps.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Demiurge reports state-of-the-art results across perceptual realism, geometric accuracy, and cross-trajectory consistency, including an 85.7% reduction in P-MSE versus the strongest baselines.

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    The generated videos support coherent dense-view lattices, rerendering through 180° rotations without input-view overlap, and high-quality 3D Gaussian Splatting reconstruction with minimal artifacts.

    Source: §1 · p. 3

    The authors note potential misuse to create misleading content or process identifiable people without consent, and possible inherited biases from pretrained models and training data.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Recent advances in video diffusion enable novel-view video synthesis from a single input video, supporting world modeling, embodied intelligence, and robotic simulation. However, existing methods process camera trajectories independently, lacking the cross-trajectory consistency needed for coherent, navigable 4D worlds. We introduce Demiurge, an autoregressive video rerendering pipeline built on a dynamic 4D point-cloud memory cache. It promotes cross-trajectory consistency through four components: (1) an appearance video diffusion model that synthesizes photorealistic novel-view videos conditioned on the 4D memory; (2) a memory-consistent video depth completion model that aligns known-region depths with cached geometry and completes missing regions, mitigating scale ambiguity and discontinuities during memory updates; (3) memory-aware dual-stream warping that provides high-fidelity visual guidance and occlusion cues for novel-view synthesis; and (4) geometry-aware video refinement that enhances perceptual quality. Extensive experiments demonstrate state-of-the-art performance and three key capabilities: coherent spatiotemporal lattice construction over dense multi-view trajectories; large-parallax video rerendering under challenging camera motions, including 180-degree rotations with no input-view overlap; and geometry-faithful outputs that can be directly optimized into high-quality 3D Gaussian Splatting representations with minimal artifacts. Project page: https://hanxinzhu-lab.github.io/demiurge_project_page/.

1 INTRODUCTION

A single video captures only partial 2D observations, while independent novel-view trajectories remain inconsistent and cannot form a coherent, traversable 4D world. Demiurge uses an evolving 4D point-cloud memory to connect trajectories and reports strong results under dense views, large parallax, and 3D reconstruction.

1 INTRODUCTION

  • Independent camera trajectories leave scene geometry, appearance, and dynamics unconstrained across views, preventing a globally consistent, traversable 4D world.A single video also leaves occluded regions unobserved because it records only 2D projections along one camera trajectory.Source: §1 · pp. 1–2
  • Demiurge autoregressively conditions each target-trajectory video on a shared 4D point-cloud memory, then fuses its appearance and recovered geometry back into memory for subsequent rerendering.The overview describes this as a generate–recover–update cycle integrating warping, appearance generation, depth completion, and optional refinement.Source: §1 · p. 2, Figure 2 · p. 2
  • Demiurge reports state-of-the-art perceptual realism, geometric accuracy, and cross-trajectory consistency, including coherent dense-view lattices, 180° rotations without input-view overlap, and geometry-faithful 3D Gaussian Splatting outputs.The outputs can be optimized into high-quality 3D Gaussian Splatting representations with minimal artifacts.Source: §1 · p. 3
  • Its coordinated designs combine memory-guided appearance generation, memory-consistent depth completion, dual-stream warping, and optional video refinement.The components provide novel-view synthesis, reliable geometry recovery and memory updates, appearance-preserving occlusion handling, and attenuation of texture flicker and noise.Source: §1 · p. 3

2 RELATED WORKS

Prior camera-controllable video methods synthesize trajectories independently, while memory-based approaches vary in their geometric grounding and ability to retain history. Demiurge’s unified 4D memory consolidates generated appearance and recovered geometry across trajectories.

2.1 CAMERA-CONTROLLABLE VIDEO GENERATION

  • Camera-controllable methods range from pose-conditioned diffusion to geometry-grounded reprojection, but recent monocular-video rerenderers still synthesize each trajectory independently and retain nothing for later views.Demiurge instead maintains explicit 4D memory of generated appearance and recovered geometry across diverse camera trajectories.Source: §2.1 · p. 3

2.2 MEMORY-AUGMENTED VIDEO GENERATION

  • Appearance- and feature-space memories provide limited geometric grounding, while geometry-aware memories often preserve static geometry rather than the full time-varying scene state.PlenopticDreamer’s fixed top-k context can discard detail as trajectories exceed capacity; Demiurge consolidates observations into unified 4D memory.Source: §2.2 · pp. 3–4

3 METHOD

Demiurge iteratively rerenders target trajectories from an evolving 4D point-cloud memory, fusing generated appearance and recovered geometry back into it to guide subsequent trajectories.

3 METHOD

  • Demiurge maintains a persistent 4D point-cloud memory, rendering it along each target trajectory and fusing generated appearance and recovered geometry back into the memory.The updated memory then guides the next trajectory in the generate–recover–update loop.Source: §3 · p. 4

3.1 MEMORY-GUIDED APPEARANCE VIDEO DIFFUSION

  • A memory-guided appearance diffusion model uses partial RGB, depth, and visibility cues while a noise monitor flags unreliable rendered-depth pixels.The model preserves reliable memory-supported content while regenerating unreliable and newly exposed regions with photorealistic details.Source: §3.1 · pp. 4–5

3.2 MEMORY-CONSISTENT VIDEO DEPTH COMPLETION

  • Memory-consistent depth completion anchors known regions to cached geometry and completes newly revealed regions, then aligns metric scale across frames and refines boundaries.Poisson refinement preserves the aligned prediction's spatial gradients while enforcing exact agreement with cached geometry.Source: §3.2 · p. 5

3.3 MEMORY-AWARE DUAL-STREAM WARPING

  • Selective-only updates preserve stable colors but can leave sparse geometry that leaks background under parallax; dense updates improve occlusion reasoning but accumulate artifacts and color drift.The two update strategies therefore trade appearance preservation against geometric completeness.Source: §3.3 · p. 6
  • Dual-stream warping renders appearance and geometry memories in parallel, retaining sharp appearance except where depth conflicts trigger denser geometry-based occlusion conditions.The fused conditions guide the next autoregressive generation step.Source: §3.3 · pp. 6–7
  • Optional geometry-aware refinement conditions on partial memory RGB and completed depth to reduce texture flicker and noise while preserving recovered scene structure.The refinement improves final video quality without affecting the core world-expansion process.Source: §3.3 · p. 7

4 EXPERIMENTS

Demiurge is evaluated across diverse in-the-wild videos and camera trajectories, with results showing stronger novel-view quality and consistency than the compared methods. Ablations examine the complementary roles of its components.

4.1 EXPERIMENTAL SETUPS

  • Evaluation covers 128 in-the-wild videos and 14 camera trajectories per input, yielding 1,792 generated videos compared across six baselines.Metrics assess video quality, consistency, 3DGS rendering, and trajectory adherence.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

  • Demiurge achieves the best reported novel-view video metrics, with FVD 334.518, P-MSE reduced by 85.7%, and View Sync improved by 39.4%.It also reduces rotation error by 11.0% and translation error by 22.8% versus the strongest baselines.Source: Table 1 · p. 7, §4.2 · p. 8
  • For reconstructed 3DGS, Demiurge leads all reported metrics, achieving PSNR 24.360, SSIM 0.769, and LPIPS 0.237.It improves PSNR by 2.148 dB and reduces LPIPS by 16.0% and aggregate reconstruction error by 21.9%.Source: Table 2 · p. 8, §4.2 · pp. 8–9

4.3 ABLATION STUDIES

  • Replacing video depth completion with monocular depth raises P-MSE from 0.0010 to 0.0020 and lowers View Sync from 64.893 to 54.226.Removing dual-stream warping raises P-MSE to 0.0049 and lowers View Sync to 51.498; refinement removes residual artifacts and improves local details.Source: §4.3 · p. 9, Table 3 · p. 9, Figure 4 · p. 9
  • The noise monitor suppresses unreliable memory projections and boundary artifacts, while dual-stream warping preserves visual details and reliable occlusion geometry.The ablation attributes complementary roles to noise monitoring, depth completion, dual-stream warping, and geometry-aware refinement.Source: §4.3 · p. 9, Figure 4 · p. 9, Table 3 · p. 9

5 CONCLUSION

Demiurge builds navigable 4D worlds by repeatedly adding generated appearance and recovered geometry to an evolving point-cloud memory. The conclusion reports state-of-the-art results and robust large-parallax rerendering, alongside geometry-faithful 3DGS reconstruction.

5 CONCLUSION

  • Experiments establish state-of-the-art performance in perceptual quality, trajectory accuracy, and cross-trajectory consistency, with robust large-parallax rerendering and geometry-faithful 3DGS reconstruction.The framework also assembles long-range consistency by consolidating generated appearance and recovered geometry into memory after each trajectory.Source: §5 · pp. 9–10

AI USE STATEMENT

  • The authors state that generative AI was used only for grammar correction and linguistic refinement, with edits reviewed and responsibility for the manuscript retained by the authors.They state it was not used for conceptualization, method development, code generation, experimental design, data analysis, or scientific claims.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The ethics statement identifies risks of misleading content, nonconsensual processing of identifiable people, and biases inherited from pretrained models and training data.It encourages disclosure, respect for privacy and data licenses, and provenance or watermarking in practical deployments.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The reproducibility statement says the paper and supplement report model, training, inference, memory, evaluation, data, and resource details, and that code, checkpoints, and evaluation scripts are intended for release.Additional videos and supporting materials are also available on the project page.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 4D POINT-CLOUD REPRESENTATION

  • The 4D memory is a time-ordered sequence of colored 3D point-cloud snapshots, with each snapshot representing the scene at one instant.RGB-D observations are added only to the snapshot at their timestamp, so moving objects occupy different 3D locations over time rather than being duplicated within one static cloud.Source: §A.1 · p. 15

A.2 POINT-CLOUD WARPING

  • Historical trajectories are warped independently and combined with a nearest-depth z-buffer, selecting the closest valid source when multiple sources cover a target pixel.Depth-aware bilinear splatting distributes source RGB and depth to neighboring target pixels while reducing contributions from distant points.Source: §A.2 · pp. 16–17

B ADDITIONAL METHOD DETAILS

The additional method details specify how Demiurge cleans memory renderings, generates appearance and depth, resolves cross-stream conflicts, and optionally refines video before memory updates.

B.1 NOISE MONITOR

  • A deterministic noise monitor marks pixels unreliable for invalid depth, large depth gradients, or high local depth variance, then masks RGB, depth, and visibility conditions together.The default applies this cleaning once after point-cloud rendering; it has no learnable parameters and negligible computational overhead.Source: §B.1 · p. 17

B.2 CACHE-GUIDED APPEARANCE VIDEO DIFFUSION

  • The cache-guided appearance model conditions video diffusion on partial RGB, depth, and visibility renderings to synthesize novel-view RGB along the target trajectory.The conditions are normalized and encoded into a shared spatiotemporal latent space before injection into the diffusion model.Source: §B.2 · p. 18

B.3 CACHE-CONSISTENT VIDEO DEPTH COMPLETION

  • Depth completion anchors predictions to cached partial disparity and uses generated RGB cues for regions the cache does not cover.Video-wide affine alignment recovers metric scale, while Poisson refinement preserves completed structure and enforces exact consistency with cached geometry.Source: §B.3 · pp. 19–20

B.4 DUAL-STREAM WARPING

  • Dual-stream warping preserves sharper appearance results where streams agree and selects the denser geometry stream at conflicting pixels for more reliable visibility reasoning.The appearance stream retains the original input and previously synthesized regions, while the geometry stream uses all valid RGB-D observations.Source: §B.4 · p. 21

B.5 GEOMETRY-AWARE VIDEO REFINEMENT

  • The optional refinement model uses partial RGB and completed dense depth to reduce residual texture flicker and high-frequency artifacts before observations enter memory.It improves appearance without modifying recovered depth, and the refined video replaces the initial appearance result when refinement is enabled.Source: §B.5 · p. 22

C TRAINING AND INFERENCE DETAILS

Training data are constructed by reprojection of monocular videos into target-aligned partial RGB-D conditions, then used to train the appearance, depth-completion, and refinement models with corresponding clean-video or depth supervision.

C.1 TRAINING DATA CONSTRUCTION

  • The unified training corpus contains 136K MultiCamVideo samples and 100K OpenVid samples, with temporally consistent depth estimated using DepthCrafter.Double reprojection synthesizes target-aligned partial RGB-D conditions from monocular videos.Source: §C.1 · p. 22
  • Double reprojection creates realistic geometry-induced missing regions while keeping partial RGB-D conditions spatially aligned with the clean target video and depth.The first reprojection introduces disocclusions, out-of-view content, and sparse rendering holes; points removed by occlusion or field-of-view changes cannot be recovered in the second reprojection.Source: §C.1 · p. 23
  • Appearance diffusion is trained on double-reprojected RGB-D conditions and visibility masks, using the clean video as supervision.For depth completion, masked complete depth supplies partial-depth input and full-depth supervision, while the generated RGB video corresponds to the RGB guidance used at inference.Source: §C.1 · p. 23
  • The refinement model is trained with partial RGB and complete depth conditions, using the clean target video as supervision.This teaches it to restore high-frequency appearance details while remaining constrained by complete scene geometry.Source: §C.1 · p. 24

C.2 TRAINING CONFIGURATIONS

The appendix specifies shared training configurations for the three video diffusion models and a sequential inference procedure that updates the 4D memory after each target trajectory.

C.2 TRAINING CONFIGURATIONS

  • The three models are trained independently from Wan2.1-Fun-1.3B-InP, with video DiT parameters optimized while the Wan VAE, UMT5 text encoder, and CLIP image encoder remain frozen.Training uses AdamW, a 1 × 10^-4 learning rate annealed to 1 × 10^-6, five epochs, and 81-frame samples at 480 × 832 resolution.Source: §C.2 · p. 24

C.3 INFERENCE CONFIGURATIONS

  • Inference sequentially renders and cleans memory conditions, generates appearance, completes metric depth, optionally refines video, then back-projects the RGB-D pair into memory before the next trajectory.Each diffusion module generates 81-frame videos at 480 × 832 using 50 flow-matching steps; rendering is chunked across at most 16 frames and eight historical trajectories per batch.Source: §C.3 · p. 24

D.1 METRIC DEFINITIONS

Demiurge’s generated views preserve appearance, scene structure, and dynamics across trajectories, including large rotations, and support coherent 3D reconstruction. The section also defines camera, consistency, reconstruction, and perceptual-quality evaluations.

D.1 METRIC DEFINITIONS

  • FVD evaluates distribution-level video quality per target trajectory, and Q-Align assesses video- and image-level quality and aesthetics.Q-Align video scores average over eight-frame segments; image scores average over frames, scenes, and trajectories.Source: §D.1 · p. 25
  • Projection MSE compares generated frames with visible regions of geometrically projected input RGB-D video, using a visibility mask.The metric is evaluated only over visible regions.Source: §D.1 · p. 25
  • Camera accuracy is evaluated against target trajectories using estimated poses; the shared first-frame pose is excluded, and rotation and translation errors are aligned for coordinate and monocular-scale ambiguities.Rotation error uses global axis-convention alignment and per-timestamp world-frame alignment; translation uses per-timestamp Sim(3) alignment.Source: §D.1 · pp. 25–26
  • View Sync counts high-confidence cross-trajectory frame correspondences, while 3DGS consistency compares generated frames with renderings of reconstructions from aligned views.Higher View Sync indicates stronger cross-view consistency; reconstruction is assessed with PSNR, SSIM, and LPIPS.Source: §D.1 · pp. 26–27

E.1 COHERENT SPATIOTEMPORAL LATTICE GENERATION

  • Ten camera trajectories form a spatiotemporal lattice in which appearance, dynamic content, and recovered geometry remain coherent across aligned views and times.Rows represent target trajectories and columns show temporally aligned frames.Source: §E.1 · p. 27

E.2 LARGE-PARALLAX VIDEO RERENDERING

  • Progressive rotations up to 180° in both directions reveal new regions while preserving scene identity, dynamics, and spatial continuity despite diminishing input-view overlap.Source: §E.2 · p. 27

E.3 GEOMETRY-FAITHFUL OUTPUTS FOR 3DGS RECONSTRUCTION

  • Independent 3D Gaussian reconstructions at three time steps preserve coherent geometry and yield consistent RGB and depth renderings across views.This supports that the observations are geometrically compatible, rather than merely plausible as separate videos.Source: §E.3 · p. 27

F MORE QUALITATIVE COMPARISONS

  • Across different trajectories, Demiurge preserves object identity, fine-grained appearance, scene layout, and temporal dynamics while revealing geometrically plausible content.Competing methods can produce plausible individual videos yet show appearance drift, deformation, inconsistent backgrounds, or insufficient camera-motion response.Source: F MORE QUALITATIVE COMPARISONS · p. 28

G ADDITIONAL ABLATION RESULTS

Ablations show that filtering unreliable projections, cache-consistent depth completion, geometry-aware refinement, and both warping streams address distinct artifact sources.

G ADDITIONAL ABLATION RESULTS

  • Without noise monitoring, unreliable projections near depth discontinuities propagate into RGB and recovered geometry, causing local boundary artifacts.Filtering those projections produces cleaner appearance and more stable object geometry.Source: G ADDITIONAL ABLATION RESULTS · p. 28
  • Cache-consistent depth completion stabilizes shapes by anchoring observed regions to cached geometry and completing newly revealed regions.Direct depth estimation instead yields temporal instability and geometric discontinuities.Source: G ADDITIONAL ABLATION RESULTS · p. 28
  • Geometry-aware refinement suppresses residual appearance artifacts and texture flicker while preserving recovered scene structure.Source: G ADDITIONAL ABLATION RESULTS · p. 28
  • The two warping streams are complementary: removing geometry weakens occlusion reasoning, while removing appearance degrades visuals through repeated updates.Together they preserve stable appearance and dense geometry for visibility reasoning.Source: G ADDITIONAL ABLATION RESULTS · p. 28

H COMPATIBILITY WITH EXISTING METHOD

Replacing Demiurge’s appearance generator with GEN3C retains coherent multi-trajectory videos and temporally stable depth for memory updates. Additional visualizations show coherent appearance, dynamics, and geometry across diverse dynamic scenes.

H COMPATIBILITY WITH EXISTING METHOD

  • With GEN3C replacing only the appearance generator, the retained depth-recovery and memory-update pipeline produces coherent videos across trajectories and temporally stable depth.The results support using the explicit 4D memory and generate–recover–update cycle as a consistency layer for existing methods.Source: H COMPATIBILITY WITH EXISTING METHOD · p. 28

I ADDITIONAL VISUALIZATIONS

  • Across additional indoor and outdoor dynamic scenes, views and recovered depths preserve scene appearance, dynamic content, and geometric structure across viewpoints and time.Rows show target trajectories; aligned samples run chronologically across columns, with RGB above depth.Source: I ADDITIONAL VISUALIZATIONS · p. 28

J RUNTIME ANALYSIS

With refinement enabled, one trajectory takes about five minutes, and inference time scales approximately linearly with trajectory count. The system has generated up to 100 consecutive trajectories without noticeable error accumulation.

J RUNTIME ANALYSIS

  • About 5 minutes per target trajectory with refinement enabled; total inference scales approximately as 5N minutes for N autoregressively processed trajectories.Disabling refinement reduces diffusion inference time to approximately 3 minutes per trajectory.Source: J RUNTIME ANALYSIS · p. 28
  • Up to 100 consecutive camera trajectories were generated without noticeable error accumulation, despite the implementation not being speed-optimized.Source: J RUNTIME ANALYSIS · pp. 28–29

K LIMITATIONS AND FUTURE WORK

Demiurge’s full pipeline is limited by inference latency, and its evolving point-cloud memory may become costly as trajectories accumulate. The authors propose asynchronous processing and more efficient memory representations as future directions.

K LIMITATIONS AND FUTURE WORK

  • The sequentially coupled pipeline takes approximately five minutes to generate one target-trajectory video, limiting interactive use.Appearance generation, depth completion, optional refinement, and memory updates exchange and synchronize intermediate representations before the next autoregressive step.Source: K LIMITATIONS AND FUTURE WORK · p. 29
  • An asynchronous fast–slow design could separate responsive RGB rendering within the cached region from background geometry recovery and memory maintenance.The proposed fast path queries the latest 4D cache, while the background path completes depth, refinement, back-projection, and memory updates.Source: K LIMITATIONS AND FUTURE WORK · p. 29
  • Repeated observations may add redundant points, increasing storage, retrieval, and rendering costs as the number of trajectories grows.The authors suggest hierarchical spatiotemporal compression that merges compatible observations while preserving dynamic boundaries and newly revealed regions.Source: K LIMITATIONS AND FUTURE WORK · p. 29

Related papers

Loading 2610.14685v1…