PaperlayerView exact revision on arXiv

Source-linked AI summary

Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion

Yushi Lan, Zeren Jiang, Koichi Namekata, Xingang Pan, Chuanxia Zheng, Andrea Vedaldi

arXiv:2610.25276v1cs.CV

TL;DR

Novel-view video reshooting must preserve observed scene geometry and motion while completing newly visible regions. HOLO4D guides video diffusion with reconstructed 3D trajectories and source appearance; it reports strong camera and view-synthesis results, with sensitivity to geometry, coverage, and camera difficulty.

  • Problem

    Novel-view reshooting must preserve observed scene geometry and motion while completing surfaces newly visible from the requested camera path.

    Source: §1 · p. 2

  • Method

    HOLO4D projects persistent reconstructed 3D trajectories with source appearance into target views and uses them with source-video and camera conditioning to guide video diffusion.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    HOLO4D achieves the lowest trajectory and translation errors on DAVIS and the best PSNR and LPIPS on iPhone with the two-stage backend, and supports different reconstructors without retraining.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Explicit trajectory conditioning grounds generation in reconstructed observations while leaving regions beyond the scaffold to the generative prior.

    Source: §5 · p. 10

    Current first-frame trajectory querying may miss later-appearing content; querying trajectories in every source frame is proposed as future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content. We introduce Holo4D, a video reshooting framework that uses reconstructed 3D trajectories as a holistic source-view 4D scaffold. We project these trajectories into the requested views and encode their geometry and source appearance to guide a pretrained video diffusion model. Together with the source video and target cameras, the scaffold grounds generation in reconstructed observations while leaving unsupported regions to the generative prior. Experiments demonstrate the value of trajectory-based conditioning for camera control and novel-view synthesis. The framework accommodates different 4D reconstructors without retraining the video generator, allowing it to benefit from future advances in source reconstruction. More details can be found in our project page.

1 Introduction

Novel-view reshooting must preserve observed scene geometry and motion while plausibly completing newly visible surfaces. HOLO4D uses persistent 3D trajectories with source appearance to guide video diffusion, combining reconstructed observations with generative completion.

1 Introduction

  • Reshooting from a new camera path must preserve the recorded scene’s events and motion while completing surfaces revealed only from the new viewpoint.The task turns partial observations of a dynamic scene into a coherent video along a user-specified camera path.Source: §1 · p. 2
  • Reconstruction alone leaves holes or distortions in novel views, while video diffusion may fail to preserve the particular scene’s geometry and dynamics.The two approaches therefore supply complementary capabilities: observed structure and plausible appearance or motion.Source: §1 · p. 2
  • Persistent 3D trajectories link each scene point’s position and appearance across time, treating static points as constant trajectories and moving points with the same representation.This makes explicit the correspondence that independent framewise point clouds lack for moving surfaces.Source: §1 · p. 2
  • HOLO4D projects reconstructed trajectories with source appearance into target views as a holistic scaffold, guiding observed content while the video prior completes newly revealed regions.The framework combines track conditioning with source-video context and the requested camera trajectory.Source: §1 · p. 2, Figure 2 · p. 3
  • The explicit-geometry conditioning can use different 4D reconstructors without retraining the video generator, and upstream reconstruction replacement can substantially improve camera adherence.The paper also evaluates camera adherence and held-out views, analyzes reconstructor substitutions and trajectory perturbations, and demonstrates robot videos.Source: §1 · p. 2

2 Related Work

Related work spans 3D/4D reconstruction, geometry-guided synthesis, camera-controlled video generation, and diffusion models for geometry. HOLO4D uses reconstructed trajectories as an explicit scaffold for video reshooting.

3D and 4D reconstruction

  • 3D and 4D reconstruction methods recover scene geometry or motion, while HOLO4D conditions a shared video generator on reconstructed geometry without per-scene optimization.Prior approaches include deformable radiance fields, time-varying Gaussian representations, and diffusion-supervised Gaussian reconstruction.Source: §2 · p. 3

Geometry-guided view synthesis

  • Geometry-guided view synthesis aggregates source information, whereas HOLO4D conditions video diffusion on reconstructed 3D trajectories for monocular reshooting under large viewpoint changes.Related work also uses temporal guidance and inpainting for multiview video.Source: §2 · p. 3

Camera-controlled video generation

  • Camera-controlled video generation uses camera poses or rays, rendered source geometry, or warped observations; HOLO4D adds an explicit trajectory-based scaffold for reshooting.Related approaches also generate multiview videos before Gaussian reconstruction or render Gaussian features for sparse-view synthesis.Source: §2 · p. 3

Track-based conditioning

  • Unlike reconstructor-specific hidden features or reference-anchored 2D tracks, HOLO4D encodes 3D trajectory positions, source color, validity, and target-camera depth alongside synchronized source video.Vista4D uses source video and rendered point clouds with temporally persistent static pixels; Edit-by-Track uses paired source and target 3D point tracks.Source: §2 · p. 3

Diffusion models for geometry

  • Geometry-focused diffusion methods predict depth, video geometry, or geometric latents, while HOLO4D takes reconstructed geometry as input and generates target-view RGB.Related systems also couple video generators with reconstruction decoders or generate plannable 4D worlds.Source: §2 · p. 3

Video models for robots

  • Robot video-generation backbones serve robot learning as world action models, whereas HOLO4D reshoots recorded robot videos from new viewpoints.Source: §2 · p. 4

3 Method

HOLO4D reconstructs persistent 3D trajectories, projects them into requested views, and uses the resulting track features with source-video and camera conditioning to generate target video. Training supervises target latents; inference denoises them into output video.

3 Method

  • HOLO4D generates the same scene and motion from requested cameras by reconstructing persistent 3D trajectories, projecting them into target views, and conditioning pretrained video diffusion on the resulting features.Source: §3 · p. 4

3.1 Constructing a 4D scaffold

  • Each reconstructed trajectory follows a scene point over time and carries a sampled source color, with positions and target cameras expressed in the first source camera’s coordinate frame.Static points have constant positions, while moving points trace their motion; any reconstructor supplying trajectories and cameras can provide the scaffold.Source: §3.1 · p. 4
  • Projected observations encode frame, position, target-camera depth, validity, and fixed trajectory color; a feature volume maps them to track features T, with null embeddings for unobserved target tokens.Validity reflects upstream validity and in-image projection but does not account for occlusion; feature-volume size is independent of trajectory count at fixed video resolution and duration.Source: §3.1 · p. 4

3.2 Conditioning video diffusion

  • Source and noisy target latents are concatenated along width, while track features enter along channels on the target half; camera features encode target cameras as Plücker rays.Channel-wise injection leaves self-attention sequence length unchanged and achieves a 1.34× denoising speedup over Vista4D.Source: §3.2 · pp. 4–5

3.3 Training and inference

  • Training corrupts target latents with Gaussian noise and uses a rectified-flow objective to predict velocity; only target latents are supervised, and the model predicts no geometry.At test time, HOLO4D builds track and camera features for the requested cameras, denoises from noise, and decodes the resulting latents into video.Source: §3.3 · p. 5

3.4 Training details and datasets

  • Training starts from Wan2.1-T2V-14B and uses two resolution stages, training conditioning modules and DiT self-attention while keeping pretrained cross-attention and feed-forward layers frozen.The training mixture includes synthetic multiview dynamic scenes, real static-scene videos, and synthetic static-scene sequences; the second stage varies trajectory density and query-frame sampling and applies mild geometric perturbations.Source: §3.4 · p. 5

4 Experiments

Experiments compare HOLO4D with camera-only and rendered-geometry baselines across camera control, novel-view synthesis, scaffold analyses, and robot-video applications. Results report improved camera control and generated-view geometry, while analyses expose sensitivity to reconstruction quality and mixed policy-transfer outcomes.

4.1 Experimental setup

  • The evaluation compares HOLO4D with camera-only and rendered-geometry baselines across camera-control datasets, held-out iPhone views, and Syn4D; the default source reconstructor is 4RC.The setup also tests a two-stage backend and uses 50 denoising steps on DAVIS and iPhone.Source: §4.1 · pp. 6–7

4.2 Quantitative comparisons

  • On DAVIS, HOLO4D reduces ATE by 54% and RPE-T by 33% versus Vista4D when both use 4RC source reconstruction, and has lower rotation error than rendered-geometry baselines except ReCamMaster.ReCamMaster has slightly lower rotation error (0.099° versus 0.102°) but the largest trajectory error.Source: §4.2 · p. 7, Table 2 · p. 8
  • On DAVIS, HOLO4D leads rendered-geometry baselines in imaging quality and subject and background consistency; on Syn4D with ground-truth geometry, it leads on camera and image metrics.With 4RC estimates on a separate 32-pair Syn4D set, it leads on camera errors and LPIPS, while ReCamMaster leads on PSNR and SSIM.Source: §4.2 · pp. 7–8, Table 2 · p. 8
  • HOLO4D has the lowest generated-depth AbsRel and highest δ1 under both output reconstructors on Syn4D, and the lowest AbsRel on iPhone; it also beats ViDAR on PSNR and LPIPS but not SSIM.ViDAR takes about 5 h 18 min per iPhone scene, while HOLO4D requires no scene-specific optimization. HOLO4D's denoising is 1.34× faster than Vista4D, but end-to-end time is similar (570.9 versus 581.9 s).Source: §4.2 · p. 8

4.3 Qualitative comparisons

  • Qualitatively, HOLO4D preserves scene details such as a diner's orientation, a car's rear window and lights, and iPhone scene objects while following target viewpoints.The comparison also reports errors by other methods, including lost or duplicated people, missing cars, and holes.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4 Analysis of the geometric scaffold

  • The same generator accepts 4RC, V-DPM, and VGGT-Ω paired with Open-D4RT or CoWTracker without retraining; their trade-offs differ, and ground-truth geometry improves all five reported metrics.With track noise α = 0.1, rotation error triples from 0.073° to 0.221° and SSIM and LPIPS degrade.Source: §4.4 · p. 9

4.5 Robot videos

  • HOLO4D-14B reshoots DROID and humanoid-robot videos without robot training data or fine-tuning; long rollouts use HOLO4D for the first 49-frame window and a continuation model thereafter.The later-window continuation model trains on OmniWorld-DROID, so those windows are not zero-shot; a 196-frame example shows consistent appearance across two window boundaries.Source: Figure 5 · p. 8, Figure 6 · p. 9, §4.5 · p. 10
  • Policy-transfer results are mixed: HOLO4D initialization gains 1.29 percentage points overall on LIBERO-plus, mainly under background-texture and lighting perturbations, but loses on the spatial suite.LIBERO is near saturation (95.60% versus 96.45%), and camera-viewpoint perturbations remain difficult for all checkpoints; the study uses a single seed.Source: §4.5 · p. 10, Table 7 · p. 10

5 Conclusion

HOLO4D conditions video diffusion on persistent 3D trajectories carrying source appearance, while regions beyond the scaffold remain to the generative prior. This version removes the rendered RGB-D branch, expands evaluation, and demonstrates performance, reconstructor flexibility, and remaining sensitivities.

5 Conclusion

  • HOLO4D achieves the lowest trajectory and translation errors on DAVIS and the best PSNR and LPIPS on iPhone with the two-stage backend.It works with different reconstructors without retraining, but remains sensitive to geometric error, incomplete source coverage, and difficult camera trajectories.Source: §5 · p. 10

A Changes from the Submitted Version

  • The final conditioning design removes rendered RGB-D and retains projected 3D tracks alongside the source video and target-camera rays.Track observations encode source color and target-camera depth; validity combines upstream validity with in-image projection rather than a target-view occlusion test.Source: A Changes from the Submitted Version · p. 16
  • The final main results use a 14B model initialized from Wan2.1-T2V-14B, trained first at 144 × 256 and then at 384 × 672 with a revised data mixture.The submitted version instead used Wan2.1-I2V-14B and Wan2.1-T2V-1.3B backbones at 144 × 256 and 288 × 512; early 1.3B checkpoints are not main results here.Source: A Changes from the Submitted Version · p. 16
  • Evaluation adds Vista4D’s 110-trajectory benchmark, Syn4D with exact target RGB and geometry, per-scene ViDAR optimization, alternative reconstructors, trajectory-noise tests, and runtime.Camera recovery changed from ViPE to VGGT-Ω; DAVIS video quality is measured with VBench and Syn4D reports FVD.Source: A Changes from the Submitted Version · pp. 16–17
  • New demonstrations include zero-shot robot-video reshooting, long-video generation with a continuation model trained partly on robot videos, and Fast-WAM policy learning initialized from a 5B HOLO4D variant.The 14B zero-shot robot reshooting uses no robot training data; the continuation model has seen robot videos.Source: A Changes from the Submitted Version · p. 17
  • Submitted qualitative results use rendered RGB-D and track conditioning, whereas the corresponding final-model results are shown separately.The submitted 14B DAVIS and iPhone results used multiframe softsplat rendering.Source: A Changes from the Submitted Version · p. 17

B Implementation Details

The implementation projects persistent source-view trajectories into target cameras and encodes them as target-aligned features for video diffusion. It also specifies camera and query-frame conditioning, training, and a windowed continuation procedure for longer videos.

Submitted-version implementation

  • The submitted pipeline rendered a hybrid target-view RGB-D cache, but the final version removes that branch and retains trajectory conditioning.Submitted qualitative results are from earlier checkpoints with RGB-D and track conditioning, not the final model.Source: Figure 7 · p. 17, Figure 8 · p. 17, Figure 9 · p. 18

Backbone and training

  • The backbone is initialized from Wan2.1-T2V-14B; self-attention and new conditioning modules are fine-tuned while cross-attention and feed-forward layers remain frozen.The model has 40 DiT blocks, and both training stages optimize RGB flow matching.Source: B Implementation Details · pp. 19–20

Track encoding

  • Track observations encode normalized time and position, validity, target-camera depth, and source color sampled at the query frame and held fixed along each trajectory.Validity combines the upstream track mask with valid target-image projection, not an exact target-view occlusion test; at 384 × 672, the track set is subsampled or padded to 258,048 identities.Source: B Implementation Details · p. 19
  • Projected track features are scattered onto the target grid with channel-wise maximum aggregation and occupancy, then causally patchified to match the DiT token grid.The source half receives null track tokens; track features are injected along channels in every DiT block without adding attention tokens.Source: B Implementation Details · pp. 19–20, Figure 11 · p. 19

Camera and query-frame conditioning

  • A shared camera encoder supplies target-camera features to both source and target tokens, while the query-frame indicator marks which source frames initialize trajectories.The query indicator is encoded through the video VAE; its control latents occupy the source half of the width-concatenated input, with zeros on the target half.Source: B Implementation Details · p. 20

Long-video rollout

  • Long-video rollout reconstructs overlapping 49-frame windows independently, registers each to the first window, and uses a continuation model for later windows.The continuation model receives the prior window’s last generated frame; identities are not shared across reconstructions, and the shared boundary image is not a source or ground-truth frame.Source: B Implementation Details · p. 20

C Training Data

Training combines synthetic and real data across static and dynamic scenes, with different source–target pairing rules for each. The second stage shifts sampling toward dynamic scenes and uses a narrower dataset mixture.

Training data

  • Stage 1 uses a broad mixture of synthetic and real static and dynamic scenes, while stage 2 concentrates on four datasets at higher resolution.Geometry supplies conditions for RGB generation rather than auxiliary prediction targets; Syn4D’s flying_group and MVS-Synth sequence 0000 are excluded from both stages.Source: C Training Data · p. 21
  • The dynamic-scene share rises from 44.30% in stage 1 to 70% in stage 2, which uses Syn4D and MultiCamVideo for dynamic scenes and DL3DV and MVS-Synth for static scenes.Stage 1 also includes two equally weighted Kubric subsets, DL3DV, RE10K, MVS-Synth, and TartanAir.Source: C Training Data · p. 21
  • Dataset geometry and tracks come from VGGT-Ω, CoWTracker, or DepthAnything 3, and the 30% second-stage MultiCamVideo weight comprises 21% dense and 9% sparse query-frame data.Table 9 reports sampling weights per stage rather than proportions of distinct videos.Source: C Training Data · p. 21
  • Dynamic training pairs come from synchronized multiview videos, whereas static pairs come from different chunks of the same monocular video because temporal synchronization is unnecessary for static scenes.Dynamic monocular videos are not used for training.Source: C Training Data · p. 21

D Camera-Adherence Evaluation on DAVIS

On DAVIS, camera adherence is evaluated on generated videos using recovered cameras, with translation and rotation metrics computed over different trajectory subsets. HOLO4D’s results vary with source geometry and preprocessing, and the comparison also reports smaller-backbone and runtime results.

Evaluation protocol

  • Camera recovery uses VGGT-Ω on 100 DAVIS video–trajectory pairs; translation metrics use 54 trajectories and rotation uses all 100.DAVIS has no captured target-view reference for these paths, so the evaluation reports camera control and VBench scores.Source: D Camera-Adherence Evaluation on DAVIS · p. 21

Metric caveats

  • ATE includes post-initial-frame orientation error and depends on the requested trajectories, so its magnitude is not directly comparable across DAVIS and Vista4D trajectory sets.Translation and rotation metrics use different trajectory counts in this evaluation.Source: D Camera-Adherence Evaluation on DAVIS · pp. 21–22

Source geometry and sampling steps

  • With the two-stage backend and GMOS, ATE falls from 1.657×10−2 to 0.719 × 10−2 at 50 steps, while VBench scores are lower.On the same condition, 25 denoising steps give lower rotation error than 50 (0.077◦ versus 0.101◦); the 5B variant with GMOS also lowers ATE from 1.464 × 10−2 to 0.608 × 10−2.Source: D Camera-Adherence Evaluation on DAVIS · p. 22, Table 11 · p. 22, Table 12 · p. 23

Runtime

  • HOLO4D-5B generates 704 × 1280 video in 202.0 s on average, versus 417.4 s for HOLO4D-14B at 384 × 672; these configurations differ in backbone, resolution, and track condition.The measured 5B sampling time is about 2.1× lower, but this is not a speedup under matched conditions.Source: D Camera-Adherence Evaluation on DAVIS · p. 22

Static/dynamic fusion

E Camera Adherence on Vista4D Trajectories

On Vista4D’s native trajectories, HOLO4D’s camera-error comparisons depend on the evaluator and source-geometry alignment; the comparison also changes several reconstruction inputs together. The supplied evidence includes a separate partially oracle Syn4D condition and an early-checkpoint study that does not measure the final model’s rotation accuracy.

E Camera Adherence on Vista4D Trajectories

  • With 4RC source geometry, HOLO4D has lower RPE-T under VGGT-Ω and lower translation error under Pi3, but higher rotation error under both evaluators.The valid-metric counts differ by method for some metrics, so these are not common-subset evaluations.Source: E Camera Adherence on Vista4D… · p. 23
  • An alternative pointmap Sim(3) alignment changes VGGT-Ω RPE-R from 1.142° to 0.348° and Pi3 RotErr from 14.98° to 17.19°.The two evaluators disagree on the effect of this preprocessing change.Source: E Camera Adherence on Vista4D… · p. 23
  • The additional Pi3-source Syn4D condition uses estimated depth and cameras with ground-truth 2D correspondences and visibility, rather than full ground-truth or 4RC geometry.The 5B model’s source-aligned rotation error with Pi3 geometry is 0.740°; this condition is partially oracle.Source: E Camera Adherence on Vista4D… · p. 23
  • In the two-stage backend, GMOS separates moving-region trajectories from those held fixed at their query-frame 3D positions, then combines both sets into one track condition.Background regions occluded by the foreground in the query frame have no trajectories.Source: Figure 12 · p. 24
  • On Vista4D’s native trajectories, source replacement changes camera estimates, geometry, and coverage together, so it does not isolate the generator’s sensitivity to reconstruction error.Separating generator sensitivity would require both methods to use the same estimated geometry on these trajectories.Source: E Camera Adherence on Vista4D… · p. 24
  • With early 1.3B checkpoints, added geometric conditions lowered Pi3 translation error from 0.987 to 0.685, while rotation error did not decrease.These checkpoints barely followed requested rotation, so the study does not establish rotation accuracy for the final model.Source: E Camera Adherence on Vista4D… · p. 24

F Novel-View Evaluation on iPhone/DyCheck

The iPhone/DyCheck evaluation uses a held-out target camera and common resized frame canvas, while depth comparisons use reconstructed target-video depth as a pseudo-reference. Conditioning comparisons report a translation–rotation trade-off for rendered streams, with distinct training schedules and evaluation sets limiting cross-block comparisons.

F Novel-View Evaluation on iPhone/DyCheck

  • The iPhone/DyCheck evaluation pairs a synchronized source-camera video with a held-out target-camera trajectory and reference RGB, scoring predictions over 49 resized frames.Masked metrics use the intersection of valid and co-visible pixels, not a foreground-object mask.Source: F Novel-View Evaluation on… · p. 24
  • The supplied early-checkpoint table caption defines rotation gain as the fitted ratio of generated to requested rotation, but provides no result values in the supplied passage.The comparison concerns early 1.3B checkpoints.Source: Table 15 · p. 25
  • On iPhone/DyCheck, target depth reconstructed from held-out RGB is a pseudo-reference rather than sensor ground truth.Both source conditions are scored with both output reconstructors under a shared scorer, crop, valid co-visible mask, and confidence threshold.Source: F Novel-View Evaluation on… · p. 25

G Conditioning Comparisons

  • The rendered-feature stage-1 variant sums encoded rendered RGB, validity, and source/target motion masks with track features before injection, without adding attention tokens.The study evaluates variants sharing initialization, training data, optimizer, and 15,000 training steps.Source: G Conditioning Comparisons · p. 25
  • Adding rendered streams in the common lower block reduces translation errors but increases rotation error.The unmatched upper block uses different training schedules and evaluation sets, so its magnitudes should not be compared with the lower block.Source: G Conditioning Comparisons · p. 25, Table 16 · p. 25

H Additional Novel-View Synthesis Evaluation

An earlier rendered-conditioning architecture improves over its RGB-D tier when tracks are added, although variant ordering depends on metric and backend. The ViDAR comparison favors HOLO4D on PSNR and LPIPS and ViDAR on SSIM, under a short-window evaluation with less temporal and viewpoint coverage than ViDAR’s full-sequence benchmark.

H Additional Novel-View Synthesis Evaluation

  • Adding tracks improves on the RGB-D tier under both source backends, while the ordering of single- and multi-frame variants depends on metric and backend.These are earlier-architecture checkpoints with a rendered-conditioning branch, not the main-paper model; neither panel tests removal of that branch.Source: H Additional Novel-View Synthesis… · p. 26, Table 17 · p. 26
  • With the two-stage backend, HOLO4D has better PSNR and LPIPS than ViDAR, while ViDAR has better SSIM; the same ordering holds for masked metrics.ViDAR takes an average of 5 h 18 min per scene on an A6000, but this is not a hardware-matched speed comparison.Source: H Additional Novel-View Synthesis… · p. 26
  • The ViDAR comparison evaluates the first 49 target frames after center-cropping and resizing, giving it less temporal and viewpoint coverage than ViDAR’s full-sequence benchmark.CAT4D is not included because its code and checkpoints were unavailable at the time of the experiments.Source: H Additional Novel-View Synthesis… · p. 26

I Synthetic Evaluation and Upstream Robustness

On Syn4D with estimated 4RC geometry, HOLO4D leads on camera errors and LPIPS, while other methods lead on PSNR, SSIM, and FVD. The study also tests alternative reconstruction backends, track corruption, conditioning costs, and runtime, with several evaluation and timing boundaries that limit direct comparisons.

I Synthetic Evaluation and Upstream Robustness

  • The ground-truth-source Syn4D comparison uses five held-out clips and 32 source–target pairs, with exact target RGB and depth and different source-geometry inputs for HOLO4D and rendered-geometry baselines.Syn4D also contributes training data for HOLO4D, and its ATE protocol differs from DAVIS and native Vista4D, so ATE values should be compared within each setting.Source: I Synthetic Evaluation and Upstream… · p. 26, Figure 13 · p. 27
  • With 4RC source geometry on Syn4D, HOLO4D has the lowest camera errors and LPIPS, ReCamMaster the highest PSNR and SSIM, and Vista4D the lowest FVD.The test uses 32 pairs from 16 held-out scenes; FVD is supplementary given the small sample, and the test differs from the ground-truth-source experiment.Source: I Synthetic Evaluation and Upstream… · p. 27, Table 19 · p. 27
  • For Syn4D depth evaluation, generated-video depth is compared with exact target depth; on iPhone/DyCheck, reconstructed target-video depth is only a pseudo-reference.The supplied table caption identifies output-depth evaluators and HOLO4D source backends but provides no metric values.Source: I Synthetic Evaluation and Upstream… · pp. 27–28, Table 20 · p. 28
  • Replacing the source backend among 4RC, V-DPM, VGGT-Ω with CoWTracker, and VGGT-Ω with Open-D4RT preserves a shared interface without retraining, but yields different quality trade-offs.The checkpoint, target trajectories, sampler, and seed are held fixed; the VGGT-Ω tracker conditions use no GMOS fusion.Source: I Synthetic Evaluation and Upstream… · p. 28, Table 21 · p. 28
  • As Gaussian track corruption increases, rotation error, SSIM, and LPIPS degrade; ATE jumps at α = 0.1 but does not continue growing under independent Sim(3) alignment.ATE alone therefore does not characterize this sensitivity.Source: I Synthetic Evaluation and Upstream… · p. 28, Table 21 · p. 28

J Conditioning Cost, Runtime, and Memory

  • HOLO4D’s channel-wise injector concatenates track features with hidden features and projects them back to the hidden width, preserving self-attention sequence length.Fewer tokens do not imply proportional total-memory savings because source-video tokens, adapters, and projections still incur cost.Source: J Conditioning Cost, Runtime, and Memory · p. 29
  • A separate warm H200 profile averages 96.99 ± 0.67 s over four videos, excluding model loading and MP4 writing, with 10.49 s preprocessing and 86.50 s denoising plus VAE decoding.The 1.46 s 4RC timing is fresh reconstruction, and the harness also times proxy rendering.Source: J Conditioning Cost, Runtime, and Memory · p. 29, Table 22 · p. 29
  • On one H200, HOLO4D’s matched sampling loop averages 7.39 s per step versus Vista4D’s 9.89 s, while full runs take 570.9 versus 581.9 s and peak memory is 49.14 versus 49.86 GiB.Training-step cost was not measured; the remaining runtime stages were not individually profiled.Source: J Conditioning Cost, Runtime, and Memory · p. 30, Table 24 · p. 30

K Additional Qualitative Results

Additional examples show HOLO4D across varied camera paths and robot videos, while also exposing limits from fine dynamic structures, disocclusion, and missing trajectory coverage.

K Additional Qualitative Results

  • Six Vista4D and six DAVIS trajectories cover lateral, orbit, crane, dolly, and zoom motions, with projected paths overlaid for visual comparison.Two further trajectories are also compared across methods.Source: K Additional Qualitative Results · p. 30
  • HOLO4D-14B is applied without fine-tuning to DROID manipulation episodes and humanoid-robot clips; further rollouts use a continuation model trained with OmniWorld-DROID.Source: K Additional Qualitative Results · p. 30
  • Fine structures such as hands can remain inconsistent across views, while large disoccluded regions lack geometric support and may receive implausible generative completions.Source: K Additional Qualitative Results · p. 30
  • With first-frame queries, objects entering later may lack trajectories, leaving target-view placement to be inferred without direct geometric support despite source-video appearance.The cited example illustrates missing scaffold coverage, not a measured timing error.Source: K Additional Qualitative Results · p. 30

L Robot Policy Learning with Fast-WAM

This section reports mixed, single-seed policy experiment results and documents the work’s data sources, methodological scope, potential benefits and misuse risks, external-asset terms, limitations, and reproducibility details.

L Robot Policy Learning with Fast-WAM

  • On LIBERO-plus, HOLO4D improves the goal suite by 6.2 points but drops 4.8 points on the spatial suite.Both arms were trained for 20,000 steps with identical data order, normalization, seed, and optimization settings, and all checkpoints used the Fast-WAM closed-loop evaluator. All runs use a single seed.Source: L Robot Policy Learning with Fast-WAM · p. 30, Table 25 · p. 35

M Broader Impacts

  • Controllable reshooting can support content creation, immersive media, simulation, and robot-data construction, but generated views can be plausible yet factually incorrect.Source-video and geometric conditioning do not guarantee fidelity in disoccluded regions; track validity is not a forensic test, so generated views should be distinguished from recorded footage.Source: M Broader Impacts · p. 32

N Existing Assets and Licenses

NeurIPS Paper Checklist

  • The paper identifies sensitivity to geometric error, incomplete source coverage, and difficult camera paths, alongside failures on fine dynamics and large disocclusions.Content entering after the query frame lacks coverage, and the long-video continuation model is not zero-shot; the policy-learning results are mixed and single-seed.Source: NeurIPS Paper Checklist · p. 37
  • The paper reports reproducibility details spanning the method, training settings, datasets, inference, and evaluation protocols.It also states that code, pretrained models, predictions, and evaluation code will be released with instructions to reproduce the main results.Source: NeurIPS Paper Checklist · pp. 38–39
  • Main video-generation comparisons report point estimates without repeated-training confidence intervals, so their variability is not quantified.The early conditioning study reports a confidence interval over pairs, and policy learning uses a single seed, but neither provides uncertainty estimates for the main results.Source: NeurIPS Paper Checklist · p. 40

10. Broader impacts

  • The checklist notes potential positive societal uses and the risk that generated views could be mistaken for recorded footage.Released checkpoints are to follow the base model’s license and usage terms.Source: §10 · p. 41
  • The work involves no crowdsourcing or newly recruited human participants, so IRB approval is not applicable.It uses existing datasets and publicly available robot videos.Source: §10 · p. 43
  • LLMs are not part of HOLO4D’s core methodology; its umT5-XXL text encoder is inherited from Wan 2.1 and used as a frozen conditioning module.Source: §10 · p. 43
Loading 2610.25276v1…