PaperlayerView exact revision on arXiv

Source-linked AI summary

Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting

Yongqi Mao, Zijia Dai, Zhishuo Liu, Wei Xu, Kaiwei Wang, Guotao Meng

arXiv:2608.28174v1cs.CV

TL;DR

Video re-shooting must follow a target camera trajectory while reconstructing and synthesizing a dynamic scene, but persistent source-video and render conditions create a trust dilemma. MANIFOLD4D injects the render into flow-matching’s initial noise and leaves the source video as the only visual condition, achieving the best reported camera control with competitive visual quality. The method remains effective under large motions and corrupted renders, although it breaks the standard Gaussian-noise assumption and makes training harder.

  • Problem

    Persistent source-video and point-cloud-render conditions force video re-shooting models to reconcile competing visual signals, with reliability beyond training motion magnitudes not guaranteed.

  • Method

    MANIFOLD4D injects a target-trajectory point-cloud render into the starting point of flow matching while keeping the source video as the only visual condition during denoising.

  • Results

    MANIFOLD4D ranks first on every camera-control metric across two benchmarks, reducing rotation error by 25% and 27% and translation error by up to 32% against the strongest baseline while matching its video fidelity.

  • Takeaways & Limitations

    The geometric prior guides generation without overriding source-video appearance, including when camera motion exceeds training ranges or the render is deliberately corrupted.

  • Takeaways & Limitations

    Injecting geometry breaks the diffusion model’s Gaussian initial-noise assumption, which in principle makes training harder and sacrifices part of the pretrained model’s capability.

Abstract

from arXiv · show

Video re-shooting re-renders a monocular video of a dynamic scene along a user-specified camera trajectory, and the dominant recipe supplies the target geometry explicitly: per-frame depth lifts the source video into a 4D point cloud, which is rasterized along the trajectory into a point cloud render. Because the render and the source video are both handed to the network as visual conditions, they compete at every denoising step, leaving the model with a trust dilemma --- how much of the render to believe --- which can degrade trajectory control or visual quality on data outside the training distribution. We argue that a render already pixel-aligned with the target view does not need to be supplied as an explicit conditioning stream at all. We propose MANIFOLD4D, which injects the render directly into the initial noise of flow matching, so that generation no longer departs from standard Gaussian noise but from a new noise manifold carrying geometric information, leaving the source video as the only visual condition. The render is thus used exactly once, and the network is never asked to learn how to read it; in subsequent denoising steps the model can focus on the source video. On our DAVIS-Traj benchmark and on the Vista4D evaluation set, MANIFOLD4D attains the best camera-control accuracy on every metric, lowering rotation error by 25% and 27% and translation error by up to 32% over the strongest baseline, while matching it in video fidelity and leading on real-world novel-view photometric quality. In a user study, our method achieves clear advantages in trajectory following and dynamic consistency. The gap widens as the yaw amplitude grows past the training range, and the model still recovers correct dynamic motion from the source video when the render is deliberately corrupted, confirming that the geometric prior guides generation without overriding it.

1 INTRODUCTION

MANIFOLD4D addresses the trust dilemma created when source video and pixel-aligned renders compete as persistent visual conditions. It instead injects geometry into the starting noise, retains the source video as the only visual condition, and achieves strong camera control with competitive visual quality.

  • Video re-shooting renders a monocular dynamic-scene video along a user-specified camera trajectory while requiring faithful reconstruction, plausible unseen content, and precise motion following.
  • Prior render-conditioned methods make the network reconcile source video and point-cloud geometry at every denoising step, creating a trust dilemma beyond training motion magnitudes.
  • MANIFOLD4D injects the point-cloud render into initial flow-matching noise once, while the source video remains the only visual condition during denoising.This design gives generation geometry-bearing noise instead of a pure Gaussian starting point and leaves the model room to correct erroneous renders.
  • 25% and 27% lower rotation error and up to 32% lower translation error are reported against the strongest baseline, while visual quality matches it across two benchmarks.MANIFOLD4D ranks first on every camera-control metric and achieves clear user-study advantages in trajectory following and dynamic consistency.
  • The analysis identifies direct consumption of pixel-aligned geometric priors at the starting point as more effective than learning them as persistent conditions.The method also experimentally adapts to the injected noise after fine-tuning without destroying the diffusion prior.

2 RELATED WORK

Related work spans pose-conditioned generation, explicit render-conditioned geometry, and signal injection at diffusion initialization. MANIFOLD4D combines these directions by using rendered geometry as a starting signal while encoding source video and trajectory information for denoising.

  • Pose-conditioned video generation: Pose-conditioned video generation encodes camera trajectories numerically and asks the network to infer the geometry implied by those signals.
  • Render-conditioned video generation: Render-conditioned generation lifts video or images into point clouds and rasterizes them along target trajectories, providing pixel-aligned explicit geometry.
  • Signal injection as a starting point: Signal-injection methods initialize diffusion from an existing or degraded signal rather than supplying it only as a condition.
  • MANIFOLD4D pipeline: MANIFOLD4D applies signal injection to video re-shooting by injecting rendered-view latents into noise, concatenating them with source-video tokens, and denoising with a DiT.

3 METHOD

MANIFOLD4D reconstructs a 4D point cloud, renders it along the target trajectory, and injects that render into flow matching's starting state while using the source video as the only visual condition. This geometry-aware initialization combines explicit spatial structure with source-video appearance information and supports camera control across evaluation settings.

  • Explicit 4D point cloud: The model builds a globally aligned 4D point cloud by back-projecting source-video pixels using estimated depth, intrinsics, and camera extrinsics.Dynamic points are retained per timestamp, while static points are accumulated into a global map.
  • Explicit 4D point cloud: Target frames rasterize the complete static map together with dynamic points from the corresponding source timestamp, producing a trajectory-aligned render.This 4D formulation exhibits stronger consistency than using only one frame's point cloud or aggregating a few frames.
  • Geometry-aware starting point: The source video is concatenated at the token level as the sole visual condition, supplying appearance, style, and fine-grained detail while the render provides geometric structure.The render is not supplied again during denoising, and the method retains trajectory encodings during training even though removing them at inference leaves performance essentially unchanged.
  • Geometry-aware starting point: MANIFOLD4D injects the point-cloud render into the initial noise once, replacing the Gaussian endpoint with a geometry-bearing starting state for flow matching.Covered tokens combine the render with residual noise, while hole tokens remain pure Gaussian noise and partial coverage interpolates the two.
  • Evaluation: MANIFOLD4D attains the highest camera-control accuracy on every reported metric, approaching or surpassing the point-cloud render reference on rotation error.The evaluation reports camera-control accuracy across benchmarks, while Table 1 identifies the best results using bold formatting.
  • Implementation: The experiments use a Wan2.1-T2V foundation, fine-tune selected attention and projection components, and train on approximately 36K clips from five data sources.Training uses AdamW for 30K steps at 49 frames and 384×672 resolution.

4 EXPERIMENTS

Experiments evaluate MANIFOLD4D on camera control, visual and photometric quality, generalization to larger motions, qualitative consistency, user preferences, and ablations. The method achieves strong trajectory control while preserving visual quality and robustness to render imperfections.

  • Experimental setup: The evaluation uses DAVIS-Traj and Vista4D-Eval, with published baselines receiving the same 4D point-cloud projection at inference.DAVIS-Traj contains 72 clips from 24 dynamic scenes; Vista4D-Eval contributes 110 evaluation clips.
  • Camera control accuracy: MANIFOLD4D keeps the lowest rotation and translation errors in both filtered and unfiltered benchmark settings.The evaluation averages metrics over three seeds, and MANIFOLD4D’s rotation accuracy approaches or surpasses the render reference.
  • Visual and photometric quality: MANIFOLD4D matches Vista4D in video fidelity and leads on all photometric metrics except SSIM and mSSIM, also leading on optical-flow error.The visual-quality evaluation uses FID, FVD, VBench, and photometric metrics on the iPhone dataset.
  • Qualitative comparisons: MANIFOLD4D shows precise control, dynamic-object consistency, and visual quality, while baselines exhibit render sparsity, geometric inconsistency, or weak trajectory adherence.The qualitative comparison attributes fragmented content to over-reliance on sparse renders and explains Vista4D’s pose accuracy through static-background cues.
  • User study and ablations: Users prefer MANIFOLD4D most clearly for trajectory following and dynamic consistency, while inference ablations show that source video and injected render are critical.Increasing σ slightly reduces trajectory control but improves imaging quality; using the raw render or a separate render-token stream causes blur, fragmentation, or geometric inconsistency.

5 CONCLUSION

The conclusion presents MANIFOLD4D as a high-quality video re-shooting model that injects explicit geometry once while retaining the source video as the only visual condition. It also identifies the broken Gaussian initialization as a training limitation and open problem.

  • Conclusion: MANIFOLD4D re-shoots monocular video along user-specified trajectories by injecting explicit geometry into noise once and keeping the source video as the only visual condition.The design is presented as resolving competition between conditioning signals while making direct use of the geometric prior.
  • Limitations and future work: Breaking the Gaussian initial-noise assumption makes training harder and sacrifices part of the pretrained model’s capability.The authors report that Schrödinger-bridge formulations did not yield encouraging experimental results.
  • Teaser figure: The teaser’s horizontal axis averages normalized rotation and translation errors, while its vertical axis averages aesthetic and imaging VBench scores.Both axes summarize benchmark-level control and visual quality.

B TRAINING DETAILS

Training builds on Wan2.1-T2V-14B with a selectively trainable subset, mixes five video sources, and optimizes render-initialized latent clips with independently dropped conditioning streams.

  • Base model and trainable subset: The model uses Wan2.1-T2V-14B and trains self-attention projections, patchify embeddings, and the per-block Plücker camera encoder.The native VAE maps 49-frame 384×672 clips to 13 latent frames.
  • Training data: Training mixes DL3DV, DynPose, OpenVid-HD, MultiCamVideo, and HuMMan.The listed sources include static scenes, monocular dynamic clips, synthetic multiview scenes, and real mult-camera human captures.
  • Trajectory visualization: DAVIS-Traj trajectory families orbit the dynamic subject, combining yaw sweeps with pitch and dolly changes.The families are f1, f2, and f3, with the exact trajectories described in the benchmark construction.
  • Optimisation: Thirty thousand AdamW steps use learning rate 10^-5, batch size 1 per GPU on eight A100s, with extra loss weight on dynamic-subject pixels.Covered tokens start from noisy renders, while covered and hole regions receive equal loss normalization.

C BENCHMARK CONSTRUCTION DETAILS

The benchmark constructs controlled camera trajectories around dynamic subjects and supplements them with a growing-yaw study to test behavior beyond training-range motion.

  • Benchmark construction: DAVIS-Traj selects 24 dynamic scenes and evaluates three trajectory families, producing 72 clips; Vista4D contributes 110 additional evaluation clips.All trajectories keep the dynamic subject on the optical axis.
  • Trajectory families: f1 sweeps yaw from −60° to +60° at source distance with zero pitch and unit dolly.This is the widest fixed-distance trajectory family.
  • Trajectory families: f2 sweeps yaw from +30° to −30° at −10° pitch and 1.3× source-camera distance.The family changes both viewpoint elevation and distance relative to the source.
  • Trajectory families: f3 sweeps yaw from +30° to −30° at +10° pitch while dollying from 1.0× to 0.7× source distance.This family combines viewpoint elevation with an inward camera motion.
  • Alignment conditioning: The trajectories pass near the source viewpoint mid-clip, preserving overlap that anchors alignment while sweep ends exercise larger camera motion.The motion span keeps sim(3) alignment well conditioned and limits systematic pose-estimation error.
  • Growing-magnitude trajectories: The growing-yaw study uses symmetric sweeps with per-side amplitudes from 10° to 90°, or total sweeps of 20° to 180°.These trajectories use zero pitch and unit dolly on ten benchmark scenes.

D EVALUATION PROTOCOL

The evaluation aligns estimated cameras to ground truth with a global similarity transform, then reports averaged per-frame pose errors while filtering catastrophic alignments. Inference-time comparisons also establish that MANIFOLD4D is competitive with comparable explicit-geometry baselines in both control and quality.

  • Centers alignment: Estimated cameras are aligned to ground truth using a global similarity transform with orientation-based Procrustes rotation, followed by least-squares scale and translation fitting.The recovered transform is applied before comparing generated cameras with the target trajectory.
  • Metrics: Three pose metrics are computed per frame and averaged over T generated frames.Median source-scene depth normalizes translation error to remove arbitrary dataset world-scale bias; focal length uses the height-normalized value fy/H.
  • Filtering: Clips with RotErr > 60° in any method or seed are removed because they indicate catastrophic alignment failures rather than generated-video failures.Seven of 110 Vista4D-Eval clips exceed this threshold, typically under nearly static source cameras and very small source-target overlap.
  • Inference comparison: MANIFOLD4D is competitive with Vista4D and GEN3C in inference time while achieving the best camera control and visual quality.The comparison uses 50 diffusion steps on one NVIDIA A100 80 GB GPU and evaluates methods at native resolution.

E.2 PI3 POSE ESTIMATION RESULTS

Pi3-based evaluation confirms MANIFOLD4D’s camera-control advantage under both filtered and raw reporting protocols. The evaluation also tracks performance across yaw amplitudes and training checkpoints.

  • Pi3 reconstruction: MANIFOLD4D still ranks best overall under Pi3 camera reconstruction, despite inflated errors across all methods.The filtered and raw protocols differ in whether the RotErr ≤60° criterion is applied; the point cloud render is a geometric reference, not a ranked generated method.
  • Training convergence: Camera control converges after 20k training steps, and visual quality stabilizes by 25k on Vista4D-Eval.The 30k checkpoint is therefore used in the main paper.

E.5 FULL QUALITY METRICS

Full quality metrics show a trade-off between trajectory control and visual-quality leadership: ReCamMaster leads several quality columns while MANIFOLD4D remains on par with Vista4D. Quality and pose are also examined across yaw amplitude and training progress.

  • Full quality metrics: MANIFOLD4D is on par with Vista4D on full VBench and CLIP-T quality metrics, while ReCamMaster leads FID, subject consistency, background consistency, and motion smoothness.The passage attributes ReCamMaster’s quality lead to considerably weaker trajectory control.
  • Real-world qualitative results: Qualitative iPhone results show Vista4D’s mild dynamic-subject misalignment, worse dynamic-object quality for GEN3C and TrajectoryCrafter, and substantial trajectory deviation for ReCamMaster.The comparison uses red guide lines to highlight 3D inconsistency.
  • Yaw-amplitude analysis: Visual quality is reported with VBench aesthetic and imaging-quality scores at per-side yaw amplitudes from 10° to 90°.The corresponding camera-control table reports rotation error and normalized translation error over the same yaw-amplitude range.

F.2 ROBUSTNESS TO IMPERFECT GEOMETRY

MANIFOLD4D remains able to recover dynamic motion when the point-cloud render contains incorrect dynamic geometry, although fine-detail quality degrades. Removing noise injection also produces blur in fine details.

  • Corrupted geometry: MANIFOLD4D recovers correct dynamic motion from the source video when the render stacks dynamic points across frames and supplies incorrect geometry.The corrupted render is created by removing the motion mask at render time.
  • Corrupted geometry: The corrupted-geometry condition causes moderate fine-detail degradation, including residual blur where stacked points overlap.The degradation is described as expected under the conflicting geometric signal.
  • Noise-injection ablation: The σ=0 variant, which omits noise injection, produces blur in fine details.The qualitative comparison also reports 3D inconsistency on dynamic objects for the Vista4D-style three-stream layout.

G USER STUDY INTERFACE

The evaluation separates trajectory adherence and dynamic consistency from visual-quality assessment, using a point-cloud overlay only for the former questions. Qualitative comparisons also examine geometry preservation, robustness to incorrect dynamic geometry, and novel-view quality.

  • User study interface: The user study evaluates trajectory adherence and dynamic consistency with a toggleable point-cloud overlay, while visual quality is assessed separately using full-video playback.Separating the visual-quality page was intended to mitigate bias from the preceding overlay-based questions.
  • Novel-view comparison: Vista4D shows mild 3D inconsistency, while TrajectoryCrafter and GEN3C appear flat and ReCamMaster lacks camera control in real iPhone novel-view comparisons.The comparison also notes that Vista4D does not follow point-cloud colors and that the other methods miss fine human details or appear flat.
  • Ablation comparison: The full model preserves dynamic subjects and target-view geometry that are lost when the start construction or three-stream conditioning layout is ablated.Raw-render starts fragment the dynamic subject, whereas the three-stream layout drifts from the prescribed geometry.
  • Robustness: MANIFOLD4D still recovers correct motion from the source video when dynamic points are stacked across frames, smearing the rendered subject.This robustness test omits the motion mask to simulate erroneous point-cloud geometry.
Loading 2608.28174v1…