PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE

Ruijie Zhu, Jiahao Lu, Wenbo Hu, Xiaoguang Han, Jianfei Cai, Ying Shan, Chuanxia Zheng

arXiv:2602.08961v2cs.CVcs.AIcs.CGcs.LG

TL;DR

Reconstructing dense 4D geometry and motion from monocular video is challenging because the problem is ill-posed and temporal correspondences fail under occlusion and large motion. MotionCrafter jointly models world-coordinate point maps and scene flows with a shared 4D VAE, achieving state-of-the-art geometry and motion reconstruction without post-processing.

  • Problem

    Monocular 4D reconstruction requires jointly recovering scene geometry and dense point motion despite ill-posedness, occlusions, and significant motion.

  • Method

    MotionCrafter uses a unified world-coordinate representation of point maps and scene flows with a shared 4D VAE and revised normalization.

  • Results

    MotionCrafter achieves state-of-the-art joint geometry and motion reconstruction without post-processing, outperforming pairwise approaches on video sequences.

  • Takeaways & Limitations

    Strictly aligning 4D latents with original RGB diffusion latents is unnecessary, while relaxed alignment and VAE retraining preserve or improve generalization.

  • Takeaways & Limitations

    The framework focuses solely on dense geometry and motion reconstruction rather than integrating multiple geometric modalities.

Abstract

from arXiv · show

We present MotionCrafter, a framework that leverages video generators to jointly reconstruct 4D geometry and estimate dense motion from a monocular video. The key idea is a joint representation of dense 3D point maps and 3D scene flows in a shared coordinate system, together with a 4D VAE tailored to learn this representation effectively. Unlike prior work that strictly aligns 3D values and latents with RGB VAE latents-despite their fundamentally different distributions-we show that such alignment is unnecessary and can hurt performance. Instead, we propose a new data normalization and VAE training strategy that better transfers diffusion priors and greatly improves reconstruction quality. Extensive experiments on multiple datasets show that MotionCrafter achieves state-of-the-art performance in both geometry reconstruction and dense scene flow estimation, delivering 38.64% and 25.0% improvements in geometry and motion reconstruction, respectively, all without any post-optimization. Project page: https://ruijiezhu94.github.io/MotionCrafter_Page

1. Introduction

MotionCrafter addresses feed-forward reconstruction of 4D scene geometry and dense motion from monocular dynamic-scene videos using a unified world-centric representation and pretrained video-generator priors. It avoids pairwise processing, post-optimization, and strict RGB-VAE value alignment through a compact 4D latent representation and canonical point-map normalization.

  • Limitations of prior work: Prior approaches rely on pixel correspondences, iterative per-scene optimization, or separate geometry and correspondence tasks, limiting generalization.These methods can produce sensor-limited noise and treat inherently related subtasks independently.
  • Limitations of prior work: Pairwise feed-forward methods such as St4RTrack, Dynamic Point Maps, and Stereo4D require post-optimization, reducing long-range motion coherence.They extend static reconstruction networks to dynamic scenes through target-timepoint map prediction.
  • Proposed framework: MotionCrafter jointly reconstructs 4D geometry and estimates dense point motion from long monocular videos in a feed-forward manner without post-optimization.It represents dynamic scenes with sequences of point maps and corresponding scene flows in world coordinates.
  • Motivation: Pretrained video generators alleviate the scarcity of large-scale in-the-wild datasets with dense geometry and motion while providing spatiotemporal consistency priors.The generator is trained on billions of visual data.
  • Representation: A unified 4D latent representation combines point maps and scene flows, transferring video-generator priors without cost volumes or dense pixel correspondences.The compact representation jointly models dense geometry and motion.
  • Normalization strategy: Canonical point-map normalization centers 3D coordinates and scales them by the scene’s mean scale instead of strictly matching the diffusion model’s original VAE value range.The paper argues that strict alignment is unnecessary because 4D geometry distributions differ substantially from natural-image distributions.

2. Related Work

Prior 4D reconstruction progresses from optimization-based NeRF methods toward feed-forward and video-generator-based approaches, but existing methods retain computational or modeling limitations. Scene-flow estimation similarly evolves from local pixel-space correspondence methods toward global matching, while video priors still often require per-scene optimization for 4D reconstruction.

  • 4D Scene Reconstruction: Early 4D reconstruction methods iteratively optimize 4D representations for monocular or multi-view videos, including time-dependent NeRFs for deformable scenes.These approaches suffer from expensive volumetric rendering.
  • 4D Scene Reconstruction: Feed-forward methods adapt static reconstructors such as DUSt3R to dynamic scenes, but DUSt3R-based approaches process pairs of frames at a time.π3 instead builds an architecture on top of VGGT for static and dynamic 3D reconstruction.
  • 4D Scene Reconstruction: Geo4D uses video generators to infer 4D point maps, while 4DGT and BTimer predict dynamic 3D Gaussian representations without explicitly modeling dense point correspondences over time.These methods represent distinct feed-forward directions for dynamic 3D reconstruction.
  • Scene Flow Estimation: Classical scene-flow methods estimate dense pixel-wise correspondences with coarse-to-fine strategies, which may fail under large motions or occlusions.GMFlow reformulates optical-flow estimation as global matching rather than local regression.
  • 4D Scene Reconstruction: Video-generator-based 4D reconstruction either generates multi-view videos before per-scene fitting or distills 4D priors directly, but these approaches still rely on iterative per-scene optimization.Their effectiveness is attributed to rich priors learned from large-scale image or video datasets.

3. Method

MotionCrafter represents monocular-video geometry and dense motion jointly as viewpoint-invariant point maps and forward scene flows in the first frame’s world coordinate system. It uses a unified 4D VAE with revised normalization and dedicated training to encode this representation and transfer video-generator priors effectively.

  • Unified 4D Representation: The network predicts a viewpoint-invariant 3D point map for each frame and forward 3D scene flow between consecutive frames in a shared world coordinate system.The final frame has no corresponding flow prediction and is not supervised for forward flow.
  • Unified 4D Representation: World-coordinate geometry and motion remove camera-induced motion components, avoid additional camera-pose estimation, and promote temporal consistency.Joint modeling also supports richer motion modeling than methods that do not use this unified representation.
  • 4D VAE: The 4D VAE jointly encodes geometry and motion into a unified latent space instead of modeling motion independently with a separate VAE.Combining the Geometry VAE and Motion VAE enables integrated 4D scene encoding and decoding.
  • VAE Training: Canonical mean normalization preserves point-map scale invariance and improves Geometry VAE reconstruction and fine structural detail, especially for large-scale outdoor scenes.It replaces max normalization to [−1, 1] and uses a small ε for numerical stability.
  • VAE Training: The revised strategy fine-tunes the full encoder-decoder with point-map, projected-depth, and surface-normal supervision, while strict diffusion-distribution alignment and KL regularization are unnecessary or harmful.The proposed normalization and training improve both the VAE and downstream diffusion U-Net; KL divergence caused a significant performance drop.

4. Experiments

MotionCrafter is evaluated on unseen dynamic-scene geometry datasets and diverse in-domain and out-of-domain motion datasets using world-coordinate metrics. Experiments report strong joint and geometry reconstruction results, while ablations show that mean rescaling with full VAE fine-tuning improves reconstruction over strict max-rescaling strategies.

  • Datasets: Geometry evaluation uses zero-shot testing on DDAD, Monkaa, and Sintel, spanning real-world and synthetic indoor and outdoor scenes.Motion evaluation uses Kubric, Spring, VKITTI2, Dynamic Replica, and Point Odyssey; the latter two provide only sparse scene-flow annotations.
  • Metrics: Evaluation is performed in world coordinates using relative point error, inlier percentage δp at threshold 0.25, End Point Error, and APD.Point clouds are aligned with ground truth using per-sequence scale and shift, while scene flow is aligned according to point-map scale.
  • Joint Geometry and Motion Reconstruction: MotionCrafter directly outputs world-coordinate prediction sequences, avoiding the post-optimization or camera-pose alignment required by most pairwise DUSt3R-based methods.For fair comparison, existing methods are evaluated after the prescribed pose-based transformations, while MotionCrafter remains without post-optimization.
  • Geometry Reconstruction: MotionCrafter achieves state-of-the-art geometry reconstruction on Monkaa, but underperforms VGGT on Sintel and DDAD.The paper attributes the outdoor gap to its single-modal design and limited outdoor training data, while reporting coherent reconstructions without post-optimization.
  • Ablations: Mean rescaling combined with fine-tuning all VAE components outperforms max-rescaling and decoder-only fine-tuning for geometry reconstruction.The ablation finds that strict normalization to [−1, 1] can produce suboptimal pretrained-VAE reconstruction accuracy, and Geo4D’s decoder-only strategy remains inferior.
  • Ablations: The original pretrained VAE reconstructs indoor geometry and motion reasonably, but struggles with the scale discrepancy among point maps, scene flow, and image-space distributions.This motivates tightly coupling geometry and motion representations for coherent 4D modeling.

5. Conclusion

MotionCrafter jointly reconstructs dense geometry and motion from monocular video using a unified world coordinate system and shared 4D VAE latent space. The framework achieves state-of-the-art performance without post-processing, while multimodal integration remains a promising direction.

  • MotionCrafter jointly reconstructs dense geometry and motion from a monocular video.
  • A unified world coordinate system and novel 4D VAE encode geometry and motion into a shared latent space.
  • State-of-the-art performance is achieved without any post-processing.
  • Strictly aligning the 4D latent distribution with the original SVD latent distribution is unnecessary.
  • Limitations: Incorporating multiple geometric modalities is identified as a promising direction for future work.Potential modalities include camera parameters, point maps, depth maps, point tracks, and novel views.

A. Data Processing … B. Additional Ablations

The supplied passages describe a preprocessing pipeline that unifies geometry and motion in the first camera’s world-coordinate system, then normalizes their global scale for training. They cover camera-pose normalization, point-map and scene-flow transformation, and scale recovery, but provide no passage on additional ablations.

  • A. Data Processing: The pipeline performs camera-pose normalization, world-coordinate transformation of point maps and scene flow, and global normalization of geometry and motion.All sequences are referenced to the first camera pose.
  • A.1. Camera Pose Normalization: Camera poses are aligned to a canonical frame defined by the first camera to remove arbitrary global rotation and translation.The normalization preserves relative motion within each sequence.
  • A.2. Point Map Transformation: Point maps are transformed from each frame’s camera coordinates into the first-frame coordinate system using the normalized camera poses.Invalid pixels are padded for VAE feature extraction but are not supervised.
  • A.3. Scene Flow Transformation: Scene flow is converted into first-frame world space by deforming points and transforming them with the next frame’s camera pose.A deformability mask can zero scene flow in non-dynamic regions.
  • A.4. Global World-Coordinate Normalization: Global normalization centers valid points and applies isotropic rescaling based on their mean scale relative to the centroid.The same affine transformation is applied to point maps, camera poses, and scene flow.
  • A.4. Global World-Coordinate Normalization: The isotropic scaling preserves geometric structure while normalizing absolute scale across datasets.Normalization parameters (µ, S) are stored to optionally recover the original metric scale.

B.1. Ablation on the Multimodal Supervision

MotionCrafter avoids auxiliary outputs and post-refinement at inference while using depth-based multimodal supervision during VAE training to improve its 4D latent representation. This supervision projects predicted and ground-truth point maps into depth and applies complementary depth losses, yielding substantial gains in point-map and depth-map reconstruction.

  • Depth supervision: Predicted and ground-truth world-coordinate point maps are projected into the depth domain using the normalized camera pose.The projection is defined as ˆD = Π(ˆX, ˜P) and D = Π(X, ˜P).
  • Depth supervision: The supervision combines a masked per-pixel L1 depth loss with a multi-scale patch depth loss over scale factors {4, 16, 64}.Patch-wise mean-depth subtraction removes global bias and suppresses depth-shift artifacts.
  • Motivation and impact: Depth-based multimodal supervision improves the learned 4D latent representation without changing inference or adding test-time modalities.The method derives depth supervision from world-coordinate point maps.
  • Results: 13.55% improvement in point map and 16.41% in depth map are achieved with depth-based multimodal supervision.These gains are reported in the ablation results.

B.2. Ablation on the Decoder Loss

The decoder loss directly supervises the VAE decoder output, providing stronger training signals to the UNet than latent regression. Incorporating it consistently improves UNet training, with average gains across four unseen datasets and especially large improvement on DDAD.

  • Motivation and implementation: The decoder loss directly supervises the VAE decoder output, providing stronger training signals to the UNet than latent regression.The decoder’s weights remain frozen, while gradients are back-propagated through it to update the UNet; gradient checkpointing reduces memory use.
  • Results: 15.01% average improvement across four unseen datasets demonstrates that incorporating the decoder loss consistently improves UNet training.The result is reported in Tab. 6.
  • Results: 36.80% performance improvement on the outdoor dataset DDAD marks the decoder loss’s particularly notable gain.DDAD shows the strongest improvement highlighted in the ablation results.

B.3. Ablation on the training paradigm · C. Implementation Details · C.1. Hyperparameter

MotionCrafter compares deterministic and denoising training paradigms, finding the deterministic approach superior for dense prediction. The implementation specifies separate prediction procedures, training datasets, and loss weights for the 4D VAE and pretrained video diffusion U-Net.

  • B.3. Ablation on the training paradigm: The framework supports deterministic and denoising diffusion paradigms on top of the pretrained SVD model.Because 4D reconstruction is deterministic, the deterministic paradigm is used by default.
  • B.3. Ablation on the training paradigm: In deterministic training, video latents directly enter the U-Net to predict 4D latents without decoder loss.In denoising training, noise is added to 4D latents, video latents are channel-wise concatenated, and multi-step U-Net denoising predicts the 4D latents.
  • B.3. Ablation on the training paradigm: 12.4% lower Relp and 12.7% higher δp result from the deterministic paradigm versus diffusion paradigms, averaged across datasets.This demonstrates deterministic training effectiveness and shows SVD prior knowledge can transfer without denoising.
  • C.1. Hyperparameter: The 4D VAE uses weights 1.0 for point map reconstruction, per-pixel L1 depth, and multi-scale depth supervision, plus 0.2 for normal consistency.These weights are specified as λpoint = 1.0, λL1-D = 1.0, λPatch-D = 1.0, and λnormal = 0.2.
  • C. Implementation Details: The training datasets are summarized in an overview table, with subsets sampled from some datasets to balance training.The passage does not specify the dataset names or subset sizes.
  • C.1. Hyperparameter: The scene flow reconstruction loss weight is 1.0, while the scene flow regulation loss weight is 0.01.These are listed among the 4D VAE training loss weights.
  • C.1. Hyperparameter: The pretrained video diffusion U-Net uses latent regression, point map decoder, and scene flow decoder loss weights of 1.0.Decoder loss is optional.

C.2. Used Training Set · C.3. Model Information

The paper documents its training datasets and visual samples, then describes a Stable Video Diffusion-based pipeline with video encoding, 4D decoding, and spatiotemporal latent denoising. On a single 40 GB GPU, the full reconstruction pass takes 138.9 ms per frame for 25-frame, 320 × 640 clips.

  • C.2. Used Training Set: Training datasets are listed in Table 8.Visual samples from the used training data are provided in Figure 7.
  • C.2. Used Training Set: Visual samples of the training data are shown in Figure 7.The datasets themselves are listed in Table 8.
  • C.3. Model Information: The system adopts the VAE and video UNet backbone from Stable Video Diffusion.These components provide the foundation for the geometry–motion reconstruction pipeline.
  • C.3. Model Information: The pipeline uses a video VAE encoder to encode per-frame latent representations.This is the first of three major pipeline components.
  • C.3. Model Information: The 4D VAE decoder reconstructs geometry and motion fields from latent space.This is the second major component of the pipeline.
  • C.3. Model Information: The 3D spatiotemporal UNet performs latent denoising.The reported Video UNet contains 1524.62M parameters.
  • C.3. Model Information: 138.9 ms per frame is the total processing time for the full forward pass.Measurements use a single 40 GB GPU on 25-frame videos at 320 × 640 resolution.
  • C.3. Model Information: 73.5 ms per frame is required for VAE decoding.The corresponding averages are 52.0 ms for VAE encoding and 13.4 ms for latent denoising.

C.4. Evaluation Metrics

The evaluation metrics align monocular geometry predictions to ground truth using per-sequence scale and shift, then measure relative point accuracy and scene-flow errors after shared scaling.

  • Geometry Alignment: Geometry predictions are aligned to ground truth with an optimized per-sequence scale s and shift t to address monocular scale ambiguity.The alignment minimizes an optimization objective over the predicted and ground-truth world-space point maps.
  • Geometry Metrics: Relative Point Error measures the relative geometry error between aligned predicted and ground-truth point maps.It is the primary continuous metric for geometry reconstruction accuracy described in the evaluation definitions.
  • Geometry Metrics: Inlier Ratio (δp) reports the percentage of points whose relative error is below τ = 0.25.The threshold used in the experiments is 0.25.
  • Scene Flow Alignment: Scene-flow predictions are scaled using the same geometry scale s before motion reconstruction is evaluated.This applies the geometry alignment scale to the predicted scene flow.
  • Scene Flow Metrics: End-Point Error averages endpoint differences, while APD measures the percentage of scene-flow vectors whose error falls below threshold γ.These metrics respectively capture average flow error and the proportion of sufficiently accurate flow vectors.

D. More Visualization Results

Zero-shot Davis results and qualitative comparisons show that MotionCrafter directly reconstructs geometry and motion across diverse scenes without post-optimization. It outperforms or matches existing methods in scene-flow accuracy, geometric structure, motion changes, and multi-view consistency.

  • Qualitative comparisons: The qualitative comparisons cover joint geometry-and-motion estimation and geometry reconstruction alone across multiple datasets and methods.Comparisons are shown against Zero-MSF, DELTA, VGGT, Geo4D, and ST4RTrack.
  • Zero-shot results: Zero-shot Davis results show that MotionCrafter generalizes well across different scene types despite limited scene-flow training samples.Its end-to-end design directly outputs results without post-optimization.
  • Joint geometry and motion estimation: MotionCrafter achieves scene-flow estimation accuracy comparable to Zero-MSF in one case and significantly outperforms existing methods in other cases.The comparable result occurs without training on the dynamic replica dataset used by Zero-MSF.
  • Geometry reconstruction: For moving objects and outdoor scenes, MotionCrafter estimates more accurate scale, motion changes, and scene structure than the compared geometric methods.It directly outputs world-coordinate point clouds without post-optimization and uses a much smaller training scale than VGGT.
  • Geometry reconstruction: On zero-shot generalization, MotionCrafter produces better multi-view consistency, smoother Geometry, and fewer stray spots than ST4RTrack.This comparison is qualitative and focuses on geometric output characteristics.
Loading 2602.08961v2…