Source-linked AI summary
Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesis
Hongyuan Chen, Xingyu Chen, Youjia Zhang, Zexiang Xu, Anpei Chen
TL;DR
4D synthesis must recover static geometry and dynamic motion from limited, ambiguous monocular evidence. Motion 3-to-4 separates shape generation from motion reconstruction using a canonical mesh and predicts temporally coherent geometry in a feed-forward framework. It reports superior fidelity and spatial consistency on standard benchmarks and a new geometry-grounded dataset.
Problem
4D synthesis remains difficult because training data are limited and monocular video makes geometry and motion recovery inherently ambiguous.
Method
Motion 3-to-4 combines static 3D shape generation with motion reconstruction, learning compact motion latents and predicting vertex trajectories from a canonical reference mesh.
Results
Motion 3-to-4 achieves superior 4D geometric accuracy across CD and F-Score metrics compared with all baselines.
Takeaways & Limitations
The decomposition supports efficient motion representation learning, generalization to synthesized and real-world shapes, and motion retargeting from different sources.
Takeaways & Limitations
The method may produce vertex sticking artifacts and struggles with topology changes because it does not explicitly model mesh topology and uses first-frame reference geometry.
Abstract
from arXiv · showhide
We present Motion 3-to-4, a feed-forward framework for synthesising high-quality 4D dynamic objects from a single monocular video and an optional 3D reference mesh. While recent advances have significantly improved 2D, video, and 3D content generation, 4D synthesis remains difficult due to limited training data and the inherent ambiguity of recovering geometry and motion from a monocular viewpoint. Motion 3-to-4 addresses these challenges by decomposing 4D synthesis into static 3D shape generation and motion reconstruction. Using a canonical reference mesh, our model learns a compact motion latent representation and predicts per-frame vertex trajectories to recover complete, temporally coherent geometry. A scalable frame-wise transformer further enables robustness to varying sequence lengths. Evaluations on both standard benchmarks and a new dataset with accurate ground-truth geometry show that Motion 3-to-4 delivers superior fidelity and spatial consistency compared to prior work. Project page is available at https://motion3-to-4.github.io/.
1. Introduction
High-fidelity 4D assets capture both an object's static shape and dynamic motion, but their reconstruction and generation remain difficult because of large solution spaces and spatial-temporal complexity. Motion 3-to-4 addresses this challenge with feed-forward motion reconstruction, scalable processing, and a new geometry-grounded benchmark.
- 4D assets are important for virtual reality, cinematography, robotics, and simulation because they capture static shape and dynamic motion over time.
- Existing 4D pipelines often rely on lengthy per-instance optimization and inherit view inconsistencies from 2D generative models.
- Motion 3-to-4 decomposes 4D generation into static 3D shape encoding and dynamic motion reconstruction from monocular video and an optional reference mesh.
- A frame-wise transformer supports varying input-mesh resolutions and videos of arbitrary length.
- Motion-80 adds ground-truth motion, realistic renderings, and geometry, complementing evaluation on a widely used benchmark.
2. Related work
Prior 3D and 4D methods use optimization, multi-view generation, reconstruction, rigging, or generative priors, but dynamic reconstruction remains limited. Motion 3-to-4 instead combines 3D shape synthesis with motion reconstruction as surface-to-video alignment for feed-forward prediction.
- 3D Object Generation: Earlier 3D generation methods used GANs, 2D-prior distillation, or multi-view reconstruction, often requiring substantial time and struggling with cross-view consistency.
- 3D Object Generation: Recent diffusion-transformer methods directly generate explicit 3D representations from point-cloud or voxel-based latent structures.
- 4D Reconstruction: SfM and SLAM support 3D structure and camera-pose estimation but often degrade on dynamic sequences.
- 4D Reconstruction: Reconstruction-based paradigms cannot hallucinate occluded or unseen geometry, often producing incomplete surfaces or missing structures.
- 4D Generation: Earlier 4D approaches use multi-view or video priors, rigging, skinning, or SDS optimization, which can introduce artifacts.
- 4D Generation: Motion 3-to-4 reformulates 4D generation as 3D shape synthesis plus alignment between surface points and video pixels, avoiding post-processing alignment steps.
3. Motion 3-to-4
Motion 3-to-4 decomposes 4D synthesis into static shape generation and dynamic motion reconstruction, using a reference mesh to predict temporally consistent 3D motion. Its motion-latent and decoding modules support compact representations, flexible video lengths, and dense point-wise supervision.
- Overview: The method decomposes ill-posed 4D generation into static shape encoding and dynamic motion reconstruction from monocular video.It recovers complete motion flow and geometry, including visible and unseen surfaces.
- Motion Latent Learning: A reference mesh is encoded with sampled surface points, whose geometry and appearance are compressed into a fixed-length latent representation.Each sampled point includes a 3D coordinate, surface normal, and RGB color; learnable queries aggregate shape information through cross-attention.
- Motion Latent Learning: Alternating attention jointly refines shape and visual tokens, with the final representation retaining shared geometry and frame-specific motion information.The first K tokens of each frame’s final latent representation serve as its motion representation.
- Motion Latent Learning: Video features are combined with the shape latent to form frame-specific motion representations that preserve temporal ordering and support arbitrary sequence lengths.Pretrained DINOv2 features provide patch-level semantic information, while temporal embeddings and frame-wise processing distinguish frames.
- Motion Decoding: The motion decoder samples reference-mesh points and predicts their per-frame 3D positions relative to the reference shape.This preserves surface correspondences, supports prediction at arbitrary spatial locations and time steps, and yields temporally consistent 4D reconstruction.
- Training: Training minimizes mean squared error between predicted and ground-truth point positions using dense point sampling.Dense supervision encourages fine-grained surface correspondences and coherent motion across the mesh.
4. Experiments
Motion 3-to-4 is evaluated on Motion-80 and Consistent4D using geometric, appearance, and qualitative comparisons against feed-forward and optimization-based baselines. It achieves spatially and temporally coherent 4D reconstruction, generalizes to in-the-wild inputs, and supports motion transfer across differently shaped objects.
- Evaluation datasets: Motion-80 contains 80 held-out subjects with short and long sequences, while Consistent4D contains 7 videos of 32 frames evaluated from four novel views.Motion-80 includes 64 short and 16 long sequences; Consistent4D lacks ground-truth meshes, so rendering-based metrics are used.
- Evaluation protocol: The evaluation compares Motion 3-to-4 with L4GM, GVFD, and V2M4 using Chamfer Distance, F-Score, LPIPS, CLIP, FVD, and DreamSim.Geometry is aligned with ICP before comparison, and appearance is evaluated from target viewpoints rather than only the input view.
- Quantitative evaluation: Motion 3-to-4 achieves superior geometric accuracy across both CD and F-Score, whereas competing methods exhibit floating artifacts or temporally inconsistent motion.The method animates an explicit mesh using reconstructed scene flow, producing temporally coherent motion.
- Appearance evaluation: The method quantitatively outperforms baselines in CLIP and DreamSim and maintains plausible, stable results from non-orthogonal novel viewpoints.L4GM can benefit from the orthogonal-view protocol but exhibits ghosting from non-orthogonal views.
- Qualitative evaluation: Qualitative comparisons show more temporally coherent and structurally consistent motion than GVFD, L4GM, and V2M4 on Motion-80.The comparison renders all generated 4D assets from an orthogonal novel view.
- More results: The framework generalizes to real-world videos and generated animations, and transfers motion from source videos to target meshes with different shapes and appearances.Examples include transferring neck, body, and leg motions from a dragon video to chicken and robotdragon meshes.
5. Conclusion
Motion 3-to-4 decomposes 4D synthesis into static 3D shape generation and motion reconstruction, supporting efficient, generalizable, and flexible dynamic-object creation. The method nevertheless remains limited by vertex sticking and later-frame topology changes.
- The pipeline separates 4D generation into 3D shape generation and motion reconstruction, reusing off-the-shelf shape models while keeping the motion branch lightweight.This decomposition reduces the scale requirements for 4D training data.
- Motion reconstruction aligns surface points with video pixels, enabling local correspondence reasoning across synthesized and real-world shapes and diverse motion patterns.
- The framework can animate static articulated objects from video conditions, including motion retargeting from different sources.
- The geometry encoder may produce vertex sticking when object parts are not clearly separated in the reference mesh.
- Using the first-frame mesh as reference geometry makes later topology changes difficult to accommodate during monocular-video motion reconstruction.
A.1. Training and Inference Details
The implementation combines point-cloud shape encoding, frame-wise video features, per-point cross-attention, and chunked temporal processing. Training uses temporally corresponding surface samples and multi-scale sequence augmentation.
- The shape encoder processes N = 4096 sampled canonical-shape points using the 3DShape2VecSet architecture.It is selected for representational capacity and compression efficiency.
- Frozen DINOv2-ViT-B/14 extracts 256 patch tokens of dimension 768 from each 224 × 224 video frame.Temporal and spatial sinusoidal positional embeddings are added to the tokens.
- Per-point queries built from position, normal, and color attend to 64 motion tokens through QK-Norm cross-attention to predict motion trajectories.
- Inference processes vertices in chunks of 4096 and videos in windows of 256 frames, using stride 255 for longer sequences.Each sliding window retains the first frame to maintain temporal consistency.
- Training samples surface points at consistent barycentric coordinates across frames, enabling trajectory supervision with MSE loss.
- Twelve-frame training sequences use stride intervals of 1, 2, or 4 frames to expose the model to different poses and motion displacements.
- Without a reference mesh, Hunyuan 2.0 generates one from the first frame, followed by vertex mapping to produce a watertight textured mesh for animation.
A.2. Evaluation Details
Evaluation uses a held-out dataset with the front view as input and three orthogonal views for assessment. Long-sequence comparison is constrained when GVFD runs out of memory beyond 128 frames.
- The held-out evaluation uses the front view as input and the remaining three orthogonal views for evaluation.
- The front view is excluded from evaluation because L4GM can reconstruct that input view losslessly, which would disadvantage generation-based methods.
- GVFD lacks official long-sequence processing scripts, and sequences exceeding 128 frames produce out-of-memory failures on the evaluation machine.
A.3. Ablation Study
The ablation study evaluates architectural variants using trajectory MSE on a held-out dataset, while additional visualizations illustrate synthesis quality and generalization across synthetic and real-world videos.
- Ablation Study: Ablation variants are trained for 30,000 steps under identical settings to compare architectural choices.
- Ablation Study: Trajectory prediction is evaluated with mean squared error over time on the held-out dataset.Rec MSE is the squared error averaged across time steps within the [−0.5, 0.5] bounding box.
- More Results: Additional visualization results and a supplementary local video webpage provide qualitative views of the synthesis results.
- More Results: Figure 8 shows high-quality results across diverse synthetic-video object types and motion patterns.
- More Results: Despite synthetic-only training, Figure 9 reports good generalization to real-world videos.
B.3. Results of Existing 3D Model
Motion 3-to-4 extends existing static 3D assets into dynamic 4D content and demonstrates robustness across viewpoints, with visual results emphasizing fidelity and temporal coherence.
- Results of Existing 3D Model: Existing static 3D assets can be extended into dynamic 4D content, broadening the method’s potential use cases.The model supports animating existing 3D models from various sources.
- Results of Existing 3D Model: Consist4D provides viewpoint-diverse inputs that test robustness beyond the held-out dataset’s frontal views.Its varied input angles differ from the frontal views used in the held-out dataset.
- Results of Existing 3D Model: Held-out results demonstrate superior visual fidelity and temporal motion coherence in generated 4D content.The results include additional visual examples from the held-out dataset.
- Results of Existing 3D Model: Under non-orthogonal views, L4GM produces inconsistent multi-view results with severe ghosting artifacts.The comparison is shown in Figure 12, while L4GM is designed for orthogonal views.
B.5. More Results from the Held-Out Dataset
Additional held-out-dataset results show high visual fidelity and temporally coherent motion, alongside a visual comparison with state-of-the-art methods on Consist4D.
- More Results from the Held-Out Dataset: Held-out-dataset results demonstrate superior visual fidelity and temporal motion coherence.The examples further validate high-quality 4D content generation with realistic appearance and consistent motion over time.
- More Results from the Held-Out Dataset: The section includes a visual comparison with state-of-the-art methods on Consist4D.The comparison is presented in Figure 12.