Source-linked AI summary
Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video
Zeren Jiang, Chuanxia Zheng, Iro Laina, Diane Larlus, Andrea Vedaldi
TL;DR
Monocular 4D reconstruction requires recovering complete 3D shape and motion from partial video observations. Mesh4D uses a skeleton-guided deformation VAE, spatio-temporal attention, and latent diffusion to predict full animations, achieving state-of-the-art reconstruction quality for geometry, correspondence, and novel-view synthesis while reducing temporal artifacts.
Problem
Monocular videos reveal only parts of dynamic objects, while accurate reconstruction requires complete 3D shape and motion beyond visible geometry.
Method
Mesh4D combines a first-frame 3D mesh with a compact deformation VAE, skeleton-guided training, spatio-temporal attention, and latent diffusion to predict complete animations.
Results
Mesh4D achieves state-of-the-art reconstruction quality for geometry, correspondence, and novel-view synthesis on Objaverse while reducing temporal artifacts.
Takeaways & Limitations
The approach supports complete 4D mesh reconstruction and vertex tracking from monocular video without requiring skeletal information at inference time.
Takeaways & Limitations
The method relies on a high-quality canonical mesh and training skeletons, cannot represent topology changes, and struggles with extremely non-rigid objects.
Abstract
from arXiv · showhide
We propose Mesh4D, a feed-forward model for monocular 4D mesh reconstruction. Given a monocular video of a dynamic object, our model reconstructs the object's complete 3D shape and motion, represented as a deformation field. Our key contribution is a compact latent space that encodes the entire animation sequence in a single pass. This latent space is learned by an autoencoder that, during training, is guided by the skeletal structure of the training objects, providing strong priors on plausible deformations. Crucially, skeletal information is not required at inference time. The encoder employs spatio-temporal attention, yielding a more stable representation of the object's overall deformation. Building on this representation, we train a latent diffusion model that, conditioned on the input video and the mesh reconstructed from the first frame, predicts the full animation in one shot. We evaluate Mesh4D on reconstruction and novel view synthesis benchmarks, outperforming prior methods in recovering accurate 3D shape and deformation.
1. Introduction
Mesh4D addresses the difficulty of recovering complete 3D shape and motion from monocular video by combining a compact, skeleton-guided deformation representation with feed-forward prediction. It also introduces a benchmark centered on 3D shape and motion reconstruction.
- Monocular 4D reconstruction must infer complete 3D shape and deformation despite videos showing only partial object geometry.
- Existing feed-forward methods largely reconstruct visible geometry and pairwise correspondences, limiting their ability to capture complete 4D structure.
- The model targets accurate 3D shape and motion rather than only visually plausible rendering, addressing a gap in prior generative approaches.
- Mesh4D represents first-frame shape with a 3D mesh and subsequent motion with a deformation field, explicitly factorizing shape and motion.
- A compact VAE latent space encodes deformation across the whole sequence, using spatio-temporal processing and skeletal cues during training.
- The authors introduce a benchmark emphasizing 3D shape and motion reconstruction, alongside a framework for monocular 4D mesh reconstruction.
2. Related Work
Related work spans iterative optimization, pretrained generative priors, and feed-forward 4D reconstruction. Mesh4D is distinguished by jointly encoding long 4D mesh sequences and directly predicting dense temporal correspondences.
- Iterative NeRF and 3D Gaussian Splatting methods optimize time-dependent 4D representations, while feed-forward methods infer representations in a single pass.
- Several generative approaches prioritize plausible novel-view synthesis over accurate geometry and tracking.
- Existing feed-forward methods often focus on pairwise frames, stereo videos, or memory-based long-video reconstruction.
- ShapeGen4D predicts latent sequences but does not directly produce dense correspondences, whereas Mesh4D operates directly on 4D meshes.
- Compared with two-frame latent encoding, Mesh4D jointly encodes long 4D mesh sequences and predicts complete 4D meshes from monocular videos.
3. Method
Mesh4D reconstructs a canonical first-frame mesh and predicts its full deformation over a monocular video using a compact latent representation and conditional diffusion. Its deformation VAE encodes corresponding spatio-temporal points with attention and training-only skeletal information, while the generator conditions deformation on the video and canonical mesh.
- Pipeline: The model reconstructs the first-frame mesh with a pretrained image-to-3D generator, then predicts deformation for the full input video.The canonical mesh provides the reference geometry for the deformation model.
- Deformation VAE: The VAE uses masked skeleton-guided attention, spatial compression, and sequential temporal, global, and spatial attention to encode deformation.Skeleton information enters through skinning weights and bone features; the learned representation does not require skeleton inputs during inference.
- Deformation diffusion model: A conditional diffusion generator predicts the deformation latent from the canonical mesh and video, extending a pretrained static 3D diffusion model.The deformation diffusion model adds spatial and temporal conditioning and cross-attention to generate the deformation field.
- Deformation VAE: A deformation VAE compresses the sequence-wide deformation field into a compact latent code and decodes it back at canonical vertices.The encoder represents the deformation field across the animated mesh sequence, while the decoder reconstructs displacements from the latent code.
- Deformation VAE: Corresponding points are formed across time using mesh correspondence and barycentric coordinates, with normals and positional embeddings added to their features.Pairing points across frames helps represent motion, while normals provide local-motion information.
4. Experiments
Mesh4D is evaluated on geometry reconstruction, tracking, and novel view synthesis using a synthetic Objaverse benchmark, with qualitative and quantitative comparisons against established baselines. The experiments show benefits from sequence-wide modeling, spatio-temporal attention, and skeleton-guided deformation learning, while revealing a texture-related trade-off.
- Experimental settings: The benchmark contains approximately 9k animated instances, with 50 disjoint mesh sequences selected for testing and videos containing up to 100 frames.The dataset is derived from filtered Objaverse-1.0 assets with skeletons, skinning weights, mesh sequences, and vertex correspondences.
- Geometry and tracking evaluation: Mesh4D achieves state-of-the-art reconstruction and tracking performance by leveraging information across the whole input sequence.Compared with frame-wise HY3D inference, sequence-wide modeling improves pose and shape estimation.
- Novel view synthesis evaluation: Temporal and global attention produces pixel-aligned pose estimates and better PSNR, SSIM, and LPIPS than HY3D in novel view synthesis.The comparison concerns inaccurate pose estimation in qualitative results and the corresponding quantitative metrics.
- Novel view synthesis evaluation: Spatio-temporal attention models dense correspondences, yielding more consistent texture and geometry and a lower FVD than frame-wise alternatives.HY3D still exhibits jittery geometry and texture flicker despite shared noise.
- Novel view synthesis evaluation: GVFD and L4GM often produce inaccurate canonical meshes, while 3D-GS methods can exhibit ghost artifacts under large motion without explicit geometry and topology constraints.These limitations affect the quality of complete 4D reconstruction and novel view synthesis.
- Novel view synthesis evaluation: Using only the first input frame for texture yields a lower CLIP score than HY3D, although Mesh4D achieves state-of-the-art results on all other reported metrics.The reconstructed deformation field supports more accurate motion and alignment.
- Ablation study: Ablations show that removing skeleton information impairs rigid transformations, while removing temporal and global attention causes jittery motion and larger errors near the feet.These observations agree with the quantitative ablation results for the deformation VAE.
5. Conclusion
Mesh4D is a feed-forward monocular 4D reconstruction framework that predicts complete 3D shape and deformation across a video. Its compact deformation representation, skeleton-supervised training, and spatio-temporal modeling support strong reconstruction and novel-view results, while several scope limitations remain.
- The deformation VAE ablation indicates that skeleton information improves rigid-deformation capture and spatio-temporal fusion reduces jitter.
- Mesh4D combines a compact deformation VAE, skeleton-based supervision, and diffusion to track vertices throughout an entire video sequence.The model predicts the object's full 3D shape and deformation rather than independent frame reconstructions.
- On the Objaverse benchmark, Mesh4D achieves state-of-the-art quality for geometry, correspondence, and novel-view synthesis while reducing temporal artifacts.
- Limitations include reliance on high-quality canonical meshes and training skeletons, inability to represent topology changes, and difficulty with extremely non-rigid objects.
6. Implementation Details
Implementation uses a deformation VAE initialized from a pretrained static 3D shape VAE and performs inference from segmented, resized monocular video. The deformation diffusion stage uses Euler ODE sampling conditioned on the canonical shape and video features.
- The deformation VAE is initialized from HunYuan3D 2.1 shape-VAE weights and trained with AdamW at a learning rate of 1 × 10^-5.Training uses batch size 80, with M = 2048 initial points and N = 256 points after farthest-point sampling.
- Inference first segments the moving foreground and resizes it to the training ratio before canonical-shape reconstruction from the first frame.
- Classifier-free guidance is applied with guidance weight 5 during inference, although the ablation table reports slightly better results without CFG.
- Pretrained large-scale 3D-generator weights are evaluated as an initialization choice for the deformation diffusion model.
- The deformation diffusion model uses 50 first-order Euler ODE steps, conditioned on the canonical-shape latent and image features from the input video.
7. Additional Analysis
Additional analyses examine classifier-free guidance, pretrained initialization, and qualitative reconstruction views. The results support pretrained initialization and show that removing CFG slightly improves reconstruction quality.
- The model without classifier-free guidance achieves slightly better results than the version using CFG.
- Pretrained HunYuan3D 2.1 weights outperform training without them by a large margin across all reported metrics.The comparison trains both deformation diffusion variants for the same number of iterations.
- The ablation supports using large-scale 3D pretraining because existing 4D reconstruction datasets are limited in size.
- Qualitative results compare input-video frames with reconstructions rendered from four different views.
8. Visualization
Mesh4D generalizes across various object types and motions in the visualizations.
- Mesh4D generalizes well across various objects and motions in the shown reconstruction results.
9. Limitations
Mesh4D can fail when animation changes topology substantially or when the canonical mesh has incorrect topology or shape. An incorrect canonical mesh preserves the wrong topology through subsequent deformation.
- Large topology changes during animation can cause Mesh4D to fail.
- If the canonical reconstruction misses separate parts, subsequent deformation retains the incorrect topology and produces an incorrect 4D reconstruction.The cited failure case involves legs that are not separated in the first-frame reconstruction.
- Choosing a different reference frame and reconstructing deformation forward and backward can address some canonical-shape failures, but selecting that frame is outside this paper’s scope.