Source-linked AI summary
ShapeGen4D: Towards High Quality 4D Shape Generation from Videos
Jiraphon Yenphraphai, Ashkan Mirzaei, Jianqi Chen, Jiaxu Zou, Sergey Tulyakov, Raymond A. Yeh, Peter Wonka, Chaoyang Wang
TL;DR
Video-conditioned 4D generation seeks time-varying 3D geometry and appearance without fragile per-frame optimization. ShapeGen4D extends pretrained 3D generators with temporal attention, aligned latent sampling, and shared noise to directly generate dynamic meshes, yielding stronger generalization and higher-quality geometry than L4GM and GVFD while retaining some pose and jitter limitations.
Problem
Existing video-to-4D methods are limited by reconstruction errors, optimization-heavy pipelines, and insufficiently native direct generation of dynamic meshes.
Method
ShapeGen4D fine-tunes a pretrained 3D generator with spatiotemporal attention, temporally aligned point sampling, and shared noise to produce mesh sequences from video.
Results
The method demonstrates stronger generalization and higher-quality geometry than L4GM and GVFD, while producing consistent poses and substantially less temporal jitter.
Takeaways & Limitations
Direct mesh-sequence generation transfers large-scale 3D priors to video-conditioned 4D generation and supports varied dynamic geometry.
Takeaways & Limitations
The method is viewpoint-agnostic, struggles with global rigid rotations, requires additional registration and texture propagation for fully animatable assets, and can retain local temporal jitter.
Abstract
from arXiv · showhide
Video-conditioned 4D shape generation aims to recover time-varying 3D geometry and view-consistent appearance directly from an input video. In this work, we introduce a native video-to-4D shape generation framework that synthesizes a single dynamic 3D representation end-to-end from the video. Our framework introduces three key components based on large-scale pre-trained 3D models: (i) a temporal attention that conditions generation on all frames while producing a time-indexed dynamic representation; (ii) a time-aware point sampling and 4D latent anchoring that promote temporally consistent geometry and texture; and (iii) noise sharing across frames to enhance temporal stability. Our method accurately captures non-rigid motion, volume changes, and even topological transitions without per-frame optimization. Across diverse in-the-wild videos, our method improves robustness and perceptual fidelity and reduces failure modes compared with the baselines.
1 INTRODUCTION
Video-to-4D generation remains difficult because existing approaches are fragile, computationally expensive, or rely on multi-stage reconstruction and optimization. ShapeGen4D addresses this gap with a feedforward framework that directly generates dynamic 3D meshes using temporal modeling, aligned latents, and shared noise.
- Motivation: Existing SDS-based methods are fragile and computationally expensive, while multi-view reconstruction pipelines accumulate reconstruction errors and imperfections.These limitations motivate a native feedforward video-to-4D approach.
- Contributions: ShapeGen4D directly generates dynamic 3D meshes from video, rather than relying on separate networks or optimization-based pipelines.The framework extends a pretrained 3D generation model and produces mesh sequences directly.
- Motivation: Pretrained 3D generators provide knowledge from larger 3D datasets, helping the framework generalize despite relatively small 4D training data.The approach preserves strong 3D priors while adapting them to dynamic generation.
- Method: The method adds spatiotemporal attention, redesigned surface-point sampling, and shared noise across frames to improve temporal consistency and stability.These design choices adapt a pretrained 3D model to dynamic 4D generation.
2 RELATED WORK
Prior 4D generation methods use optimization, multi-view reconstruction, or limited representations, whereas direct 4D generation with high-quality dynamic meshes remains underexplored. ShapeGen4D builds on direct 3D generation to address this gap.
- SDS-based 4D generation: SDS-based methods can produce promising 4D results but remain computationally intensive and susceptible to artifacts such as Janus artifacts and oversaturation.These methods combine diffusion priors or specialized 4D representations to improve fidelity and coherence.
- Multi-View diffusion then reconstruction: Multi-view diffusion methods first generate images and then reconstruct 3D or 4D representations, but joint consistency across time and viewpoints is difficult.The approach requires video diffusion models that handle both temporal and viewpoint variation.
- Direct 3D generation: Direct 3D generators produce representations from images using diffusion models and learned latent tokenizations of unstructured point clouds.These models provide the foundation for extending direct generation from 3D to 4D.
- Direct 4D generation: Existing direct 4D methods suffer from limited quality, weak geometry, or complex registration and optimization pipelines.L4GM uses an image-based Gaussian representation, while V2M4 relies on per-frame meshes followed by registration and geometry optimization.
3 DYNAMIC MESH GENERATION
ShapeGen4D extends a pretrained 3D latent diffusion model to generate temporally consistent mesh sequences from video. Its core mechanisms align latent queries across deformation, model cross-frame dependencies, and stabilize diffusion with shared noise.
- Dynamic mesh generation: The model uses a flow-based latent diffusion transformer to generate mesh sequences from monocular video through temporally aligned latents and spatiotemporal processing.A pretrained 3D generator is extended from single-shape generation to dynamic shape sequences.
- Dynamic mesh generation: The VAE encodes meshes into latent codes and decodes them into truncated signed distance fields for mesh reconstruction.The decoder predicts signed-distance values at 3D positions from decoded shape features.
- Temporally-aligned latents: Independent query sampling creates jittery latents that make smooth temporal dynamics harder to learn.Each frame otherwise receives inconsistent query points, even at corresponding physical locations.
- Temporally-aligned latents: Query points are sampled from the first frame and warped through the animation so each latent sequence tracks the same physical surface points.Aligned latents substantially reduce jitter and improve diffusion training.
- Spatiotemporal diffusion transformer: Spatiotemporal transformer layers attend jointly across shape and image hidden states from all frames to capture temporal dependencies.These layers are inserted after blocks of the pretrained diffusion transformer.
- Spatiotemporal diffusion transformer: Independent noise can produce unstable poses because the base 3D model lacks explicit positional embeddings, so the method shares noise across frames.Shared noise reduces flickering and shape artifacts, especially when viewpoint ambiguity would otherwise cause pose changes.
4 MESH REGISTRATION AND TEXTURIZATION
After generating mesh sequences, ShapeGen4D applies global pose registration and global texturization to align geometry with the input video and produce temporally consistent appearance. These steps address canonical-coordinate mismatch and framewise texture inconsistency.
- Pipeline: The post-generation pipeline performs global pose registration followed by global texturization.These stages align generated geometry with the video and produce consistent textures.
- Noise sharing: Noise sharing reduces flickering and improves shape quality in challenging examples such as the flag.The comparison also shows that independent noise can cause arbitrary orientations and pose changes across frames.
- Global pose registration: Global pose registration reposes generated geometry from the canonical coordinate system to match the pose observed in the input video.The mesh generator is trained on objects aligned to coordinate axes, which may differ from the video pose.
- Global texturization: Independent per-frame texturization can produce inconsistent appearance, especially in occluded or unseen regions such as the backside.The method uses pairwise mesh registration to improve consistency across frames.
5 EXPERIMENTS
Experiments evaluate geometry, rendering, qualitative behavior, and component ablations against feedforward 4D baselines. The method achieves stronger geometric accuracy and temporal consistency, while rendering comparisons reflect viewpoint-alignment differences.
- Quantitative comparison: Consistent gains across all three geometric metrics show that the method produces more precise geometry than the baselines.The comparison uses the held-out Objaverse test set.
- Quantitative comparison: The method produces more consistent renderings than Step1X-3D and GVFD, although L4GM scores higher because its outputs are aligned with input views.This alignment favors view reconstruction at the expense of plausible 4D shapes from other viewpoints.
- Qualitative comparison: Qualitatively, the method generates high-quality meshes with consistent poses and substantially less temporal jitter than the baselines.L4GM shows ghosting and incorrect shapes, GVFD produces distortions, and per-frame Step1X-3D drifts across time.
- Ablations: Aligned latent queries are central: independently sampled queries degrade quality and increase flickering.The ablation evaluates removing components from the full method.
- Ablations: Shared noise reduces flickering and shape artifacts, while denoising time shifts improve result stability.Shared noise can also improve geometry in challenging cases; the time shift helps during denoising rather than training.
6 CONCLUSION
The work presents a feedforward video-to-4D model that generates temporally consistent 3D shapes and generalizes more strongly than recent Gaussian-splatting baselines. Its remaining limitations include viewpoint insensitivity, extra steps for fully animatable assets, and local temporal jitter.
- Conclusion: The feedforward model generates temporally consistent 3D shapes from input videos and shows stronger generalization than L4GM and GVFD.It produces higher-quality geometry than their 3D Gaussian-splatting representations.
- Conclusion: Figure 5 qualitatively compares the method with baselines on the held-out Objaverse test set.The supplied passage identifies the figure’s evaluation scope but does not state a specific visual winner.
- Limitations: The framework is viewpoint-agnostic, so it struggles with global motions such as rigid object rotations.Viewpoint-aligned reconstruction requires additional pose registration.
A MESH REGISTRATION AND TEXTURIZATION DETAILS
The mesh registration and texturization procedure aligns generated untextured meshes to the reference-video poses before applying texture.
- Inputs: The procedure takes a reference video sequence and generated untextured 4D geometry as inputs.The generated meshes are indexed across the video sequence.
- Pipeline: It first performs global pose registration and then global texturization, following a two-step process inspired by V2M4.The meshes are aligned to the poses in the reference video before texturization.
A.1 GLOBAL POSE REGISTRATION
Global pose registration estimates a camera pose for each generated mesh by matching rendered views to the corresponding video frame, then transforms the mesh into the video pose.
- A.1 GLOBAL POSE REGISTRATION: Camera pose estimation aligns each untextured mesh with its corresponding video frame by optimizing semantic similarity, predicted point-cloud alignment, and foreground-mask agreement.The estimated camera motion is inverted and applied to obtain a pose-registered mesh.
- A.1 GLOBAL POSE REGISTRATION: The method parameterizes each camera pose with yaw, pitch, radius, and look-at coordinates.
- A.1 GLOBAL POSE REGISTRATION: Candidate cameras are first sampled around the object and ranked by DreamSim similarity to the reference video frame.The top n candidates seed subsequent pose estimation.
- A.1 GLOBAL POSE REGISTRATION: VGGT refines pose estimation by aligning predicted point clouds from rendered candidates and the reference frame through a similarity transformation optimized with Chamfer distance.The resulting projection consistency yields an initial reference-camera estimate, which is further refined with particle swarm optimization.
A.2 GLOBAL TEXTURIZATION
Global texturization first converts the dynamic geometry into a topology-consistent mesh sequence, then reuses the initial mesh’s texture map and UV coordinates across later frames.
- A.2 GLOBAL TEXTURIZATION: Independent per-frame texturization causes appearance inconsistencies when topology varies across frames, motivating sequence-wide topology conversion before texturization.
- A.2 GLOBAL TEXTURIZATION: Topology-consistent conversion registers consecutive meshes globally and locally so later meshes share the initial mesh’s topology while retaining their target shapes.Global transformation uses Chamfer and differentiable rendering losses, while local registration adds ARAP-constrained preconditioned optimization.
- A.2 GLOBAL TEXTURIZATION: After topology alignment, the initial mesh is texturized once and its texture map and UV coordinates are reused for subsequent meshes.
B ADDITIONAL RESULTS
Qualitative comparisons show that ShapeGen4D produces temporally consistent shapes while preserving fine-grained motion details more reliably than the compared baselines.
- B ADDITIONAL RESULTS: ShapeGen4D produces the most temporally consistent shapes and captures fine-grained motion details in the Consistent4D qualitative comparison.The comparison is visualized in Figure 6.
- B ADDITIONAL RESULTS: Step1X-3D captures motion but exhibits jitter and random object poses, while GVFD often fails to model motion.
- B ADDITIONAL RESULTS: L4GM can generate high-quality results in some cases but fails on challenging water-splash examples with ghosting artifacts from misfused Gaussian splats.