PaperlayerView exact revision on arXiv

Source-linked AI summary

ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion

Remy Sabathier, David Novotny, Niloy J. Mitra, Tom Monnier

arXiv:2601.16148v2cs.CV

TL;DR

Animated 3D generation remains constrained by restricted input setups, lengthy optimization, and insufficient production quality. ActionMesh uses temporal 3D diffusion and a temporal autoencoder to generate rig-free, topology-consistent animated meshes, achieving state-of-the-art geometric accuracy and temporal consistency with high-fidelity shape and motion in 2 minutes.

  • Problem

    Existing animated 3D generation methods often require restricted input setups, lengthy optimization loops, or outputs that do not meet production criteria.

  • Method

    ActionMesh extends pretrained 3D diffusion models with a temporal axis and uses a temporal 3D autoencoder to deform a reference mesh into a consistent animation.

  • Results

    ActionMesh achieves state-of-the-art geometric accuracy and temporal consistency while producing high-fidelity shape and motion in 2 minutes.

  • Takeaways & Limitations

    The model provides a fast, practical route to rig-free, topology-consistent animated meshes for downstream uses such as texturing and retargeting.

  • Takeaways & Limitations

    ActionMesh assumes fixed mesh connectivity, so it cannot model topological changes.

Abstract

from arXiv · show

Generating animated 3D objects is at the heart of many applications, yet most advanced works are typically difficult to apply in practice because of their limited setup, their long runtime, or their limited quality. We introduce ActionMesh, a generative model that predicts production-ready 3D meshes "in action" in a feed-forward manner. Drawing inspiration from early video models, our key insight is to modify existing 3D diffusion models to include a temporal axis, resulting in a framework we dubbed "temporal 3D diffusion". Specifically, we first adapt the 3D diffusion stage to generate a sequence of synchronized latents representing time-varying and independent 3D shapes. Second, we design a temporal 3D autoencoder that translates a sequence of independent shapes into the corresponding deformations of a pre-defined reference shape, allowing us to build an animation. Combining these two components, ActionMesh generates animated 3D meshes from different inputs like a monocular video, a text description, or even a 3D mesh with a text prompt describing its animation. Besides, compared to previous approaches, our method is fast and produces results that are rig-free and topology consistent, hence enabling rapid iteration and seamless applications like texturing and retargeting. We evaluate our model on standard video-to-4D benchmarks (Consistent4D, Objaverse) and report state-of-the-art performances on both geometric accuracy and temporal consistency, demonstrating that our model can deliver animated 3D meshes with unprecedented speed and quality.

1. Introduction

ActionMesh is a feed-forward generative model that produces production-ready animated 3D meshes from diverse inputs. It combines temporal 3D diffusion with a temporal 3D autoencoder to generate fast, rig-free, topology-consistent animations.

  • Contribution: ActionMesh generates production-ready animated 3D meshes from diverse inputs using a simple, scalable, feed-forward model.The model is introduced as computing 3D meshes “in action” from diverse inputs.
  • Method: Temporal 3D diffusion extends pretrained 3D latent diffusion with time to produce synchronized latents for time-varying, independent 3D meshes.The temporal axis separates 3D generation from animation prediction in the first stage.
  • Practical properties: 2 minutes for 16-frame video demonstrates ActionMesh’s speed while its rig-free, topology-consistent meshes support complex shapes and preserve attributes such as textures.The design also extends to generative tasks such as text-to-4D.
  • Method: A temporal 3D autoencoder deforms a reference mesh to assemble independent shapes into a single topology-consistent animation.This second component enables animation while preserving consistent topology.

2. Related Work

Prior work spans scalable 3D latent foundations, optimization-based video-to-4D reconstruction, feed-forward 4D generation, and animation-ready asset preparation. ActionMesh combines temporal 3D diffusion with a temporal 3D autoencoder to generate animated meshes with consistent topology in two stages.

  • 3D foundational models: 3DShape2VecSet introduced vecset-based neural fields for scalable mesh and point-cloud encoding, while Trellis and Craftsman extended structured latent modeling and 3D-native diffusion.Trellis decodes into radiance fields, Gaussian splats, and meshes; Craftsman couples latent-set diffusion with geometry refinement.
  • Video-to-4D via optimization: Optimization-based video-to-4D methods synthesize multi-view or multi-frame supervision before optimizing dynamic or deformable 4D representations.SV4D, SV4D 2.0, and CAT4D improve spatio-temporal consistency but require per-scene optimization.
  • ActionMesh: ActionMesh generates an animated mesh in two stages: temporal 3D diffusion produces independent time-varying shapes, and a temporal autoencoder predicts reference-vertex deformations with consistent topology.The first stage uses a reference mesh latent and video; the second converts the shape sequence into deformation fields of the reference mesh vertices.
  • Feed-forward 4D reconstruction: Feed-forward approaches avoid test-time optimization by directly predicting dynamic representations, including latent sets for surfaces and temporally upsampled Gaussian splats.Motion2VecSets reconstructs dynamic surfaces from point-cloud sequences, whereas L4GM predicts Gaussian-splat sequences from monocular video in one pass.
  • Animation-ready 4D assets: Animation-ready asset methods predict rigs, skinning weights, kinetic codes, or deformation fields, focusing on animation rather than reconstructing animated meshes from video.Examples include Make-It-Animatable, MagicArticulate, SMF, RigAnything, RigGS, and methods deforming meshes from video or text prompts.

3. Method

ActionMesh reformulates diverse animated-mesh generation tasks as video-to-4D generation, extending pretrained 3D latent diffusion with temporal synchronization and masked conditioning. It then represents the result as topology-consistent vertex deformations of a reference mesh.

  • Temporal 3D diffusion: Temporal 3D diffusion augments a pretrained 3D diffusion model with inflated attention and masked generation to produce synchronized 3D latents across frames.The method builds on the 3D latent diffusion framework of 3DShape2VecSet and adopts TripoSG as its backbone.
  • Terminology and problem setting: A 3D mesh is a triangular mesh M = (V, F), while a 4D mesh is a sequence of time-varying independent meshes that may have different topologies.An animated 3D mesh is a 4D mesh whose sequence shares one face connectivity F.
  • Terminology and problem setting: Given a monocular video, ActionMesh predicts a reference mesh and per-frame vertex updates whose motion matches the observed frames.The reference mesh is M = (V, F), while each frame uses updated vertices Vk and the shared connectivity F.
  • Temporal 3D diffusion: Inflated attention lets tokens attend across all frames while reusing pretrained attention layers, and rotary positional embeddings reduce jitter between consecutive frames.FlashAttention2 is used to reduce the (NT)2 complexity overhead of cross-frame attention.
  • Masked generation: Masked generation preserves selected noise-free 3D latents and denoises only the remaining latents, enabling known meshes to guide animation generation.During training, source latents use flow matching step 0 and receive no diffusion loss; during inference, clean encoded latents are copied before each denoising step.

4. Experiments

ActionMesh is evaluated on video-to-4D benchmarks and additional applications, achieving stronger quantitative and qualitative results than prior methods while enabling fast, temporally consistent animated meshes. Real-world, motion-transfer, and ablation experiments further examine robustness and key design choices.

  • Qualitative comparison: On Consistent4D, ActionMesh preserves the highest geometric fidelity across frames with strong temporal consistency, sharper motion fidelity, and fewer artifacts or drift.LIM and DreamMesh4D produce coarse geometries, while V2M4 and ShapeGen4D recover sharper details but exhibit artifacts and partial temporal drift.
  • Quantitative comparison: ActionMesh outperforms LIM, DreamMesh4D, V2M4, ShapeGen4D, TripoSG, and TRELLIS across all ActionBench metrics.ActionBench contains 128 animated scenes based on Objaverse and was released for reproducible evaluation.
  • Quantitative comparison: 5%, 35%, and 39% are ActionMesh’s improvements over the best prior results for CD-3D, CD-4D, and CD-M, respectively.Inference takes 2min versus 15–45min for prior works, with all methods evaluated on a single H100 GPU.
  • Additional applications: Despite training only on synthetic data, ActionMesh reconstructs accurate animated meshes from challenging DAVIS videos containing large-scale motions, subtle movements, multiple objects, and occlusions.Examples include a jumping horse and a walking bear.
  • Additional applications: ActionMesh transfers motion from a source video to a different target mesh without explicit task-specific training when semantic correspondences can be established.The paper demonstrates animating a 3D dragon using a video of a flying bird.
  • Ablation study: The ablation study finds that stage II preserves 3D reconstruction quality while enabling animated mesh prediction, whereas stage I is the critical component for the overall system.Removing both stages reduces the experiment to applying TripoSG independently to each frame.

5. Conclusion

ActionMesh is a fast, feed-forward model that generates topology-consistent, rig-free animated 3D meshes from diverse inputs using temporal 3D diffusion and a temporal 3D autoencoder. It delivers high-fidelity shape and motion while remaining limited by topological changes and strong occlusions.

  • Method: ActionMesh extends pretrained 3D diffusion models with a temporal axis and uses a temporal 3D autoencoder to deform a reference mesh consistently over time.The diffusion stage generates synchronized shape latents, while the autoencoder translates independent shapes into reference-mesh deformations.
  • Benefits: 2 minutes yields high-fidelity shape and motion, while topology consistency and rig-free outputs support rapid iteration, texturing, and retargeting.The model is presented as a simple, general, and practical path to production-ready animated 3D meshes.
  • Limitations and directions: Fixed connectivity prevents ActionMesh from modeling topological changes.A proposed direction is topology-aware latent updates that instantiate, fuse, or remove local parts without manual connectivity edits.
  • Limitations and directions: Strong occlusions can cause failures when regions are missing from the reference frame or during motion, despite the model’s ability to hallucinate unseen parts.These failure cases are illustrated in Figure 7.

A. Additional results

Additional results show that ActionMesh produces sharper, temporally cleaner, and more faithful reconstructions across benchmark and real-world videos. Its autoregressive rollout supports longer animations, while fixed-topology outputs preserve coherent textures across multiple animation applications.

  • Video-to-4D: On Consistent4D, ActionMesh produces sharper geometry, fewer temporal artifacts, and more faithful motion than competing approaches.Videos for all scenes and baselines are rendered from three viewpoints each.
  • Video-to-4D: Despite training exclusively on synthetic videos, ActionMesh generalizes to real-world DAVIS footage with sharp geometry and plausible motion.Additional qualitative results are provided for in-the-wild videos.
  • Long animation generation: 61-frame sequences are generated by recursively feeding each inference’s last 3D output into the next prediction.The model is trained on 16-frame sequences and extends them autoregressively after one standard inference pass followed by three additional rollouts.
  • 3D+text-to-4D: Fixed-topology animated meshes keep textures coherent and consistently propagated throughout sequences.This property supports animation from static textured meshes and motion descriptions, including benchmark and externally generated meshes.
  • Mesh Animation: ActionMesh integrates known 3D shapes into generation for 3D+video-to-animation and 3D+text-to-animation tasks.The paper qualitatively compares this capability with AnimateAnyMesh, a text-driven feed-forward model reported to animate meshes in approximately 6 seconds on one NVIDIA H100 GPU.

B. Additional ablation studies

Additional ablations show that temporal design choices affect reconstruction quality, supported applications, and efficiency across both stages. The final model uses masked conditioning, surface normals, a first-frame reference, and a context window of cw = 1 based on these studies.

  • Temporal 3D diffusion (stage I): Masked conditioning with noise-free image-to-3D latents broadens supported applications and improves video-to-4D geometric and motion fidelity over video-only conditioning.Disabling the mechanism removes support for {3D+text}-to-4D, {image+text}-to-4D, and autoregressive long-horizon generation, while underperforming the full model on video-to-4D reconstruction metrics.
  • Temporal 3D autoencoder (stage II): Removing surface normals from query-point representations reduces temporal 3D autoencoder performance.The default representation combines each query point’s position and normal to disambiguate spatially close but topologically distant surface points.
  • Autoregressive generation: Both temporal stages use sequences of 16 keyframes by default, with additional variants trained using 4 or 8 keyframes and evaluated on 16-frame sequences.The ablation evaluates pairwise combinations of stage I and stage II configurations for autoregressive generation.
  • Autoregressive generation: cw = 1 is used in both stages because enlarging the context window has limited reconstruction impact on 31-timestep sequences while increasing computational cost.Larger context windows also implicitly increase total inference steps because fewer new frames are generated per pass.
  • Reference frame: Using the middle or last frame instead of the first as the reference produces slightly worse quantitative results.The first frame is typically more favorable for 3D reconstruction, with fewer topological ambiguities and less motion blur.

C. Implementation details

ActionMesh is implemented with temporal diffusion and a temporal autoencoder trained on 13,200 animated sequences, using multi-view frames and canonical point-cloud trajectories as inputs. Evaluation uses normalized animated scenes, ICP3D/ICP4D alignment, and Chamfer-based metrics including CD-3D, CD-4D, and CD-M.

  • Inputs: The model trains on 13,200 animated sequences using 16 rendered viewpoints per keyframe and canonical point clouds with Np = 500,… points.Sequences contain at least 16 and up to 128 keyframes, with viewpoints sampled at uniformly spaced azimuths and elevations in [40◦, 85◦].
  • Inputs: Video preprocessing uses a sequence-wide union bounding box and consistent cropping, while deforming one canonical point cloud over time stabilizes temporal 3D diffusion training.The bounding box is resized to occupy 90% of the frame height or width, and point clouds are encoded into latents with a frozen E3D.
  • Temporal 3D autoencoder: The temporal 3D autoencoder follows TripoSG with 16 self-attention and 1 cross-attention DiT layers, trained on the same corpus and optimization setup using 16-frame clips.Its architecture uses 8 heads and 1024 width; training provides deformed point-cloud trajectories.
  • Quantitative evaluation: Evaluation normalizes 128 animated Objaverse scenes to a canonical cube, aligns predictions with ICP3D or ICP4D, and reports CD-3D, CD-4D, and CD-M.CD-3D averages frame-wise symmetric Chamfer distance under ICP3D; CD-4D uses one transform from the first frame, while CD-M propagates fixed first-frame correspondences.
Loading 2601.16148v2…