PaperlayerView exact revision on arXiv

Source-linked AI summary

BLARM: Animating 3D Objects from Video via Blending Latent Rigid Motion Primitives

Pradyumn Goyal, Yizhak Ben-Shabat, Hsueh-Ti Derek Liu, Haomiao Jiang, Snehasish Mukherjee, Kyle Spence, Mark Stauber, Evangelos Kalogerakis, Yunze Zeng

arXiv:2608.31113v1cs.CV

TL;DR

Animating meshes from monocular video requires motion that is both sequence-adaptive and structured without relying on explicit rigs or high-dimensional vertex regression. BLARM learns video-conditioned rigid motion components and fixed vertex assignments to generate temporally coherent animations. The method reports improved geometric accuracy and temporal consistency across the evaluated benchmarks, while its compact components provide interpretable motion structure.

  • Problem

    Existing approaches are either insufficiently motion-adaptive when using fixed or geometry-only rigs, or high-dimensional and poorly aligned with animation workflows when predicting vertices directly.

  • Method

    BLARM predicts time-varying rigid motion components from monocular video and blends them using time-invariant vertex-to-component skinning weights on a fixed mesh.

  • Results

    BLARM improves geometric accuracy and temporal consistency compared with several animation baselines, achieving the best average performance on all three reported benchmarks.

  • Takeaways & Limitations

    A compact learned rigid-motion subspace can produce geometrically plausible, temporally coherent, and rig-free mesh animations while preserving fixed vertex correspondences.

  • Takeaways & Limitations

    Performance is limited when skinning assignments are incorrect or when the canonical mesh topology cannot reasonably support the target motion.

Abstract

from arXiv · show

We introduce BLARM, a feed-forward method for video-driven 3D mesh animation. Given a monocular video and a static object mesh, BLARM predicts a temporally coherent animated mesh whose motion follows the video. Rather than relying on explicit rigs or directly regressing high-dimensional vertex motion, we represent animation using a compact set of learned, time-varying rigid motion components and time-invariant vertex-to-component skinning weights. This yields a low-dimensional deformation space without requiring skeletons, cages, skinning weights, or rig annotations. Our architecture conditions geometry-derived deformation latents on video features through factorized spatial-temporal attention, then decodes rigid transformations blended by predicted skinning weights. Trained with trajectory reconstruction, entropy regularization, and motion-aware contrastive learning, BLARM produces accurate and temporally stable animations while recovering compact, interpretable motion structure from monocular video.

1 Introduction

BLARM addresses the challenge of animating static meshes from monocular video while preserving geometry, appearance, temporal coherence, and editability. It introduces a motion-adaptive intermediate representation between fixed rigs and high-dimensional vertex-space prediction.

  • Animation remains challenging because dynamic assets must preserve geometry and appearance while remaining temporally coherent and usable.
  • Geometry-only or fixed rig structures may fail to represent sequence-specific deformations observed in a target video.
  • Direct vertex-space prediction can model complex deformation but is high-dimensional and poorly aligned with standard animation workflows.
  • BLARM represents motion with learned, time-varying rigid components and time-invariant vertex-to-component skinning weights.
  • The proposed representation provides a low-dimensional, motion-adaptive deformation space without predefined skeletons, cages, or rig annotations.

2 Related Work

Related work spans dynamic representations, direct video-driven vertex deformation, classical rigging, and motion-subspace methods. BLARM combines feed-forward video conditioning with compact learned rigid components while retaining fixed mesh correspondences.

  • Gaussian splats and dynamic mesh generators model time-varying assets, but Gaussian representations are less directly compatible with standard mesh animation and simulation.
  • Video-driven vertex-space methods are expressive and rig-free, yet their high-dimensional predictions provide limited inductive bias for coherent part motion.
  • BLARM predicts compact rigid transformations and time-invariant skinning weights, constraining deformation while retaining feed-forward video conditioning.
  • Per-frame mesh generation requires registration or post-processing for temporal correspondences, whereas BLARM preserves correspondences through a fixed mesh.
  • Motion-subspace methods improve scalability and temporal regularity, but many rely on predefined structures or sequence-specific optimization.
  • BLARM learns latent rigid components, predicts their transformations from video, and assigns vertices to them with time-invariant weights.

3 Method

BLARM animates a canonical mesh from monocular video by factorizing motion into compact, time-varying rigid components and time-invariant vertex assignments. Geometry-aware deformation latents are compressed from mesh features, conditioned on video features, and decoded into transformations blended with learned skinning weights.

  • Overview: BLARM takes a canonical mesh and monocular video, then estimates a temporally coherent animated 3D mesh.The canonical mesh may be provided or reconstructed, while experiments can use ground-truth meshes to isolate animation quality.
  • Motion Representation: The representation separates time-varying rigid motion components from time-invariant vertex-to-component assignments.This factorization describes how the object moves separately from which mesh regions each component affects.
  • Motion Representation: Learned latent components replace predefined skeletons, cages, and rig annotations while preserving a compact, object-specific motion basis.The approach targets objects that may not conform to human-designed kinematic templates.
  • Motion Representation: Each latent component decodes to a time-varying SE(3) transformation, with the first component reserved for global root motion and the others representing root-relative local motion.Rotations lie in SO(3), translations in R^3, and the non-root components form the local motion basis.
  • Blended Motion: Linear blend skinning combines the predicted rigid transformations using non-negative, normalized weights that remain fixed on the canonical mesh.The formulation blends non-root transformations in the root-local frame before applying the global root transformation.
  • Architecture: The architecture compresses dense geometry features into a compact J × D set of deformation tokens before conditioning them on frame-wise video features.Learnable queries aggregate sampled surface geometry, and subsequent attention injects frame-specific visual evidence into the geometry-aware latents.

4 Experiments

BLARM is evaluated against feed-forward mesh animation baselines on ActionBench and Motion80 using quantitative and qualitative comparisons. It achieves stronger geometric and appearance-based results while producing more temporally stable animations.

  • Experimental setup: BLARM is compared with ActionMesh, Mesh4D, and Motion3-to-4, which take video and a canonical mesh as input without predefined rigs.Unlike these baselines, BLARM predicts root-relative rigid transformations blended with time-invariant skinning weights.
  • Qualitative results: Qualitatively, BLARM better preserves canonical mesh structure and avoids the jitter, part distortion, and inconsistent deformation seen in baselines.The comparison spans diverse object shapes on ActionBench.
  • Quantitative results: BLARM achieves the best average performance on ActionBench and Motion80, including geometric and appearance-based metrics.The reported metrics include CD-3D, CD-4D, CD-M, LPIPS, CLIP similarity, FVD, and DreamSim.
  • Quantitative results: BLARM improves geometric alignment and visual stability, with FVD gains reflecting reduced temporal artifacts in rendered videos.Lower CD-3D and CD-4D indicate better frame-level sequence matching, while CD-M reflects more stable temporal motion.
  • Qualitative results: Predicted skinning-weight maps often align with coherent moving regions, suggesting structured, part-aware decompositions learned without explicit rig supervision.Examples include bending shoe parts and feather groups.

5 Discussion and Limitations

BLARM’s learned skinning weights often recover coherent moving regions without rig supervision, while the method’s behavior depends on assignment quality and mesh topology. The paper also points to appendices for ablations, implementation details, metrics, limitations, and broader impact.

  • Discussion: Learned vertex-to-component assignments align with coherent moving regions, indicating structured, part-aware decompositions without rig or skinning supervision.The assignments are visualized by coloring vertices according to their learned component associations.
  • Limitations: Incorrect assignments between nearby or visually similar regions can produce less coherent animations in challenging cases.This is identified as a limitation of the learned skinning weights.
  • Limitations: The method assumes the canonical mesh topology can reasonably support the target motion; violating this assumption may produce artifacts.This scope boundary concerns compatibility between input topology and desired deformation.
  • Additional material: Appendix A analyzes component contributions through ablations, while Appendix B reports evaluation details, datasets, additional results, and limitations.Appendices C and D cover implementation and motion-aware contrastive-loss details, respectively.
  • Additional material: Appendix E discusses the broader societal impact of the proposed method.

A Ablations

Ablations show that semantic vertex features, sparse skinning assignments, contrastive supervision, and an appropriate primitive count jointly improve structured, temporally coherent animation.

  • Component ablations: Removing PartField features, entropy regularization, and contrastive learning increases CD-3D, CD-4D, and CD-M to 2.95, 5.78, and 13.53.The ablated model also collapses toward using very few non-root latent components.
  • Component ablations: The direct per-vertex offset baseline remains worse than the full model, particularly on CD-M, supporting structured latent motion for temporally coherent deformation.It improves over collapsed latent variants but lacks the compact latent rigid-motion formulation.
  • Component ablations: The full model achieves CD-3D 1.71, CD-4D 3.07, and CD-M 7.15 on ActionBench.Removing major components substantially worsens reconstruction and motion accuracy.
  • Component ablations: PartField features reduce CD-M from 12.90 to 10.51, while entropy regularization further reduces it to 9.48 in the controlled setting.These components encourage meaningful part alignment and sparse, confident vertex-to-component assignments.
  • Number of motion primitives: J = 31 provides the best balance between deformation capacity, sparsity, and representation compactness.Too few primitives restrict deformation capacity, whereas larger counts increase fragmentation or active latents without clear gains.
  • Qualitative comparison: BLARM better maintains part structure and produces more stable deformations over time than baselines on Motion80.The resulting animations more closely match the ground-truth motion.

B Experiments

The experiments evaluate geometric reconstruction, sequence consistency, motion, and rendered appearance for predicted textured mesh sequences.

  • Geometric metrics: CD-3D measures per-frame reconstruction accuracy after aligning each predicted mesh to its corresponding ground-truth frame with ICP.The metric computes Chamfer Distance after per-frame alignment.
  • Geometric metrics: CD-4D measures sequence-level consistency using one global rigid alignment estimated from the first frame and Chamfer Distance averaged across the sequence.This differs from CD-3D by enforcing a single sequence-level alignment.
  • Appearance metrics: Appearance evaluation renders each predicted textured mesh sequence from four evenly spaced viewpoints.When ground-truth textured sequences are available for ActionBench and Motion80, they are rendered from the same viewpoints for video comparison.

B.2 Evaluation Datasets

The evaluation spans ActionBench, Motion80, and in-the-wild Consistent4D, using canonical meshes and textures when available and reconstructed inputs otherwise.

  • ActionBench: ActionBench supplies animated-asset identifiers rather than textured meshes, so corresponding Objaverse GLBs are retrieved and rendered for 16 frames.All methods use the first-frame ground-truth mesh as the canonical input, retaining original textures for appearance evaluation and qualitative comparisons.
  • Additional comparisons: Additional ActionBench comparisons are presented against feed-forward mesh animation baselines.These results are shown in Figure 5.
  • Motion80: Motion80 provides animated meshes with textures, using the first-frame mesh as the canonical input and retaining the provided texture.The same canonicalization is applied to all methods.
  • Consistent4D: Consistent4D contains 12 in-the-wild video sequences without ground-truth meshes, requiring first-frame shape and material reconstruction with TRELLIS2.The same reconstructed canonical mesh is provided to all baselines.
  • Consistent4D: For Consistent4D, rendered animations are evaluated against the input video from four uniformly spaced viewpoints.Because ground-truth meshes and camera parameters are unavailable, appearance metrics are averaged over the rendered views.

B.3 Additional Results

Additional results show that BLARM preserves geometry and temporal stability across benchmark and in-the-wild settings, while achieving favorable appearance metrics on Consistent4D.

  • Qualitative results: BLARM produces more temporally stable animations, preserves geometrical structure, and reduces visible distortions and part entanglement relative to baselines.On ActionBench and Motion80, its motion more closely follows the ground-truth sequence.
  • Qualitative results: On Consistent4D, BLARM obtains favorable LPIPS, FVD, and DreamSim scores while performing comparably on CLIP.Qualitative results suggest that the animation maintains reconstructed canonical structure over time.

B.4 Limitations

BLARM can produce stable, visually faithful mesh animations, but incorrect assignments between visually similar regions can entangle independently moving parts.

  • Incorrect vertex-to-component assignments in visually similar regions can entangle nearby parts and reduce motion coherence.The limitation arises when adjacent regions look similar but should move independently.
  • BLARM achieves the best LPIPS, FVD, and DreamSim scores among feed-forward mesh animation baselines on Consistent4D.These results indicate improved visual fidelity and temporal consistency on in-the-wild sequences.
  • Inference runtime is reported for 16-frame videos on an RTX 4080.The supplied passage identifies the evaluated video length and hardware but does not provide runtime values.

C Implementation Details

The implementation normalizes sequences, encodes sampled canonical surface points, and processes video-conditioned latent motion through factorized attention and rigid-motion decoders.

  • Preprocessing: Training sequences are centered and uniformly scaled using the first frame, with the same transformation applied to later frames.The maximum side length of the first-frame bounding box is normalized to one.
  • Preprocessing: The model samples 10,000 canonical-mesh surface points and uses coordinates and normals for positional encoding.A frozen Triposg shape encoder provides geometry features.
  • Architecture: The architecture uses 31 non-root motion components plus one root component, with hidden dimension 512 throughout.Visual features come from a frozen DINOv3 backbone at 512 × 512 resolution.
  • Architecture: The motion encoder stacks cross-attention to visual tokens with spatial and temporal self-attention.Spatial attention operates across latent motion components within each frame, while temporal attention models frame-to-frame structure.
  • Decoders: Two-layer MLP decoders separately predict rotation and translation for rigid transformations.The softmax temperature is 1 for both skinning-weight prediction and contrastive loss.
  • Training and inference: Training uses point-offset supervision with Adam for 1000 epochs, while inference processes 16 frames and applies autoregression to longer videos.Training takes approximately 1.5 days on 8 NVIDIA H200 GPUs; the factorized design is reported as fastest among compared methods.

D Motion-aware Contrastive Loss

The motion-aware contrastive loss organizes skinning weights by trajectory similarity, combining compact motion descriptors with spatially informed positive and negative sampling.

  • Motivation: The loss encourages points with similar ground-truth motion trajectories to receive similar predicted non-root skinning weights.It complements entropy regularization, which promotes sparse assignments but does not enforce spatial or kinematic coherence.
  • Trajectory descriptor: Each trajectory descriptor encodes translational direction, rotational direction, motion magnitude, and whether the point is static.Descriptors are normalized before pairwise similarity computation.
  • Limitation: Incorrect assignments in visually similar regions can entangle nearby parts and produce less coherent motion.This failure mode is shown as a limitation of the resulting motion decomposition.
  • Pair selection: Trajectory similarity is computed by averaging cosine similarity over time, then thresholded to define positive and negative sets.The thresholds are θ+ = 0.9 for positives and θ− = 0.5 for negatives.
  • Sampling strategy: Canonical-space distance determines sampling probabilities after motion similarity identifies candidate pairs.Nearby motion-consistent points are preferred as positives, while negatives include spatially distant easy negatives and nearby hard negatives.
  • Sampling strategy: The sampling strategy promotes global separation between unrelated motions and sharp local transitions between neighboring parts moving differently.Figure 8 illustrates the contrastive triplet construction using positives, hard negatives, and easy negatives.
  • Contrastive objective: A symmetric InfoNCE objective compares predicted non-root skinning-weight distributions for sampled positives and negatives.The objective uses a temperature parameter and separates vertices exhibiting different motion.
Loading 2608.31113v1…