PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuROK: Generative 4D Neural Object Kinematics

Chen Geng, Guangzhao He, Yue Gao, Yunzhi Zhang, Shangzhe Wu, Jiajun Wu

arXiv:2605.30347v1cs.CVcs.GR

TL;DR

Generating plausible 4D object dynamics without category-specific physical priors remains difficult. NeuROK learns a latent kinematic space and decoder, demonstrating generalizable and effective simulation across diverse object types.

  • Problem

    Existing 4D dynamics methods rely on category-specific physical models, limiting generalization beyond predefined categories and scalability to diverse large-scale datasets.

  • Method

    NeuROK learns a latent kinematic state space whose vectors decode into plausible object deformations, enabling dynamics generation through Lagrangian mechanics.

  • Results

    The framework generates 4D dynamics across diverse objects, generalizes to unseen categories, and preserves basic energy conservation.

  • Takeaways & Limitations

    NeuROK provides a data-driven framework for simulating object-centric physical systems without heuristic priors or physical annotations.

  • Takeaways & Limitations

    The framework assumes one plausible physical configuration can be generated from a static snapshot and specified physical conditions.

Abstract

from arXiv · show

Data-driven approaches have revolutionized 3D vision, enabling transformers to effectively reconstruct and generate static 3D objects. However, generating simulative 4D dynamics -- realistic temporal deformations of static objects under various physical conditions -- remains challenging and often ad hoc, despite its importance in building comprehensive 3D world models. Most existing methods assume a predefined physical model and use system identification to estimate parameters, restricting these methods to specific categories and small-scale datasets. We propose that these restrictions can be overcome by learning a data-driven kinematic state parameterization for object-centric physical systems. Specifically, we learn both a latent space representing all possible states of the object and a decoder that maps any sampled latent to a plausibly deformed shape of the object. We refer to this parameterization as Neural Object Kinematics (NeuROK), and learn a transformer-based encoder-decoder model on a curated large-scale 4D dataset. This formulation and the learned model significantly simplify the generation of simulative dynamics since we only need to consider the dynamics within a low-dimensional latent space from the Lagrangian mechanics' perspective in classical physics. We demonstrate the effectiveness and generality of this neural simulation framework across diverse dynamic object types, showing clear advantages over prior works. Project page: https://chen-geng.com/neurok

1. Introduction

NEUROK addresses the challenge of generating category-agnostic 4D object dynamics by learning a latent kinematic state parameterization from 4D geometric trajectories. Its transformer-based framework simplifies simulation to low-dimensional latent dynamics and demonstrates broad generalizability across dynamic objects.

  • Generating 4D reactive behaviors from a 3D snapshot without category-specific physical priors remains difficult, despite its importance for 3D world models, embodied AI, and robotics.
  • Existing methods use predefined category-specific physical models and system identification, limiting generalization beyond articulated objects, continuum bodies, and cloth.
  • NEUROK learns an automatically discovered latent space in which any sampled vector decodes into a plausible deformation of the modeled object.
  • The framework models transitions between low-dimensional latent vectors using category-agnostic energy functions from a Lagrangian-mechanics perspective.
  • A transformer encoder-decoder learns latent distributions and deformation fields from 4D geometric trajectories, enabling simulation without physical or action annotations.
  • The authors curate a large-scale 4D object dataset and report superior generalizability and effectiveness across a wide range of objects compared with existing methods.The paper describes this as the first data-driven framework for simulating object-centric physical systems without heuristic priors or physical annotations.

2. Related Work

Prior 4D generation methods typically rely on predefined physical models and system identification, while reduced-order, dynamic-system, and deformation-prior methods are often limited in efficiency, data, generalization, or category scope. NeuROK instead formalizes kinematic state parameterization as a general interface for physically inspired 4D generation.

  • Physically-Inspired 4D Generation: Physically inspired 4D generation typically first specifies a domain-specific physical model and then estimates its parameters through system identification.Existing work models rigid objects directly and elastic objects with methods including MPM and projective dynamics.
  • Reduced-Order Simulation: Reduced-order simulation primarily targets efficiency by accelerating existing systems with known physical constraints, rather than versatility in category-agnostic settings.These methods commonly learn instance-specific reduced kinematic spaces instead of generalizable amortized inference on large datasets.
  • Machine Learning for Dynamic Systems: Physics-agnostic dynamic-system methods often learn end-to-end dynamics from synthetic action-state pairs, but struggle to generalize to real-world objects because action-labeled data are scarce.The passage contrasts physics-agnostic and physics-aware approaches beyond 3D vision.
  • Neural Deformation Priors: Learned neural deformation priors are mostly category-specific and commonly support character animation or embodied-agent control rather than physically inspired 4D generation.Examples span humans, faces, and animals.
  • Neural Deformation Priors: NeuROK formalizes kinematic state parameterization as a general interface for physically inspired 4D generation.The paper presents this formulation as having broad potential beyond the category-specific and task-specific prior work discussed above.

3. Overview

NeuROK generates plausible 4D object dynamics by learning a low-dimensional kinematic state parameterization from static geometry and physical conditions. Its neural decoder maps latent states to valid deformations, enabling generic dynamics modeling in latent space.

  • Framework inputs and outputs: The pipeline takes a static 3D object snapshot and physical conditions, then generates a temporally evolving sequence of 3D shapes.Physical conditions include actions, forces, and initial velocities; the output is one plausible 4D sequence satisfying a valid physical configuration.
  • Kinematic representation: Plausible deformations form a low-dimensional configuration manifold Vkint within the full vertex-position space R3n.Randomly sampled deformation vectors in R3n will almost certainly produce shapes outside the distribution of valid object poses.
  • Kinematic representation: A k-dimensional kinematic state parameterization is a pair (Z, F), where Z ⊆Rk is the state space and F maps latent states to vertex configurations.The mapping operates on objects represented by meshes with n vertices and outputs configurations in R3n.
  • Limitations of prior approaches: Prior geometry-derived parameterizations use high-dimensional solution spaces and category-specific equations, limiting modeling beyond designated object categories.These constraints are introduced to prevent under-determined dynamics in redundant parameterizations.
  • Neural Object Kinematics: NEUROK learns an instance-specific latent space and neural decoder whose range is Vkint, mapping sampled latent states to plausibly deformed object shapes.The encoder-decoder model infers this parameterization from the input object’s static geometry using a generative objective.
  • Latent-space simulation: Latent-space dynamics eliminate interparticle equations for preserving plausible shape and can be simulated generically through Lagrangian mechanics and Euler-Lagrange equations.The learned NEUROK serves as generalized coordinates, while dynamics are modeled between latent vectors in a low-dimensional space.

4. Generative Learning of NEUROK

NeuROK learns an instance-conditioned generative distribution over plausible deformation fields using a conditional variational auto-encoder with transformer-based encoders and decoder. Training on a large-scale 4D deforming-mesh dataset enables generalizable kinematic priors, followed by active-subspace reduction to a lower-dimensional latent space.

  • Generative formulation: NeuROK models an object’s latent state space by learning a generative distribution over all plausible deformation fields conditioned on its input mesh.The deformation field ϕ maps points in R3 to R3.
  • Model components: The conditional variational auto-encoder uses Econd to produce an instance-specific latent prior, EVAE to infer a deformation-conditioned posterior, and D to decode latent samples into deformed meshes.The prior is pM0(z), the posterior is qM0(z | ϕ), and D takes z and M0 as inputs.
  • Architectures: Transformer backbones, including a perceiver-based mesh encoder with cross- and self-attention, are used to scale the models to large datasets.Econd encodes sampled surface points into learnable-token features, while the decoder combines query points with latent tokens through attention blocks.
  • Training: Training jointly samples an object instance and two frames with shared topology, uses the first mesh as M0, and supervises reconstructed deformation from the sampled deformation.The dataset combines instances curated from existing works and physical simulation.
  • Latent reduction: Active Subspace Method reduces the learned VAE latent space Z ⊆ Rk to Q ⊆ Rkq, where kq ≪ k, by identifying directions influencing predicted deformation.The surrogate function is defined using the 2-norm of predicted deformation.

5. Generative 4D Simulation

NeuROK converts dynamic mesh generation into latent-space trajectory generation, where the latent space acts as the object’s configuration space. Lagrangian mechanics and learned decoding then produce plausible 4D dynamics from physical conditions and initial states.

  • Latent-space dynamics: Dynamic mesh sequences are generated by sampling a series of latent states z_i, which the learned decoder maps to plausibly deformed object shapes.The mapping F produces valid configurations within the studied object-centric physical system.
  • Latent-space dynamics: The latent space Z(M_0) functions as the system’s configuration space, with each latent vector z serving as generalized coordinates for Lagrangian dynamics.This formulation enables dynamics generation directly in the learned latent space.
  • Lagrangian simulation: For most studied systems, the Lagrangian is defined as L(z, ˙z) = T(z, ˙z)−V (z), combining kinetic energy T and potential energy V.The resulting Euler-Lagrange dynamics are solved numerically to obtain the latent trajectory.
  • Conditioning and initialization: Input conditions such as actions determine initial latent position and velocity by minimizing reconstruction errors for selected particle positions and velocities.The optimized (z_0, ˙z_0) initialize the numerical solution of the latent dynamics.

6. Experiments

Experiments show that NEUROK consistently outperforms existing methods for inverse kinematics and generates physically plausible 4D dynamics across diverse objects. Analyses further demonstrate real-scene simulation, energy consistency, generalization to unseen categories, and contributions from key design choices.

  • Inverse Kinematics: NEUROK consistently outperforms existing methods in inverse kinematics on PartNet-Mobility using Chamfer distance and volumetric IoU.The evaluation estimates a latent state that deforms an input object toward a target pose.
  • Physically-Inspired 4D Generation: A user study with 105 users evaluates action alignment and realism, alongside VBench and WorldScore metrics for generated-motion plausibility and visual quality.The evaluation addresses the inherent ambiguity of predicting 4D dynamics from 3D shapes.
  • Physically-Inspired 4D Generation: Existing baselines generalize poorly outside specialized domains, whereas NEUROK consistently generates the most physically plausible 4D motions across settings.Physically based methods handle certain material categories, while AnimateAnyMesh lacks fine-grained conditioning and struggles with rarely encountered object types.
  • Simulating Real Objects: NEUROK successfully simulates and manipulates real scenes, including the closing motion of a laptop on a desk.The pipeline scans a real scene and simulates object dynamics within it.
  • Analysis and Generalization: NEUROK preserves basic energy conservation through a physically inspired Lagrangian framework and generates plausible dynamics for entirely unseen object categories.A variant trained only on PartNet-Mobility categories generalizes to novel object types, while generated trajectories maintain approximately constant total energy.
  • Ablation Studies: Model reduction, training-data augmentation, and the deformation parameterization each contribute significantly to overall performance.These design choices are evaluated in the ablation studies.

7. Conclusion

NEUROK introduces a framework for generating 4D simulative dynamics from static 3D shapes by bridging physical principles with learned latent spaces, establishing a new paradigm for 4D visual generation.

  • Conclusion: NEUROK generates 4D simulative dynamics from static 3D shapes through a physically inspired formulation that bridges physical principles and learned latent spaces.The framework is presented as a new research paradigm in 4D visual generation.
Loading 2605.30347v1…