Source-linked AI summary
RigidFormer: Learning Rigid Dynamics using Transformers
Zhiyang Dou, Minghao Guo, Haixu Wu, Doug Roble, Tuur Stuyck, Wojciech Matusik
TL;DR
Mesh-dependent, vertex-level simulators make high-fidelity rigid-body dynamics from point clouds difficult and computationally expensive. RigidFormer uses object-centric Transformer interactions with compact anchors, geometry-aware positional encoding, and differentiable rigid projection; it matches or surpasses mesh-based methods from point inputs while improving efficiency, generalization, and scalability. Its current scope is limited by reliance on object labels and learned contact handling, with severe partial observations remaining challenging.
Problem
Existing learned rigid-body simulators often require mesh connectivity and vertex-level interaction, limiting mesh-free point-cloud modeling and inference efficiency.
Method
RigidFormer models multi-object dynamics with object tokens and compact anchors, uses Anchor-based RoPE for unordered geometry, and projects updates onto the rigid-body manifold.
Results
RigidFormer matches or surpasses prior mesh-based methods using point positions while generalizing across resolutions and datasets, supporting step-size control and scaling beyond 200 objects.
Takeaways & Limitations
Object-level mesh-free simulation can maintain accuracy while reducing vertex-level computation and supporting efficient, scalable rigid-body modeling.
Takeaways & Limitations
The method assumes object labels, and severe partial observations are challenging when visible points capture too little of an object’s shape.
Abstract
from arXiv · showhide
Learning-based simulation of multi-object rigid-body dynamics remains difficult because contact is discontinuous and errors compound over long horizons. Most existing methods remain tied to mesh connectivity and vertex-level message passing, which limits their applicability to mesh-free inputs such as point clouds and leads to high computational cost. Efficiently modeling high-fidelity rigid-body dynamics from mesh-free representations, therefore, remains challenging. We introduce RigidFormer, an object-centric Transformer-based model that learns mesh-free rigid-body dynamics with controllable integration step sizes. RigidFormer reasons at the object level and advances each object through compact anchors; Anchor-Vertex Pooling enriches these anchors with local vertex features, retaining contact-relevant geometry without dense vertex-level interaction. We propose Anchor-based RoPE to inject anchor geometry into attention while respecting the unordered nature of objects and anchors: object-token processing is permutation-equivariant, and the mean-pooled anchor descriptor is invariant to anchor reindexing while preserving shape extent. RigidFormer further enforces rigidity by projecting updates onto the rigid-body manifold using differentiable Kabsch alignment. On standard benchmarks, RigidFormer outperforms or matches mesh-based baselines using point inputs, runs faster, generalizes to unseen point resolutions and across datasets, and scales to 200+ objects; we also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components.
1 Introduction
RigidFormer addresses the cost and representation limits of mesh-dependent, vertex-level simulators by modeling mesh-free rigid-body dynamics at the object level with compact anchors. It combines geometry-aware attention, rigid-manifold projection, and controllable integration, achieving efficient and scalable simulation from point representations.
- Motivation: Mesh-free point inputs avoid unavailable connectivity, but vertex-, edge-, and facet-level simulators become costly as resolution increases.Point clouds are topology-free and resolution-flexible, whereas many learned simulators require explicit mesh connectivity.
- Object-centric modeling: RigidFormer models interactions among object tokens rather than vertices, matching the coherent motion of rigid bodies and reducing computation.The object-centric design accepts partial point clouds and uses Transformers to capture multi-object interactions without hand-designed graphs.
- Efficiency: 23.9 FPS versus 3.0 FPS demonstrates the efficiency gain from shifting interactions from vertices to objects while maintaining simulation quality.The cited comparison reports inference speed for the object-centric design against vertex-level processing.
- Anchor-based state advance: Compact anchors represent each object’s low-dimensional rigid motion while retaining geometry-aware information for contact dynamics.Anchor-based state advancement exploits the 6-DoF structure of rigid-body motion, and Anchor-Vertex Pooling aggregates local point features invariantly to vertex ordering.
- Geometry-aware attention: Anchor-based RoPE encodes object geometry while preserving object-token permutation equivariance and anchor-order invariance.The mean-pooled anchor descriptor is invariant to anchor reindexing, while object processing remains equivariant to object permutations.
- Stability and scope: RigidFormer projects updates onto the rigid-body manifold and supports step-size control, generalization across point resolutions and datasets, and scaling beyond 200 objects.The pipeline uses Verlet integration followed by differentiable Kabsch alignment, and the reported evaluation includes point-only inputs and preliminary articulated-body extension.
2 Related Work
Prior rigid-body simulators include classical physics engines, differentiable mesh-based engines, graph-based neural models, and recent point-based approaches. RigidFormer differs by using object-level Transformer interactions for point-based dynamics.
- Physics-based simulators: Classical and differentiable simulators rely on explicit physics engines and typically assume mesh-based geometry rather than mesh-free point inputs.Classical methods solve constrained contact problems, while differentiable engines support gradient-based learning and inverse problems.
- Learning-based dynamics: Early learning-based dynamics models established object- and relation-centric inductive biases, motivating graph-based simulators for dynamics modeling.These models often targeted simpler systems with explicit low-dimensional states, frequently in 2D.
- Mesh-based neural simulation: State-of-the-art rigid-body neural simulators commonly use mesh inputs to capture dynamics faithfully, including mesh-discretized message passing.MeshGraphNets extend message passing to mesh discretizations and achieve strong mesh-based simulation performance.
- Point-based dynamics: Recent point-based methods use hierarchical point convolutions or disentangle visual observations from physical states, but contact-rich accuracy can degrade in the latter setting.RigidFormer instead uses object-level Transformer interactions for point-based dynamics.
3 Methodology
RigidFormer models mesh-free rigid-body dynamics with object tokens and compact anchors, enriching anchor updates with local contact geometry. It conditions Transformer interactions on step size and geometry, then projects predicted anchor motion to full point sets through differentiable rigid alignment.
- Object-Centric Interaction Modeling: Each object is encoded as one token from point-level features, shortening Transformer sequences relative to vertex-level modeling while supporting variable point counts.The shared hierarchical encoder captures fine and coarse geometry before object-token interaction.
- Object-Centric Interaction Modeling: Step-size FiLM conditioning adapts decoder features across temporal discretizations, while query-conditioned attention gates attenuate noisy interaction reads during rollouts.The step code uses s and s^2 to reflect first- and second-order temporal scaling.
- Anchor-based State Advance: Four FPS anchors reduce interaction complexity from O((MNv)^2) to O((MNa)^2), with Na ≪ Nv, before predicting per-anchor accelerations and advancing anchors with Verlet integration.Full-resolution points are recovered by broadcasting the estimated rigid transform to every vertex.
- Anchor-based State Advance: Anchor-Vertex Pooling aggregates nearby vertex features with a learnable distance kernel, adding collision-aware local context to each anchor without full vertex-level attention.The normalized pooling is invariant to vertex ordering and unchanged by common rigid transforms of points and anchors.
- Anchor-based State Advance: Differentiable Kabsch alignment projects candidate anchor positions onto a rigid transform, enforcing rigidity by construction and improving long-horizon rollout stability.The transform is applied to the full point set, with robust differentiability implemented using RoMa.
- Anchor-based Rotary Positional Embedding: Anchor-based RoPE encodes object spatial extent through mean-pooled anchor rotary descriptors, preserving object-token permutation equivariance and anchor-reindexing invariance.The encoding remains geometry-aware by depending on world-frame positions and capturing centroid and shape extent.
4 Experiments
Experiments evaluate RigidFormer on accuracy, generalization, efficiency, robustness, scalability, and controllability. Across these settings, point-based RigidFormer matches or improves prior methods while supporting variable resolutions, step sizes, partial inputs, and large or articulated systems.
- Main comparison: RigidFormer achieves the best orientation error in all reported MOVi columns and the best or second-best translation error in most columns using point inputs.On MOVi-B at 100 frames, it improves over HopNet from 0.176 m/17.91° to 0.161 m/15.33°.
- Generalization: Cross-dataset transfer consistently improves over FIGNet and remains competitive with HopNet, while larger step sizes reduce long-horizon errors in every transfer split.For MOVi-B-trained transfer, gains are more pronounced in orientation than translation.
- Resolution generalization: At 100 steps, unseen 768-point evaluation yields 0.137/11.13° with step size 10, 0.161/14.83° with step size 5, and 0.189/16.22° with step size 1.Training uses randomly sampled point counts {128, 256, 512, 1024}.
- Step sizes: Step size 10 gives the lowest 100-frame errors on MOVi-A, MOVi-B, and MOVi-Sphere, while step size 5 provides an intermediate rollout trade-off.Step size 1 remains the matched setting for comparison with prior one-step protocols.
- Scalability and controllability: Partial inputs remain stable after masking 25% of points, retaining accurate contacts and low long-horizon drift without retraining or specialized completion modules.RigidFormer also produces coherent heading-conditioned motion for ASE humanoid and Unitree G1 body-part components, and achieves 8× and 101× speedups over FIGNet and HopNet.
- Ablations: At 100 steps, gated attention reduces position error from 0.259→0.161, 0.191→0.136, and 0.152→0.115 for step sizes 1, 5, and 10.Differentiable rigid projection further reduces position error from 0.169→0.161, 0.146→0.136, and 0.121→0.115.
- Anchor robustness: Four anchors provide the default efficiency–quality trade-off; eight anchors can lower orientation error in some long-horizon settings but require twice as many anchor queries.Randomized FPS with four anchors gives the strongest translation accuracy.
5 Conclusion
RigidFormer is presented as an object-centric, mesh-free Transformer simulator that uses compact anchors, geometry-aware attention, and differentiable rigid projection. The conclusion reports efficiency, accuracy, generalization, scalability, and step-size control, while identifying object labels and insufficient partial shape observations as scope boundaries.
- Conclusion: RigidFormer models multi-object rigid-body contact dynamics from point clouds using object-level interaction and compact anchor-based state updates.The design reduces vertex-level computation while maintaining accuracy.
- Conclusion: Anchor-based RoPE injects 3D geometry while preserving object-token permutation equivariance and anchor-order invariance.The descriptor uses mean-pooled anchor features.
- Conclusion: Differentiable Kabsch alignment projects updates onto the rigid-body manifold, and step-size conditioning supports controllable integration within one model.These mechanisms are part of the simulator’s reported design.
- Limitations and future work: The formulation primarily targets object-level rigid-body contact dynamics and relies on object labels to identify point membership.Partial observations become challenging when they capture too little of an object’s shape.
A More Qualitative Results
Additional qualitative evaluations show RigidFormer rolling out stable multi-object interactions from partial and held-out point-cloud inputs. The visualizations span MOVi-A, MOVi-B, and MOVi-Sphere across four rollout frames per sample.
- Partial point clouds: With 25% of points masked inside each object’s bounding box, RigidFormer predicts accurate rigid-body interactions with low drift and stable contacts.Meshes are shown only for visualization; the model operates on point inputs.
- Held-out qualitative results: Figures A7–A9 show held-out MOVi-A, MOVi-B, and MOVi-Sphere samples, with each row representing one test sample and four rollout frames.The examples show stable multi-object rollouts across different geometries and contact configurations.
B Theoretical Properties of the Object-Anchor Representation
The object-anchor representation provides indexing guarantees for anchors, vertices, and objects, while rigid projection preserves intra-object distances. These guarantees do not imply full physical SE(3) equivariance because the simulator operates in world-frame coordinates.
- Anchor reindexing invariance: ARoPE’s mean-pooled descriptor is invariant to anchor ordering, and the same property holds for shared per-anchor maps followed by symmetric pooling.Mean pooling is used for simplicity and scale stability.
- Vertex-order invariance: AVP is invariant to vertex ordering because its normalized aggregation is unchanged by reindexing, and its weights depend only on point-anchor distances.The attention weights are also unchanged by common rigid transforms of an object’s points and anchors.
- Object permutation equivariance: Object-level decoder blocks are permutation-equivariant because shared attention, normalization, FiLM, gating, and feed-forward operations commute with object-token permutations.Register tokens remain a shared permutation-invariant workspace.
- Anchor prediction and projection: Shared anchor prediction and Kabsch alignment preserve object reordering equivariance and anchor reindexing invariance under consistent correspondences.The guarantees apply to the stated indexing choices and rigid projection.
- Rigidity by construction: Kabsch projection preserves intra-object distances exactly in the final output, independent of raw anchor accelerations predicted by the network.The full simulator is not SE(3)-equivariant because gravity, ground contact, and evaluation coordinates use the world frame.
C.1 Training Schedule
RigidFormer is trained directly on short and long temporal discretizations, using Smooth L1 objectives before and after rigid projection. The MOVi datasets span simple primitives and substantially more complex object geometries.
- Training schedule: Training uses sequence length T=8 with random integration steps ∆t ∈{1, 5, 10} sampled near-uniformly, without curriculum learning or scheduled sampling.The FiLM-conditioned model is exposed to short and long temporal discretizations throughout training.
- Training objective: Smooth L1 loss is applied element-wise at anchor locations and averaged over valid anchors, objects, and time steps.Compared with ℓ2 loss, Smooth L1 is less sensitive to occasional large rollout errors while remaining smooth near zero.
- Training objective: Position and acceleration losses are computed both before and after rigid projection, while full-resolution vertices are supervised indirectly through the induced rigid transform.The rigid projection uses predicted anchors after Kabsch alignment; acceleration targets follow the same Verlet discretization.
- Training objective: The reported experiments use λpos = 10 and λacc = 1, with multi-step residuals normalized by ∆t^2 before Smooth L1 reduction.This normalization matches the acceleration scale used by Verlet integration.
- Datasets: MOVi-A contains simple primitives, whereas MOVi-B includes 11 categories ranging from 51 to 1,142 vertices, representing substantially greater geometric complexity.The supplied dataset table references MOVi object types and dataset statistics.
D.1 Cross-Dataset Transfer Analysis
Cross-dataset transfer remains competitive across MOVi variants, with larger-step conditioning reducing long-horizon errors in every transfer split. Training-time rotation and object-permutation augmentation improve robustness to allowed scene transformations and input orderings.
- Transfer results: RigidFormer consistently improves over FIGNet and remains competitive with HopNet in matched step-size-1 cross-dataset transfer.The strongest results occur when training on MOVi-Sphere or MOVi-A.
- Transfer results: MOVi-B training transfers less well than MOVi-A or MOVi-Sphere training, averaging 0.221, 0.178, and 0.145 position errors at step sizes 1, 5, and 10.These averages are computed over 100-step position errors on two held-out target datasets.
- Augmentation: Random Z-axis rotations augment scenes while preserving gravity and ground-contact structure, targeting robustness to yaw rather than full 3D rotation invariance.The angle is sampled from 5° increments, and the whole sequence is rotated consistently.
- Augmentation: Randomly permuting object order during training checks and reinforces robustness to arbitrary input arrangements while preserving all per-object attributes consistently.The architecture is already permutation-equivariant for object tokens, and the augmentation is applied with 50% probability.
F.1 Runtime Performance
RigidFormer’s measured model core runs at approximately 54 FPS, while geometric processing lowers the full per-step rate to approximately 43 FPS. Object-level attention greatly reduces the quadratic attention term relative to hypothetical vertex-level attention.
- Evaluation setting: The runtime profile uses a 10-object MOVi-B scene with 4,016 vertices; the model has 174.8M parameters and a 1.80 GB peak memory footprint.The profile evaluates a 50-step rollout setting.
- Measured runtime: 18.61 ms per step corresponds to approximately 54 FPS for the model core, while geometric processing takes 23.25 ms per step, or approximately 43 FPS.The model core includes point encoding, object-state interaction, and anchor-object interaction.
- Measured runtime: CUDA KNN search consumes 22.80 ms per step and dominates geometric processing, whereas differentiable Kabsch takes only 0.45 ms.The reported shares are 54.5% for KNN search and 1.1% for Kabsch.
- Attention efficiency: Vertex-level attention would require approximately 49.5 GFLOPs for 4,016 vertices, compared with approximately 2.1 MFLOPs for object-level attention over 10 objects and 16 registers.This is a roughly 2.4×10^4 reduction in the quadratic attention term before projections and feed-forward networks.
G Detailed Network Architecture
RigidFormer combines hierarchical point encoding, object-level Transformer processing, anchor-based feature compression, and rigid projection. Its architecture is also used for large multi-object scenes and a preliminary command-conditioned articulated-body extension.
- Core architecture: The main architecture contains a hierarchical PointNet encoder, a four-layer object-level Transformer decoder with gated attention and register tokens, and a query-based anchor predictor.The reported model does not use spatial state tokens.
- Anchor processing: Anchor-Vertex Pooling forms normalized weighted sums of vertex features and projects them from 1024 to 256 dimensions before concatenation with anchor queries.The final projection layer is zero-initialized, and the resulting anchor feature has 256 channels.
- Anchor processing: Each anchor attends in parallel to four decoder representations, whose outputs are concatenated and fused back to D=768 before predicting per-anchor acceleration.The scales are the encoder-side input plus three internal decoder-layer outputs.
- Cross-object interaction: Anchor cross-attention uses keys and values from all valid objects, giving an anchor on one object direct access to contextualized states of other objects for contact reasoning.This adds an anchor-level cross-object pathway beyond object-level self-attention.
- Scalability: WreckingBall evaluates dense collision scenes with 28, 65, 126, and 217 objects arranged as cube grids struck by a spherical projectile.The four configurations use 3×3×3 through 6×6×6 cube arrangements.
- Scalability: Object-level tokenization changes attention complexity from vertex-based scaling to O(M^2) in object count, reducing WreckingBall-217 attention pairs by approximately 58×.The comparison is approximately 3.1M vertex-level pairs versus 54K object-level pairs.
- Articulated bodies: The articulated-body extension treats each body part as an object-level component and conditions the model on command signals.It covers a 17-part ASE humanoid with speed, direction, and facing controls, and a 31-part G1 robot with speed and direction controls.
- Conditioning and control: FiLM conditioning incorporates temporal codes [s, s^2] and task-specific control vectors, with zero-initialized modulation beginning as an identity.The controls have dimension 5 for ASE and 3 for G1, while larger integration steps support cheaper long-horizon planning signals.
K.1 Limitations and Future Work
The study identifies scope and deployment limits for point-cloud rigid-body dynamics, including object-label assumptions, difficult severe partial observations, learned contact handling, and distribution-shift risks.
- RigidFormer assumes object labels identifying which points belong to each object, constraining use with raw observations lacking segmentation.
- Severe partial observations may be challenging when visible points capture too little of an object’s shape.
- Future work includes stronger occlusions, real sensor noise, online object segmentation, mixed rigid–deformable scenes, and finer-grained adaptive time stepping.The work also describes efficient learned simulation as potentially useful for rollout prediction, interaction planning, and controllable physical reasoning.
- Contact handling is learned from data rather than solved with an explicit complementarity-based physics engine.The rigid projection step preserves intra-object distances while retaining a mesh-free, efficient design.
- Deployment requires validation under intended geometry, material, and contact conditions because reliability under distribution shift is the main consideration.High-stakes use should include uncertainty checks, safety constraints, and human or system-level oversight.