Choreographing a World of Dynamic Objects

• Scene-level 4D data describing both object deformation and interaction is extremely rare, limiting existing data-driven methods to generating single-object dynamics.Source: §1 · p. 2
• CHORD iteratively optimizes object-level Lagrangian deformations, rendering the scene from selected viewpoints for video models to judge motion plausibility and consistency.The video models serve as high-level choreographers for individual objects and their interactions.Source: §1 · p. 2
• Its hierarchical 4D representation separates fine motion details from coarse transformations and uses cumulative temporal ranges to encourage coherence and long-horizon learnability.The spatial representation is bi-level and control-point-based; the temporal structure is inspired by Fenwick trees.Source: §1 · p. 2
• CHORD derives an SDS target for modern rectified flow video models, addressing their incompatibility with traditional distillation algorithms.The paper identifies that incompatibility as a challenge for applying distillation to these models.Source: §1 · p. 2
• CHORD generates physically grounded Lagrangian trajectories for diverse real-world objects that can guide zero-shot robot manipulation.The paper presents a framework for generating physically grounded motions and applying them to real-world robotic manipulation policies.Source: §1 · p. 2

Object-Level 4D Generation

• Object-level methods rely on category-specific rigging or datasets dominated by human-like models, while video-distillation approaches can produce minor, unrealistic motion.CHORD assumes neither category-specific kinematic structure nor large-scale 4D datasets and generates motion for arbitrary objects.Source: §2 · pp. 2–3

Scene-Level 4D Generation

• Scene-level generation must coordinate plausible motion across interacting objects, but prior methods restrict categories, impose physical constraints, use symbolic structures, or yield largely 2.5D reconstructions.CHORD tackles scene-level 4D motion without category-specific inductive bias.Source: §2 · p. 3

4D Representations

• CHORD's hierarchical 4D representation reduces dimensionality and adds hierarchy in both spatial and temporal domains for generation with noisy supervision.Earlier high-dimensional deformation fields work well for dense reconstruction inputs but are unsuitable for generative tasks with noisy supervision.Source: §2 · p. 3

3.1. Preliminary: Score Distillation Sampling

• The method extends SDS to videos by adding noise to rendered 4D output and using diffusion-model gradients to update the representation.The preliminary SDS formulation weights its gradient by a function of noise level.Source: §3.1 · p. 3

3.2. Distilling from Rectified Flow Models

• For rectified-flow video models, CHORD derives an SDS target aligned with the model’s training loss to provide gradients for optimizing the 4D representation.The target uses noisy videos and the model’s predicted velocity.Source: §3.2 · p. 3
• Noise is sampled according to a domain-specific schedule and annealed from higher to lower levels, yielding more realistic motion when the RFSDS gradient weighting is removed.The decreasing noise level supports coarse motion early and finer deformation later.Source: §3.2 · pp. 3–4

3.3. Hierarchical 4D Representation

• Meshes are converted to canonical 3D-GS, while coarse and fine control points blend neighboring SE(3) transforms to encode broad motion and residual local detail.The fine layer is introduced later in optimization to add residual deformations.Source: §3.3 · pp. 4–5
• Fenwick-tree nodes store cumulative deformations over frame ranges, sharing parameters across overlapping intervals to improve temporal coherence and support long-horizon motion.A frame’s deformation is composed from the active nodes returned by the tree query.Source: §3.3 · p. 5

3.4. Regularization

• Temporal flow-map regularization encourages smoothness over time, while ARAP regularization on deformed near-surface points encourages local spatial consistency.The spatial term uses a point cloud sampled near each object’s surface.Source: §3.4 · pp. 5–6

4.1. 4D Scene Motion Generation

• The comparison covers six multi-object scenes and four baselines: Animate3D, AnimateAnyMesh, MotionDreamer, and TrajectoryCrafter-based 4D reconstruction.The evaluation includes a 99-participant user study and VideoPhy-2’s SA and PC metrics.Source: §4.1 · p. 6
• CHORD achieves the highest Semantic Adherence (SA) score and second-highest Physical Commonsense (PC) score among baselines; AnimateAnyMesh’s top PC score reflects static-object failures.Qualitatively, CHORD also shows stronger prompt alignment and more natural motion than the compared approaches.Source: Table 1 · p. 6, §4.1 · p. 6

4.2. Extensions and Applications

• Using the final deformation frame as the next input extends motion sequences, while the video model’s real-world training supports animation of scanned objects.Figure 6 presents real-world object animation results.Source: §4.2 · p. 6, Figure 6 · p. 7
• Generated dense object flow guides robot grasping or pushing of rigid, articulated, and deformable objects, with planning based on flow alignment, reachability, and pose smoothness.The manipulation results are shown in Figure 7.Source: §4.2 · p. 7, Figure 7 · p. 8

4.3. Ablation Studies

• Uniform noise-level sampling produces unrealistic motion, including a laptop appearing to float, because it insufficiently covers noise levels that inject motion.Figure 8 compares the proposed sampling strategy with uniform sampling with weighting.Source: §4.3 · p. 8, Figure 8 · p. 8
• Removing the Fenwick tree causes later-frame artifacts, while removing fine control points prevents detailed motion and starting with fine points too early introduces artifacts.The ablations also show that coarse control points are needed to avoid distortions.Source: §4.3 · p. 8, Figure 9 · p. 8
• Removing temporal regularization causes flickering, whereas removing spatial regularization causes visual distortions.Figure 10 illustrates these effects.Source: §4.3 · p. 8, Figure 10 · p. 9

5. Conclusion

• CHORD generates scene-level 4D object motion from 3D shapes and supports diverse natural phenomena and downstream robotic manipulation.The conclusion describes the approach as robust, scalable, and versatile.Source: §5 · p. 9

A.1. Pipeline Implementation Details

• The pipeline uses Wan 2.2 at 832 × 464 resolution and optimizes 41-frame deformation sequences; control points are initialized from mesh occupancy and optimized with a split training schedule.Each asset is trained for 2,000 iterations with batch size 4, requiring approximately 20 hours on an NVIDIA H200 GPU.Source: §A.1 · p. 13

A.2. Robot Manipulation Implementation Details

• Robot manipulation uses direct scans for the banana-picking and lamp-lowering cases, and dimension-matched digital cousins for the other objects.The digital cousins are created in Blender when accurate scanning is challenging.Source: §A.2 · p. 13

A.3. Baseline Implementation Details

• Baseline setups merge scene objects for Animate3D and AnimateAnyMesh, train a separate NJF per object for MotionDreamer, and use a regularized Fenwick-tree representation for TrajectoryCrafter reconstructions.The TrajectoryCrafter setup uses coarse control points with temporal and spatial regularization.Source: §A.3 · p. 13

B. Derivation of SDS for Rectified Flow Models

• Under uniform noise sampling, the Rectified Flow loss uses a latent linearly interpolated between the original latent and Gaussian noise.The appendix defines the noise as ϵ ∼ N(0, I) and zτ = (1 − τ)z + τϵ.Source: B. Derivation of SDS for Rectified… · p. 13
• The derivation differentiates the Rectified Flow loss with respect to z, then omits backpropagation through the RF model and applies the chain rule to the 4D parameters θ.This yields the RF-SDS gradient used in the main text.Source: B. Derivation of SDS for Rectified… · p. 13

C.2. Full Table for User Study

• Across all scenes, CHORD receives the highest preference for both prompt alignment and motion realism.The full user-study results are reported in Tables 2 and 3.Source: Table 2 · p. 15, Table 3 · p. 15, §C.2 · p. 15

C. More Experiment Results

• Table 2 records scene-level participant votes, while Table 3 reports quantitative comparisons for single-object 4D motion generation.Source: Table 2 · p. 15, Table 3 · p. 15

C.1. Comparison on Single Mesh Animation

• 89.6% of participants rated CHORD highest in prompt alignment, and 84% rated it highest in motion realism for single-mesh animation.The comparison included all baselines.Source: Table 3 · p. 15, §C.1 · p. 15

• Failure cases stem from video-generator limitations and the inability to create objects missing from the initial static scene.Source: §C.3 · p. 15, Figure 12 · p. 16
• When the video model repeatedly fails to sample prompt-consistent videos, its guidance can lead CHORD to produce incorrect motion.Source: §C.3 · p. 16, Figure 12 · p. 16
• Because CHORD deforms only geometry present initially, it cannot produce newly appearing objects such as liquid prompted to enter the scene.Source: §C.3 · p. 16, Figure 12 · p. 16

D. Limitation and Future Work

• Video-model failures may ease as video generation improves, while handling newly appearing objects may require generating new geometry during optimization.Source: D. Limitation and Future Work · p. 16
• A substantial portion of runtime is spent backpropagating through the VAE, motivating a possible distillation strategy that avoids those gradients.The authors suggest full VAE gradients may not be necessary when the objective is motion rather than RGB appearance.Source: D. Limitation and Future Work · p. 16

E. User Study Template

• The user-study template shows screenshots for prompt alignment and motion realism; participants ranked results as best, second-best, and third-best among five methods.Source: E. User Study Template · pp. 17–18, Figure 13 · p. 17, Figure 14 · p. 18