Inferring Compositional 4D Scenes without Ever Seeing One

1. Introduction

• Existing methods often handle one object at a time, rely on category-specific dynamic models, or simplify scenes as static or globally consistent, limiting coverage of real-world objects and motions.In-the-wild 4D reconstruction also faces scarce multi-object data and scenes with occlusion, interactions, and heterogeneous geometry and motion.Source: §1 · p. 2
• The target is a complete, persistent 4D representation of multiple static and dynamic objects in the wild from monocular video, without test-time optimization.The introduction identifies this as an unmet capability among existing models.Source: §1 · p. 2
• COM4D learns spatial structure from static multi-object observations and temporal dynamics from single-object animations, then unifies the learned reasoning at inference.The method's attention mixing alternates spatial and temporal reasoning to reconstruct multiple interacting objects without compositional 4D training examples.Source: §1 · p. 2
• The authors report coherent, persistent reconstructions across diverse in-the-wild scenes and significantly better results than specialized dynamic single-object or static-scene baselines.The contribution is framed as generalization across scenes and multiple interacting objects.Source: §1 · p. 2

4D Object Reconstruction

• Earlier 4D object reconstruction used low-rank assumptions [6, 14, 57, 60, 75] and physics-based priors, followed by category-specific parametric human models [21] [22] [23] [38] [62].Later approaches include score distillation [2] and generative or diffusion-based methods [46].Source: §2 · p. 2

Composed 4D Scene Reconstruction

• Learning-based static-dynamic scene reconstruction [12] [18] [28] [95] has often focused on human motion [12] [23] [51], omitted persistent object decomposition [11] [37] [39], or used online test-time optimization [39] [44] [45].Some approaches recompose static and dynamic objects through their poses and depth using test-time optimization [13].Source: §2 · pp. 2–3

Generative Scene Reconstruction

• Generative scene reconstruction includes multi-instance attention for object placement in MIDI and [54], while PartCrafter alternates in-part and inter-part attention without object-wise masks.The authors state that prior optimization-free approaches do not solve object-decomposed 4D scene reconstruction from video.Source: §2 · p. 3

3. Method

• The task represents a monocular video as image embeddings and seeks static and dynamic object geometry latents conditioned on those embeddings.The formulation allows multiple static objects and multiple dynamic objects across video frames.Source: §3 · p. 3
• A single DiT learns object-aware 3D scene reconstruction and deformable-object dynamics, then combines the capabilities through Attention Mixing for scenes it was not explicitly trained to generate.Diffusion Forcing fine-tuning is used to enhance temporal coherence.Source: §3 · p. 3

3.2. Attention Parsing: Dual-Objective Training

• Attention Parsing trains on static multi-object 3D-FRONT scenes and single-object DeformingThings sequences, assigning even blocks multi-instance attention and odd blocks multi-frame attention for their respective data sources.Independent latent noising trains the model to handle different denoising stages together, supporting clean static guidance and previously denoised frames as context at inference.Source: §3.2 · pp. 4–5

3.3. Attention Mixing: Compositional 4D Denoising

• At inference, Attention Mixing alternates scene-wide spatial reasoning across static objects and current dynamic frames with per-object temporal reasoning over each dynamic object's history.Spatial blocks use full-scene conditioning; temporal blocks use corresponding per-frame conditions, and a sliding temporal window supports any number of frames while static latents bypass motion processing.Source: §3.3 · p. 5

3.4. Implementation Details

• Implementation uses a 21-block pretrained TripoSG DiT, caps spatial targets at 8 parts and temporal targets at 8 frames, and trains in two stages for about 2 days.The second stage enables Diffusion Forcing; monolithic samples are included with probability 0.3 to regularize the model and preserve its object prior.Source: §3.4 · p. 5

4. Experiments

• Evaluation uses fixed-camera monocular videos, 3D-FRONT and animation datasets, task-specific CD, F-Score and IoU metrics, and average inference time on an NVIDIA H200.The 4D sets include 40 Objaverse assets and 30 human plus 30 animal DeformingThings4D animations; MIDI-3D receives ground-truth instance masks for static-scene comparison.Source: §4 · pp. 5–6

4.2. Single Object 4D

• In novel views, COM4D captures shape details, while V2M4 and TripoSG struggle with temporal consistency in some frames and Gaussian baselines show inconsistent underlying geometry.TripoSG provides strong static reconstruction; GVFD and L4GM can look plausible from the input view despite novel-view geometry inconsistency.Source: §4.2 · p. 7
• On DeformingThings, COM4D’s IoU of 0.4191 significantly outperforms all baselines across every metric; on Objaverse, it achieves the best F-Score and IoU.Its Chamfer Distance on Objaverse remains highly competitive with TripoSG.Source: §4.2 · p. 7, Table 1 · p. 7

4.3. 3D Scene Reconstruction

• On 3D-FRONT, COM4D achieves the best Chamfer Distance (0.0909) and F-Score (0.8069), including an advantage on the occluded split.Qualitatively, it produces complete, detailed layouts, while PartCrafter can produce partial meshes and MIDI struggles with occlusions.Source: §4.3 · p. 7, Table 2 · p. 7

4.4. Ablation Study

• Static/dynamic embeddings reduce DeformingThings CD from 0.1525 to 0.1284 and raise IoU from 0.2018 to 0.4034; the full model achieves the best overall performance.Without Attention Mixing, dynamic objects can overlap a lamp stand or miss their intended seating; alternating spatial and temporal attention improves spatial accuracy.Source: §4.4 · p. 8, Table 3 · p. 8

4.1. Compositional 4D Reconstruction

• For compositional 4D, users prefer COM4D over a baseline without Attention Mixing by 87% to 6.9%, and mixing reduces average CMU Panoptic CD from 35.91 cm to 7.42 cm.Reconstructions capture inter-object interactions in real and synthetic sequences.Source: §4.1 · p. 6, Figure 3 · p. 6, §4.4 · p. 8

5. Conclusion

• COM4D factorizes spatial and temporal priors into disentangled attention pathways and fuses them at inference to produce persistent, compositionally consistent 4D reconstructions.The conclusion reports state-of-the-art performance in 3D compositional scene reconstruction and 4D dynamic object reconstruction.Source: §5 · p. 8
• Because motion is learned from data without explicit physical causality, occlusion can challenge plausible reasoning about continued trajectories and interactions.The authors identify integrating physical causality as a direction for future work.Source: §5 · p. 8
• The current method is designed for fixed-camera videos and does not support camera motion.Extending the method to dynamic-camera inputs is proposed as future work.Source: §5 · p. 8

A. Training Details

• The temporal loss mirrors the spatial loss but indexes frames, uses DeformingThings samples, and conditions each frame on its own image embedding.The spatial loss instead uses a shared image embedding for all objects.Source: A. Training Details · p. 14
• Spatial, temporal, and regularization losses are sampled in a 0.35:0.35:0.3 ratio, with regularization samples containing one static object from Objaverse.The regularization loss is identified as the TripoSG loss.Source: A. Training Details · p. 14

B. User Study Details

• The user study compared full COM4D against a version without Attention Mixing using randomized sample order and assignment, with participants rating input match on a five-point scale.All ratings were included in the reported preference-score distribution.Source: B. User Study Details · p. 14, Figure 8 · p. 14

C. Evaluation on CMU Panoptic Dataset

• The protocol registers each sequence's first generated frame to ground truth, then applies that same scale, rotation, and translation to every later frame.This provides a common initial alignment for comparing subsequent-frame Chamfer Distance.Source: C. Evaluation on CMU Panoptic Dataset · p. 15
• After first-frame registration, later reconstructed frames still align well with ground truth, with reportedly small drift during motion.The evaluation computes per-frame Chamfer Distance after applying the initial alignment to subsequent frames.Source: C. Evaluation on CMU Panoptic Dataset · p. 15

D. Scalability with Respect to Object Count

• COM4D supports denoising with up to 8 parts, or up to 16 parts with additional finetuning.The static dataset averages 6.19 objects per scene, with five objects the most common configuration at 16.67%.Source: D. Scalability with Respect to Object… · p. 15
• Reconstruction quality gradually degrades as interacting-object counts and static-scene complexity increase.The authors describe compositional reasoning as more challenging with more interacting objects.Source: D. Scalability with Respect to Object… · pp. 15–16

E. Additional Qualitative Results on Compositional 4D

• Without attention mixing, the chair pose and the handshaking interaction, as well as the lady-chair relationship, are captured incorrectly in the shown examples.Figure 12 compares input frames with results using and omitting the mixing strategy.Source: Figure 12 · p. 16

F. Additional Qualitative Results with Moving

• Figure 13 presents qualitative results for a sample with a dynamic camera.Source: Figure 13 · p. 16

G. Physical Plausibility and Object Interactions

• Average mesh-intersection IoU is 0.0096 across eight video sequences, compared with 0.0018 on 3D-FRONT, suggesting comparable physical plausibility without explicit physical constraints.The method models object interactions implicitly through latent-space attention.Source: G. Physical Plausibility and Object… · p. 17
• Figure 14 places input frames in the top row and 4D reconstructions in the bottom row for interaction scenarios.Source: Figure 14 · p. 17

H. Comparison with Optimization-Based Baselines

• COM4D achieves 7.42 cm Chamfer Distance versus Shape of Motion's 11.30 cm on [32] [33].The comparison reports that Shape of Motion often produces incomplete or noisy geometry under occlusion; Figure 15 provides a qualitative comparison.Source: Figure 15 · p. 17, H. Comparison with Optimization-Based… · p. 17

• Evaluation compares the generated scene-level mesh with static reference geometry and computes dynamic scores per moving object and frame.Static geometry receives one CD/F-score pair per scene; dynamic scores are averaged within each sample and then across samples.Source: §I · p. 18, Table 4 · p. 18
• The synthetic evaluation reports CD, where lower is better, and F-score, where higher is better.Source: Table 4 · p. 18
• Synthetic-scene scores are worse than the corresponding single-object 4D and static compositional 3D results, mainly because small placement errors accumulate in CD and F-score.The reconstructed objects and coarse arrangements may look consistent even when relative positions remain inaccurate.Source: §I · p. 18, Table 4 · p. 18

• Figure 16 presents an additional qualitative result on the synthetic compositional 4D dataset.Source: Figure 16 · p. 18, J. Additional Qualitative Results on… · p. 18
• Figure 17 presents an additional qualitative result on the synthetic compositional 4D dataset.Source: J. Additional Qualitative Results on… · p. 18, Figure 17 · p. 19
• Figure 18 presents an additional qualitative result on the synthetic compositional 4D dataset.Source: J. Additional Qualitative Results on… · p. 18, Figure 18 · p. 19

K. Additional Qualitative Results on 4D Object Reconstruction

• The Objaverse comparisons show reconstructed input views above rendered novel views, with the ground-truth novel view at bottom left.Source: Figure 19 · p. 20
• COM4D typically shows less shape or pose misalignment than the compared methods in Objaverse novel views.TripoSG and V2M4 show moderate, consistent misalignment; GVFD has rotation and skeletal-pose misalignment, while L4GM often fails to provide good novel views.Source: Figure 20 · p. 20
• In the stronger-motion DeformingThings sequences, V2M4 fails completely.The examples compare rendered novel views with ground truth across three time steps.Source: Figure 21 · p. 21

L. Additional Qualitative Results on 3D Scene Generation

• In the 3D scene comparison, COM4D more consistently reconstructs and composes parts, while PartCrafter and MIDI often miss large objects such as the bed.Source: Figure 22 · p. 21