PaperlayerView exact revision on arXiv

Source-linked AI summary

PhyMo: Learning Physical Dynamics with Accurate and Continuous Motion from Multi-View Videos

Shangjia Liu, Jinxi Li, Siyuan Zhou, Bo Yang, Bing Wang

arXiv:2610.29107v1cs.CV

TL;DR

Rendering supervision can match observed images without recovering accurate motion, leaving dynamic geometry and future evolution unreliable. PhyMo combines motion-aware trajectories with continuous motion evolution, and reports strong interpolation and extrapolation performance across four benchmarks.

  • Problem

    Rendering supervision alone can let appearance compensate for inaccurate deformation, leaving physical motion states under-constrained and potentially discontinuous.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Method

    PhyMo combines a Motion-Aware Trajectory Representation that favors deformation over appearance shortcuts with Continuous Motion Evolution and bidirectional kinematic continuity.

    Source: §1 · pp. 2–3, §3 · p. 3, §3.2 · p. 5

  • Results

    Across four dynamic scene benchmarks, PhyMo shows strong interpolation and extrapolation performance, with more accurate dynamic geometry and more continuous, physically plausible future predictions.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The framework aims to recover accurate, continuous evolving dynamics rather than merely fit visual appearance.

    Source: §5 · p. 10

    Highly deformable scenes such as Newtonian fluids remain challenging because discrete Gaussian primitives struggle to capture fine-grained topology and appearance changes.

    Source: §A.9 · p. 21

Abstract

from arXiv · show

Recovering dynamic 3D scene geometry, appearance, and the physical motion states that govern scene evolution from multi-view videos is important yet challenging. A key difficulty is that appearance can be deceptive. Under rendering supervision alone, a model may fit the observed images by changing color or opacity to compensate for inaccurate deformation, rather than recovering the true evolving geometry and motion. This issue is especially severe in scenes with fast or locally complex motion, where inaccurate trajectories can still produce plausible images over the observed frames, but lead to unreliable reconstruction and discontinuous future prediction. In this paper, we propose PhyMo, a framework for learning evolving dynamics from multi-view videos. We introduce a Motion-Aware Trajectory Representation that drives the model to recover accurate motion from deformation, rather than relying on appearance shortcuts to explain observed motion. On top of this, we propose a Continuous Motion Evolution module with bidirectional kinematic continuity constraint to promote temporally consistent evolution of displacement, velocity, and acceleration. Together, these designs yield geometrically faithful interpolation and continuous motion extrapolation. Experiments on four dynamic scene benchmarks demonstrate state-of-the-art performance.

1 Introduction

The introduction frames dynamic scene learning as recovering physically meaningful motion, not merely matching rendered appearance. PhyMo combines motion-aware trajectories and continuous motion evolution to target faithful interpolation and continuous extrapolation.

1 Introduction

  • Existing approaches leave open how scene representations should encode dynamics and how physical motion states can be recovered from visual supervision.Trajectory-based methods mainly interpolate, while integration-based methods can extrapolate but risk error accumulation [3] [9] [10].Source: §1 · pp. 1–2
  • Rendering fidelity can conceal inaccurate motion, especially during rapid changes or state transitions, and unconstrained motion states may become locally discontinuous.These issues can undermine dynamic geometry and motion details in both interpolation and extrapolation.Source: §1 · p. 2
  • PhyMo targets geometrically faithful interpolation and continuous motion extrapolation by recovering plausible scene evolution rather than only geometry and appearance.The introduction presents qualitative results on two representative tasks in Figure 1.Source: §1 · p. 2, Figure 1 · p. 2
  • Motion-Aware Trajectory Representation directs dynamics into deformation rather than appearance shortcuts to support geometrically faithful reconstruction.It addresses the appearance-shortcut problem in dynamic reconstruction.Source: §1 · p. 2
  • Continuous Motion Evolution uses bidirectional kinematic continuity to constrain displacement, velocity, and acceleration over time.The method reports strong interpolation and extrapolation performance across four benchmarks.Source: §1 · p. 3

2 Related Works

Prior work extends static 3D representations into dynamic scenes, learns physical attributes or embeds physics systems, and uses hybrid approaches to learn motion states. The cited approaches face limits in motion reliability, physical-system generality, or scalability.

2 Related Works

  • Dynamic NeRF and 3DGS methods often learn deformation or motion fields under rendering supervision, which can weaken dynamic-geometry reliability and future extrapolation [30] [35] [36] [38].Some recent Gaussian methods relax canonical-space assumptions or model short-range motion, visibility, and lifespan, but do not explicitly address the competition between appearance fitting and motion recovery [38].Source: §2 · p. 3
  • Methods that learn physical attributes may depend on specific governing equations, while embedded physics systems can be limited in generalizability and scalability [39] [43] [46].Hybrid methods learn motion states from videos, but rendering supervision can leave those states under-constrained [15].Source: §2 · p. 3

3 Method

PhyMo combines a motion-aware trajectory representation with continuous motion evolution to recover dynamic scene geometry and motion from multi-view videos. Its method uses deformation-guided appearance and bidirectional kinematic continuity to support faithful interpolation and continuous extrapolation.

3.1 Motion-Aware Trajectory Representation

  • A deformation-derived, per-Gaussian motion gate suppresses canonical appearance residuals until local motion emerges, reducing shortcuts that lower rendering error without reliable motion.The shortcuts include changing static-region color and making fast-moving primitives transparent.Source: §3.1 · pp. 4–5

3.2 Continuous Motion Evolution

  • The Motion Network predicts linear and angular velocity and acceleration, then uses two-sided RK2 propagation and kinematic continuity to align states at each observed time.Neighboring trajectory states are sampled on both sides of the observation time and propagated toward it.Source: §3.2 · pp. 5–6

3.3 Training

  • Training warms up canonical Gaussians, gradually expands deformation supervision, then jointly fits motion and deformation using rendered observations and bidirectional continuity.The training schedule first establishes deformation trajectories before training motion states on top of them.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate PhyMo on four dynamic-scene datasets across geometrically faithful interpolation and continuous motion extrapolation. Results favor PhyMo on both tasks, while ablations indicate that motion guidance and continuous motion evolution contribute complementary benefits.

4 Experiments

  • The evaluation covers four datasets with diverse real-world and synthetic scenes, comparing reconstruction, prediction, and hybrid physics-aware baselines using rendering and geometric metrics.The datasets include articulated motion, object interactions, and synthetic scenes with physical effects; metrics include PSNR, SSIM, LPIPS, and Chamfer Distance.Source: §4 · pp. 6–7

4.1 Evaluation for Geometrically Faithful Interpolation

  • PhyMo achieves the best interpolation performance in RGB rendering metrics and dynamic geometric accuracy on NVIDIA Dynamic and PhysInOne.The reported qualitative comparisons also show better preservation of moving geometry and more coherent trajectories than competing methods.Source: §4.1 · pp. 7–8, Table 1 · p. 7

4.2 Evaluation for Continuous Motion Extrapolation

  • PhyMo achieves the best future-extrapolation performance across all four datasets, with predictions described as kinematically continuous and physically plausible.The evaluation observes frames in [0, 0.75] and predicts future frames in [0.75, 1].Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Ablation Study

  • Ablations show motion guidance supports geometry, continuity losses support extrapolation, the motion network supports physical-state prediction, and results are stable across moderate Δt changes.The analysis describes the trajectory representation and continuous motion evolution as complementary: reliable trajectories supervise motion-state learning, while continuous states also improve dynamic reconstruction.Source: §4.3 · pp. 9–10

5 Conclusion

PhyMo combines motion-aware trajectories with continuous motion evolution to recover accurate, continuous scene dynamics. Across four benchmarks, it achieves strong interpolation and extrapolation performance, with more accurate dynamic geometry and more continuous, physically plausible future predictions.

5 Conclusion

  • PhyMo combines motion-aware trajectories to reduce appearance shortcuts with continuous motion evolution to enforce temporal kinematic consistency.Source: §5 · p. 10
  • Experiments on four dynamic-scene benchmarks show strong interpolation and extrapolation performance.Source: §5 · p. 10
  • The reported predictions have more accurate dynamic geometry and more continuous, physically plausible future motion.Source: §5 · p. 10

A Appendix

The appendix lists further method, training, evaluation, and dataset details, alongside additional results and a demo video.

A Appendix

  • Appendix A.1–A.3 provide details on the motion-aware trajectory representation, continuous motion evolution, and training implementation.Source: A Appendix · p. 14
  • Appendix A.4–A.9 cover additional ablations and qualitative and quantitative results, datasets, resources, and model limitations.Source: A Appendix · p. 14
  • A demo video is included in the supplementary materials for visualization.Source: A Appendix · p. 14

A.1 More Details of Motion-Aware Trajectory Representation

Motion confidence controls when each Gaussian’s canonical appearance is allowed to change, using local deformation activity to limit appearance shortcuts. Its training behavior and ablations indicate that this design supports more accurate dynamic geometry.

A.1 More Details of Motion-Aware Trajectory Representation

  • The per-Gaussian confidence gate uses deformation change between adjacent timestamps to release anchored appearance parameters through gated residuals.The implementation uses a sigmoid gate with τ = 0.25, β = 50, and qmin = 0.0001; the gate is detached from back-propagation.Source: §A.1 · p. 14
  • After warm-up, activated Gaussian counts rise in stages as Chamfer Distance falls, and rendering loss decreases after more Gaussians activate.The repeated activation pattern reflects different motion regions becoming active at different times.Source: §A.1 · pp. 14–15, Figure 7 · p. 15
  • Removing motion confidence degrades rendering quality and dynamic geometric accuracy, while local deformation change better reflects current motion activity than cumulative displacement.Unlike a global freeze-release schedule, per-Gaussian confidence can adapt appearance freedom to spatially varying motion difficulty.Source: §A.1 · p. 15, Table 4 · p. 15
  • Algorithm 1 takes canonical Gaussians, a target timestamp, a step size, a deformation field, and a motion network to produce integrated Gaussians.The listed procedure propagates from neighboring timestamps around the target.Source: §A.1 · p. 16

A.2 Implementation Details of Continuous Motion Evolution Module

The Motion Network predicts translational and rotational motion states for each Gaussian, and bidirectional RK2 propagation uses neighboring states to estimate the target-time geometry. The resulting propagated state is used for rendering supervision, alongside kinematic continuity constraints.

A.2 Implementation Details of Continuous Motion Evolution Module

  • The Motion Network predicts translational and angular velocity and acceleration from each Gaussian’s position and timestamp.It is an 8×256 MLP with ReLU activation and an 8-degree positional embedding of the input position.Source: §A.2 · p. 16
  • RK2 estimates midpoint translational and angular velocities, then uses them to propagate position and rotation with a second-order local approximation.The midpoint states are formed using velocity and acceleration over the step.Source: §A.2 · p. 16
  • Propagation runs forward from the left neighbor and backward from the right neighbor; positions are averaged and rotations fused by midpoint SLERP.The propagated state is used for rendering supervision, while the two directional motion states receive kinematic continuity constraints.Source: §A.2 · p. 16

A.3 Implementation Details of Training Schedule

Training proceeds from canonical initialization through gradual temporal expansion to joint deformation and motion-network training. Neighboring-time RK2 propagation and bidirectional kinematic continuity losses are used in the final stage.

A.3 Implementation Details of Training Schedule

  • The schedule begins with 3,000 iterations of canonical Gaussian warm-up using observations only at t = 0.This stage initializes canonical geometry and appearance.Source: §A.3 · p. 16
  • Deformation learning then runs for 20,000 iterations across 10 temporal windows, progressively adding windows and updating motion confidence within the revealed support.Appearance freedom is released adaptively according to local deformation change.Source: §A.3 · p. 17
  • The final 20,000 iterations jointly train deformation and the Motion Network using bidirectional neighboring timestamps, RK2 propagation, rendering loss, and continuity losses.Both continuity-loss weights are set to 0.3.Source: §A.3 · p. 17

A.4 Additional Ablation Results

Qualitative ablations show distinct errors when motion guidance or continuity loss is removed: an arm disappears during interpolation, while balloon motion is predicted in the wrong direction during extrapolation.

A.4 Additional Ablation Results

  • Removing motion guidance makes the moving arm disappear in Skating interpolation, while removing continuity loss gives the balloon an incorrect direction in future-frame extrapolation.These examples illustrate appearance shortcuts and broken temporal consistency, respectively.Source: Figure 8 · p. 17, §A.4 · p. 17

A.5 Additional Qualitative Results

Additional qualitative comparisons show more accurate intermediate geometry and motion than strong 4D reconstruction baselines, and more accurate, temporally continuous future motion than FreeGave and TRACE. Over an extended prediction horizon, PhyMo maintains stable, continuous predictions, though only qualitative results are available there.

A.5 Additional Qualitative Results

  • For Laptop and Skating interpolation, PhyMo recovers more accurate intermediate geometry and motion than strong 4D reconstruction baselines.Source: §A.5 · p. 17
  • For Bus and Freefall extrapolation, PhyMo produces more accurate, temporally continuous future motion than FreeGave and TRACE.Those methods tend to accumulate motion errors or generate less stable predictions over time.Source: §A.5 · p. 18, Figure 9 · p. 18
  • When the prediction interval extends from [0.75, 1.0] to [0.75, 1.5], PhyMo maintains stable, continuous predictions, assessed qualitatively because extended-interval ground-truth frames are unavailable.Source: §A.5 · p. 18

A.6 Additional Quantitative Results

This section presents additional interpolation results comparing PhyMo with conventional reconstruction-based methods, alongside information on dataset coverage, training resources, limitations, research practices, LLM use, and planned code release.

A.6 Additional Quantitative Results

  • On MultiPart and FreeGave-GoPro interpolation, PhyMo consistently outperforms conventional reconstruction-based methods in challenging scenes.The reported results support its effectiveness in recovering temporally continuous appearance and motion.Source: §A.6 · p. 18

A.7 Details of All Datasets.

  • The datasets span real-world, multipart, and synthetic scenes involving occlusion, contact, articulation, deformation, and physical effects.Examples include object interactions and articulated motions in real scenes, alongside synthetic scenes governed by gravity, buoyancy, wind, and friction.Source: §A.7 · pp. 18–20

A.8 Training and Evaluation Resources

  • Per-scene optimization ranges from 30k to 2M Gaussians; training runs at about 0.98× FreeGave speed, and inference rendering is comparable to Deformable3DGS.All models are trained on a single NVIDIA RTX 3090 GPU.Source: §A.8 · p. 21

A.9 Limitation of Our Model

  • PhyMo is limited on highly deformable scenes such as Newtonian fluids, and its kinematic-level dynamics omit explicit forces, mass, and material parameters.Discrete Gaussian primitives struggle to capture fine-grained topology and appearance changes under continuous complex deformation; the omitted physical quantities may also limit handling of complex interactions.Source: §A.9 · pp. 21–22

NeurIPS Paper Checklist

  • The authors report providing a detailed method description and plan to release code upon acceptance, although code is not included yet.The checklist also states that training and test details are provided in the experiment section and appendix.Source: NeurIPS Paper Checklist · pp. 25–26
  • The authors report that they have not identified significant negative social implications and point to applications in the introduction for positive impacts.Source: NeurIPS Paper Checklist · p. 27
  • The paper reports no crowdsourcing or human-subject research, marking both participant-related disclosures and IRB approval as not applicable.The justification is that the research does not involve crowdsourcing or human subjects.Source: NeurIPS Paper Checklist · p. 29
  • LLMs were used only for writing and editing; core method development did not involve them as important components.The checklist marks disclosure of LLM use as not applicable.Source: NeurIPS Paper Checklist · p. 30

Related papers

Loading 2610.29107v1…