PaperlayerView exact revision on arXiv

Source-linked AI summary

FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity

Jinxi Li, Ziyang Song, Siyuan Zhou, Bo Yang

arXiv:2506.07865v1cs.CVcs.AIcs.CEcs.LGcs.RO

TL;DR

Accurate physical models of complex dynamic 3D environments matter, but learning their motion from video without object priors remains a challenge. FreeGave learns per-Gaussian physics codes and divergence-free velocities, and achieves the best reported performance across datasets for future-frame extrapolation.

  • Problem

    Accurate physical models of complex dynamic 3D environments are important for applications including robot planning, embodied interaction, and dynamic 3D/4D generation.

    Source: §1 · p. 1

  • Method

    FreeGave extends 3D Gaussian splatting with per-Gaussian physics codes and a divergence-free velocity module to learn scene geometry, appearance, and motion from video without object priors.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 3

  • Results

    FreeGave achieves the best performance on all evaluated datasets for future-frame extrapolation and outperforms strong baselines on novel-view interpolation.

    Source: §4.1 · p. 5

  • Takeaways & Limitations

    The learned physics codes describe Gaussian motion patterns over time, supporting predictions of future frames and motion decomposition without object priors.

    Source: §1 · p. 2, §3.2 · p. 3, §5 · p. 7

    The canonical representation assumes each Gaussian's opacity and color remain unchanged over time.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

In this paper, we aim to model 3D scene geometry, appearance, and the underlying physics purely from multi-view videos. By applying various governing PDEs as PINN losses or incorporating physics simulation into neural networks, existing works often fail to learn complex physical motions at boundaries or require object priors such as masks or types. In this paper, we propose FreeGave to learn the physics of complex dynamic 3D scenes without needing any object priors. The key to our approach is to introduce a physics code followed by a carefully designed divergence-free module for estimating a per-Gaussian velocity field, without relying on the inefficient PINN losses. Extensive experiments on three public datasets and a newly collected challenging real-world dataset demonstrate the superior performance of our method for future frame extrapolation and motion segmentation. Most notably, our investigation into the learned physics codes reveals that they truly learn meaningful 3D physical motion patterns in the absence of any human labels in training.

1. Introduction

Many dynamic 3D methods render observed scenes well but do not learn the physical properties needed for future prediction. FreeGave aims to learn scene geometry, appearance, and per-Gaussian motion from multi-view video without PINN losses or object priors.

  • Most dynamic 3D approaches lack future prediction because they do not effectively learn the physical properties of 3D space.Source: §1 · p. 1
  • PINN-based methods can blur future predictions at boundaries, while methods with explicit physics models are often limited to certain objects or motions and may need masks.Source: §1 · p. 1
  • FreeGave treats each 3D Gaussian as a rigid particle and learns a physics code to describe its motion over time.Source: §1 · p. 2
  • The framework jointly learns geometry, appearance, and velocity using a divergence-free Gaussian velocity module and deformation-aided optimization.Source: §1 · p. 2
  • The method learns scene physics from RGB videos without object priors or PINN losses, and is evaluated for future-frame extrapolation across four datasets.Source: §1 · p. 2

2. Related Works

Related work spans explicit 3D representations, neural representations, dynamic-scene reconstruction, video prediction, and 3D physics learning. FreeGave builds on 3D Gaussian kernels and encodes physics directly in its learning process.

  • Classic point clouds, meshes, voxels, and primitives can have limited shape fidelity at their spatial resolutions.Source: §2 · p. 2
  • Neural fields support view rendering and shape recovery, while 3DGS offers real-time rendering and high-quality novel views using Gaussian kernels.Source: §2 · p. 2
  • Dynamic-scene methods can render observed times well but often lack physically meaningful future prediction because they fit observations without physics priors.Source: §2 · p. 2
  • Video-generation methods can produce strong results but may predict physically implausible motion when training lacks explicit physics priors.Source: §2 · p. 2
  • PDE-loss approaches can struggle at boundaries, while explicit physics systems often target specific objects; FreeGave instead encodes a generic divergence-free constraint.Source: §2 · p. 2

3. FreeGave

FreeGave learns a dynamic scene from posed multi-view video by combining canonical Gaussian representation, per-Gaussian physics codes and velocities, and deformation-aided optimization. Its velocity design decomposes motion into basic components and constrains the resulting field to be divergence-free.

3. FreeGave

  • The pipeline uses posed multi-view RGB frames to jointly learn scene geometry, appearance, and velocity through three modules.Source: §3 · p. 3

3.1. Canonical 3D Representation Module

  • Canonical 3DGS kernels represent geometry and appearance at time zero, with opacity and color assumed constant over time.Source: §3.1 · p. 3

3.2. Divergence-free Gaussian Velocity Module

  • Each canonical Gaussian receives a physics code predicted from its position; the code represents underlying physical information and describes its motion pattern across timestamps.Source: §3.2 · p. 3
  • The velocity module decomposes a rigid particle’s motion into six basic components and designs Vt to be independent of position, ensuring a divergence-free field.Source: §3.2 · pp. 3–4
  • A bottleneck derived from the physics code and timestamp-generated weights combine by matrix multiplication to produce the basic velocity component Vt.Source: §3.2 · p. 4

3.3. Deformation-aided Optimization

  • An auxiliary deformation field predicts position, quaternion, and scale changes to aid optimization, while sparse multi-view supervision makes direct joint training difficult.Source: §3.3 · p. 4
  • The interleaved midpoint procedure transports Gaussians using velocity estimates at the start and midpoint, then renders them for supervision with standard 3DGS losses.Source: §3.3 · p. 4

4. Experiments

FreeGave is evaluated on three synthetic datasets and a newly collected real-world dataset for future-frame extrapolation and motion segmentation. It achieves the best reported performance across datasets, while its learned physics codes support object segmentation and reveal distinct motion patterns.

4. Experiments

  • Experiments cover three synthetic datasets and a newly collected six-scene GoPro dataset; RGB view synthesis is scored with PSNR, SSIM, and LPIPS on both tasks.The GoPro scenes include motions such as an object moving tightly beside a static object or a thin object passing through a tiny hole.Source: §4 · p. 5
  • Training uses 67 frames from 17 views; 67 frames from three held-out views test interpolation, while the final 22 frames from all 20 views test extrapolation.The cameras remain fixed over time.Source: Figure 4 · p. 5, §4 · p. 5

4.1. Evaluation for Future Frame Extrapolation

  • Comparisons include NVFi and NeRF baselines, plus DefGS and DefGSnvfi, which pairs DefGS with NVFi's velocity field for a 3DGS extrapolation baseline.DefGS itself is described as lacking future-prediction ability.Source: §4 · p. 5
  • FreeGave achieves the best performance across all datasets for future-frame extrapolation and also beats strong baselines on novel-view interpolation.The reported comparison covers the two synthetic datasets and the FreeGave-GoPro dataset, with ParticleNeRF included for extrapolation.Source: §4.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.2. Analysis of Physics Code and Motion Patterns

  • Gaussian kernels grouped by bottleneck vectors share velocity trajectories, whereas kernels in different groups exhibit distinct motion trajectories.The analysis samples trajectories at 10 timestamps and clusters the learned bottleneck vectors with K-means.Source: §4.2 · p. 6, Figure 5 · p. 7
  • Physics-code grouping yields nearly perfect object-segmentation results across all metrics, outperforming baselines on 7,200 rendered masks.Masks are evaluated over 30 views, 60 timestamps, and all four indoor scenes.Source: §4.2 · p. 6, Table 3 · p. 6

4.3. Ablation Study

  • Removing the deformation field hurts most; autoregressive transport then uses half-target-time steps to control memory, while further ablations remove code conditioning or scale updates.The study also tests removing bottleneck-vector decomposition and varying the number of motion patterns.Source: §4.3 · pp. 6–7, Table 4 · p. 7
  • Replacing the physics-code network with per-Gaussian learnable codes has the next-largest impact; removing divergence-free parametrization or motion-pattern decomposition also reduces performance.The ablations additionally report that MLP local continuity helps convergence, physics-code input to the deformation field helps, too few patterns hurt, and over-parameterization is not sensitive.Source: §4.3 · pp. 6–7

5. Conclusion

FreeGave learns scene geometry, appearance, and physics from multi-view videos without object priors, using a physics code and divergence-free per-Gaussian velocity field. Experiments cover future-frame prediction and object motion decomposition, and the paper proposes 4D generation as a possible application.

5. Conclusion

  • FreeGave learns geometry, appearance, and physics from multi-view videos without object priors, using a physics code and divergence-free velocity field for each Gaussian kernel.The method avoids inefficient PINN losses.Source: §5 · p. 7
  • Experiments on three public datasets and a newly captured challenging real-world dataset report strong performance for future-frame prediction and object motion decomposition.Figure 6 presents qualitative comparisons with baselines for future-frame extrapolation and unsupervised motion segmentation.Source: §5 · p. 7, Figure 6 · p. 8
  • The paper suggests applying FreeGave to 4D generation, where 3D physics learning is important for generating plausible motions.Source: §5 · p. 7

A. Proof of Divergence-free Property

The proof aims to establish that the velocity field has zero divergence by checking its basis vector functions.

A. Proof of Divergence-free Property

B. Implementation Details

The implementation specifies MLP configurations for the physics-code, neck, weight, and deformation networks, along with dataset-specific training time steps.

B. Implementation Details

  • The physics-code network uses four 128-neuron hidden layers, degree-8 positional encoding, and a 16-dimensional output code; the neck outputs K=32 or K=16 by dataset.K is 32 for Dynamic Indoor Scene and 16 for the other two datasets.Source: B. Implementation Details · p. 12
  • The weight network has five 128-neuron hidden layers, degree-8 positional encoding, a third-layer ResNet connection, and a K × 6 output matrix; the deformation network varies by dataset.The deformation network uses six 128-neuron layers for the synthetic datasets and eight 256-neuron layers for FreeGave-GoPro; it concatenates the physics code with encoded position.Source: B. Implementation Details · p. 12

C. Details of Deformation-aided Optimization

D. Details of All Datasets

The evaluation covers synthetic dynamic-object, indoor, and ParticleNeRF scenes, challenging real-world FreeGave-GoPro scenes, NVIDIA Dynamic Scene scenes, and collision cases.

D. Details of All Datasets

  • The Dynamic Object dataset contains six moving objects spanning accelerated part-wise rigid motions and self-propelling deformable motion, with held-out views and future frames for evaluation.Training uses the first 46 frames from 12 views; future extrapolation uses the remaining 14 frames across 15 views.Source: D. Details of All Datasets · p. 12
  • The Dynamic Indoor Scene dataset contains four scenes with three to five moving objects each, all undergoing different rigid motions.Training uses 46 frames from 25 views, with five other views for interpolation and the remaining 14 frames across 30 views for future extrapolation.Source: D. Details of All Datasets · p. 12
  • ParticleNeRF includes six dynamic objects spanning robot actions, rotation, oscillations, and cloth folding, with training and test observations covering a full oscillation period for three objects.The dataset includes robot, robot-task, wheel, spring, pendulums, and cloth scenes; splits vary by object group.Source: D. Details of All Datasets · p. 13
  • FreeGave-GoPro comprises six challenging real-world scenes involving penetration, occlusion, tight moving-static combinations, contact, opposing motions, and deformable motion.The scenes include pen and tape, a closing box, a hammer moving on a box, a cube and cup moving toward each other, and a bending wrist rest.Source: D. Details of All Datasets · p. 13

E. Quantitative Results on NVIDIA Dynamic Scene Dataset

F. Quantitative Results on Collision Cases

  • Two collision cases extend the dining scene, differing in whether collision is absent or observed during training; Table 6 reports results on oscillation or collision scenes.The first case trains without observing collision and extrapolates across the collision; the second trains with the collision observed.Source: F. Quantitative Results on Collision… · p. 13, Table 6 · p. 13

G. Analysis of Computational Costs

FreeGave is generally competitive in computational cost and outperforms baselines on future-frame extrapolation; the appendix also describes its motion-segmentation procedure and notes a limitation on abrupt motions.

G. Analysis of Computational Costs

  • The appendix reports average training time and memory, test speed and memory, and model size for the compared methods.Source: G. Analysis of Computational Costs · p. 13
  • FreeGave has a clear training time and memory advantage over DefGSnvfi, while generally matching or improving on other baselines’ computation costs and achieving significantly better extrapolation.The reported comparisons cover training and test costs; extrapolation results are in Tables 1 and 2.Source: G. Analysis of Computational Costs · pp. 13–14

H. Limitation of Our Model

  • The method may fail on abrupt motions such as explosions because their underlying physics rules may not be observable or learnable from visual frames.Source: H. Limitation of Our Model · p. 14

I.1. More Details of Our FreeGave

  • FreeGave segments its trained Gaussians by clustering concatenated bottleneck vectors and smoothed canonical positions with K-means.The position weight λ is 0 for Genome House and Chessboard, and 0.5 for Factory and Dining Table; the cluster count is 13 for Dining Table and 8 for the other scenes.Source: §I.1 · p. 14

I.2. More Details of Segmenting DefGS / DefGSnvfi

  • For DefGS and DefGSnvfi, per-Gaussian object codes are optimized using scene-flow-based rigid consistency and spatial smoothness losses.The rigid-consistency loss aims to distinguish objects with different motions; spatial smoothness aims to avoid over-segmentation.Source: §I.2 · p. 14

J. More Results of Ablation Study

The appendix reports ablations at K = 16, noting a slightly better K = 32 setting on the Dynamic Indoor Scene dataset.

  • Ablations use K = 16, although K = 32 is slightly better on the Dynamic Indoor Scene dataset; this does not affect the analysis of influencing factors in the main paper.Source: J. More Results of Ablation Study · p. 14

K. More Results on Dynamic Object Dataset

The appendix provides per-scene quantitative results across four datasets, computational-cost and scene-decomposition tables, and additional qualitative results for extrapolation and motion segmentation.

K. More Results on Dynamic Object Dataset

L. More Results on Dynamic Indoor Scene Dataset

N. More Results on ParticleNeRF Dataset

O. Additional Qualitative Results

Loading 2506.07865v1…