Source-linked AI summary
Seeing the Wind from a Falling Leaf
Zhiyuan Gao, Jiageng Mao, Hong-Xing Yu, Haozhe Lou, Emily Yue-Ting Jia, Jernej Barbic, Jiajun Wu, Yue Wang
TL;DR
Computer vision has left the invisible physical interactions behind object motion relatively underexplored, despite their relevance to understanding video. This paper introduces a differentiable inverse-graphics framework to recover forces from object motions, and reports force recovery on synthetic and real-world scenarios, with physically plausible video generation and editing as potential applications.
Problem
Recovering invisible forces from video matters because object motion provides visible cues to physical interactions that cannot be seen directly.
Source: §1 · p. 2Method
A differentiable inverse-graphics framework jointly models object properties, forces, and physical processes to estimate forces from video observations via backpropagation.
Source: §1 · pp. 2–3Results
Experiments on synthetic and real-world scenarios demonstrate force recovery, and estimated forces support physically plausible motion generation and editing.
Source: §1 · pp. 2–3Takeaways & Limitations
Changing object types, physical properties, or boundary conditions enables potential physics-based video generation and editing with estimated forces.
Source: §1 · p. 2The framework primarily addresses objects with small or bending-only deformations; fluids and other object types requiring different differentiable physical processes are out of scope.
Source: §5 · p. 10
Abstract
from arXiv · showhide
A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing a leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world scenarios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including physics-based video generation and editing. We hope our approach sheds light on understanding and modeling the physical process behind pixels, bridging the gap between vision and physics. Please check more video results in our \href{https://chaoren2357.github.io/seeingthewind/}{project page}.
1 Introduction
The paper addresses recovering invisible forces from visible object motions, a capability that remains underexplored in computer vision. It introduces a differentiable framework and sparse tracking objective for estimating forces, and reports force recovery across synthetic and real-world scenarios with potential applications in physics-based video generation and editing.
1 Introduction
- The central challenge is inferring unseen forces from visible video cues alone, such as a falling leaf’s motion under wind.The paper frames this as an underexplored computer-vision problem of recovering physical interactions from visual observations.Source: §1 · p. 2
- The framework jointly models object geometry, appearance, physical properties, forces, and physical processes, estimating forces from video motions via backpropagation.It uses 3D Gaussians for shape and appearance, a causal tri-plane for forces, and a differentiable simulator for deformable objects.Source: §1 · p. 2
- Sparse keypoint tracking reduces the prediction space and helps estimate dense forces while addressing unstable gradients from simulator time integration.Lagrangian elements are controlled by neighboring keypoints through barycentric interpolation.Source: §1 · p. 2
- Experiments on synthetic and real-world scenarios demonstrate force recovery and physically plausible motion generation and editing.The generated motions can vary object types, physical properties, or boundary conditions.Source: §1 · pp. 2–3
2 Related Works
Prior intuitive-physics research has largely estimated individual physical parameters, while force recovery is challenging because forces are vector-valued and can occur throughout 3D space. Related approaches also face gradient instability, controlled-environment or strong-prior requirements, or lack explicit force representations for editing.
2 Related Works
- Earlier intuitive-physics methods primarily estimate individual properties such as mass, friction, or Young’s modulus, whereas this work targets forces distributed through 3D space.The paper characterizes forces as more challenging because they are vectors that can exist throughout the scene.Source: §2 · p. 3
- Differentiable simulators can produce vanishing or exploding gradients; the proposed sparse tracking objective is intended to stabilize estimation and recover high-dimensional forces.The cited challenges include discontinuous behavior and time integration.Source: §2 · p. 3
- Many prior force-estimation methods rely on controlled robotic settings, tactile sensors, or strong shape and property priors, unlike this method’s natural-video setting with minimal property assumptions.Source: §2 · p. 3
- Prior generation methods often require manually specified forces, while displacement-based edits lack underlying forces and velocity-field methods lack explicit force representations for parameter-aware edits.The paper contrasts these approaches with recovering forces and physical conditions from natural videos for physics-driven generation without manual parameter tuning.Source: §2 · p. 3
3 Method
The method jointly models object properties, force fields, and physical dynamics from videos, then uses sparse motion tracking to optimize force recovery. It combines 3D Gaussian object representations, a causal tri-plane force model, and differentiable simulation.
3.1 Object Modeling
- 3D Gaussians represent object shape and appearance while carrying physical properties estimated from video with vision-language and grounded-segmentation models.The Gaussians are initialized from the first frame using metric-depth point clouds; assigned properties include mass, Young’s modulus, and Poisson ratio.Source: §3.1 · pp. 3–4
3.2 Force Representations
- A causal tri-plane represents spatially continuous forces, with a recurrent time encoding for modeling their evolution.The force field is decoded from combined spatial tri-plane features and a time encoding initialized from the preceding timestep.Source: §3.2 · p. 5
3.3 Physical Process
- A differentiable Material Point Method simulator evolves Gaussian object states under external and internal forces.External forces are applied to particles before particle-to-grid transfer, yielding finer, less noisy fields aligned with moving mass while avoiding empty grid cells.Source: §3.3 · p. 5
3.4 Recovering Forces from Videos
- For bending-only or small deformations, sparse tracked keypoints provide motion targets that are interpolated to Gaussians and used to optimize the force field.3D keypoint motion is estimated by minimizing reprojection error while keeping the object skeleton as rigid as possible, avoiding per-frame depth estimation; spatial and temporal regularizers are also applied.Source: §3.4 · pp. 6–7
4 Experiments
The experiments assess force recovery on synthetic and real-world videos, then test representation and loss choices, material-property estimates, and physics-based generation and editing. Across these settings, the method recovers plausible forces and supports physically plausible motion changes.
4.1 Experimental Setup
- The experimental setup combines Internet videos and force-gauge experiments with synthetic scenarios spanning multiple materials, force fields, and camera viewpoints.Synthetic evaluations compare re-simulated videos and recovered forces against simulator ground truth.Source: §4.1 · p. 7
4.2 Force Recovery
- In controlled real-world experiments, reapplying recovered forces in simulation reconstructs object motion and deformation close to observations.Known forces were applied with a force gauge before video capture.Source: §4.2 · p. 7, Figure 3 · p. 8
- Synthetic scenarios show low numerical errors when recovering ground-truth force fields across objects and physical properties.The evaluation uses simulator-generated ground-truth forces.Source: §4.2 · p. 7, Table 1 · p. 8
4.3 Empirical Study
- The causal tri-plane representation outperforms K-planes and point contact forces by modeling spatial continuity and temporal dependence.The comparison evaluates alternative force representations.Source: Table 2 · p. 8, §4.3 · p. 8
- Sparse tracking loss performs better than image reconstruction and Flow+Depth losses, especially for force accuracy.The authors attribute this to more robust motion estimates providing more accurate optimization signals.Source: Table 3 · p. 8, §4.3 · p. 8
- On the daily-item dataset of single, well-documented engineering materials, the VLM achieves overall material-classification F1 1.0, density MAPE 4.65%, and log-MAPE 7.17%.Source: Table 4 · p. 8, §4.3 · p. 9
4.2 Force Recovery
- In-the-wild videos show plausible recovered force fields that adapt dynamically to object motion over time.The evaluation uses object deformations and trajectories across varied scenes.Source: §4.2 · p. 7, Figure 4 · p. 8
4.4 Physics-based Generation
- With recovered forces, the method supports plausible video generation and editing through novel-object insertion and changes to physical properties, force strengths, and boundary conditions.The authors report greater controllability and physical accuracy than other video-generation methods; qualitative comparisons include interactive editing / motion-driven methods [48] [49] and velocity-field learning baselines [50] [51].Source: §4.4 · pp. 9–10, Figure 5 · p. 9, Figure 6 · p. 9
5 Conclusion
The framework recovers invisible forces from video object motions and supports controllable physics-based video generation and editing. Its scope is limited to objects with small or bending-only deformation.
5 Conclusion
- The framework models object properties, forces, and physical processes to recover invisible forces from video object motions, with accurate force recovery and controllable video generation and editing demonstrated on synthetic and real-world data.Source: §5 · p. 10
- The framework primarily applies to objects with small or bending-only deformation; fluids and objects needing other differentiable physical processes are out of scope.Source: §5 · p. 10
- The demonstrations model foreground physics over a static background because off-the-shelf per-frame inpainting can introduce temporal flicker that obscures the contribution.Source: §5 · p. 10