Source-linked AI summary
What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards
Minh-Quan Le, Yuanzhi Zhu, Vicky Kalogeiton, Dimitris Samaras
TL;DR
Video generators can produce visually compelling clips while violating physical laws, and human or VLM feedback does not precisely evaluate many such constraints. NewtonRewards uses optical-flow and appearance proxies to enforce Newtonian kinematic and mass-conservation constraints; it improves physical plausibility and temporal coherence across five motion primitives and ID/OOD settings.
Problem
Video generation can produce visually realistic clips that violate physical principles, while human and VLM judgments cannot precisely evaluate many physical constraints.
Source: §1 · p. 1Method
NewtonRewards post-trains video generators with verifiable Newtonian rewards built from optical-flow proxies for velocity and appearance embeddings as mass proxies.
Source: §1 · p. 2, §4.1 · p. 4, §4.1.2 · p. 4Results
NewtonRewards consistently improves physical plausibility, motion smoothness, and temporal coherence across five Newtonian Motion Primitives in ID and OOD settings.
Source: §1 · p. 2Takeaways & Limitations
Measurable proxies can support explicit, differentiable constraints that guide video generators toward physically valid behavior.
Source: §7 · p. 8The optical-flow setup assumes a static camera and fixed background; the gravity formulation assumes vertical-axis alignment or, for tilted cameras, weak perspective and a static camera.
Source: §3.2 · p. 3, §4.2.3 · p. 5
Abstract
from arXiv · showhide
Recent video diffusion models can synthesize visually compelling clips, yet often violate basic physical laws-objects float, accelerations drift, and collisions behave inconsistently-revealing a persistent gap between visual realism and physical realism. We propose $\texttt{NewtonRewards}$, the first physics-grounded post-training framework for video generation based on $\textit{verifiable rewards}$. Instead of relying on human or VLM feedback, $\texttt{NewtonRewards}$ extracts $\textit{measurable proxies}$ from generated videos using frozen utility models: optical flow serves as a proxy for velocity, while high-level appearance features serve as a proxy for mass. These proxies enable explicit enforcement of Newtonian structure through two complementary rewards: a Newtonian kinematic constraint enforcing constant-acceleration dynamics, and a mass conservation reward preventing trivial, degenerate solutions. We evaluate $\texttt{NewtonRewards}$ on five Newtonian Motion Primitives (free fall, horizontal/parabolic throw, and ramp sliding down/up) using our newly constructed large-scale benchmark, $\texttt{NewtonBench-60K}$. Across all primitives in visual and physics metrics, $\texttt{NewtonRewards}$ consistently improves physical plausibility, motion smoothness, and temporal coherence over prior post-training methods. It further maintains strong performance under out-of-distribution shifts in height, speed, and friction. Our results show that physics-grounded verifiable rewards offer a scalable path toward physics-aware video generation.
1. Introduction
Video generators can produce visually compelling clips that violate physical laws, while existing feedback often relies on human or VLM judgments rather than explicit law enforcement. NewtonRewards uses measurable proxies to construct verifiable Newtonian rewards and improves physical plausibility and temporal coherence across five motion primitives and ID/OOD settings.
1. Introduction
- Physical plausibility matters because generated videos can serve as data for perception, reasoning, and action in applications including autonomous driving and robotic control.The paper notes that nonphysical dynamics can lead to inconsistent learning signals and unrealistic affordances.Source: §1 · p. 1
- Generated videos can look realistic yet violate physical principles, including momentum conservation, force–acceleration proportionality, and consistent gravitational effects.Human and VLM judgments are described as unable to precisely evaluate physical constraints beyond egregious violations.Source: §1 · pp. 1–2
- Optical flow and visual features act as proxies for velocity and mass, enabling explicit kinematic and mass-conservation rewards for post-training video generators.The framework uses these measurable proxies to formulate verifiable, rule-based constraints on generated motion.Source: §1 · p. 2
- NewtonRewards improves physical plausibility, motion smoothness, and temporal coherence across all five Newtonian Motion Primitives in both ID and OOD settings.The primitives include free fall, horizontal and parabolic throws, and ramp sliding down or up.Source: §1 · p. 2
- The authors propose that measurable proxies of physical variables could support verifiable rewards for other physical laws beyond Newtonian motion.They frame NewtonRewards as an initial application of this general approach.Source: §1 · p. 2
2. Related Works
Prior physics-aware video-generation work uses feedback, simulation or 3D representations, and physics-rich data or post-training signals. The related work describes limits of indirect feedback and conditioning, while NewtonBench-60K illustrates five controlled Newtonian motion primitives.
2. Related Works
- Physics-aware video-generation approaches use instruction or feedback-based fine-tuning, simulation or 3D representations, and physics-rich datasets or post-training.The cited examples include LLM/VLM feedback methods, simulation-guided generation, and data-level grounding such as PISA [36].Source: §2 · pp. 2–3
- Feedback-based methods such as PhyT2V [62] use indirect judgments that can reflect perception rather than physics consistency.The method checks physical behavior using an LLM reasoning over captions from a video-captioning model.Source: §2 · p. 2
- Simulator- and 3D-guided approaches provide motion or geometry priors, but external simulators enforce physics indirectly through the generator’s interpretation of conditioned data.PhysGen [39], for example, warps images using simulated motion dynamics given user-defined forces and torques.Source: §2 · p. 2
- PISA [36] post-trains on optical flow, depth, and segmentation for free-fall videos, while NewtonBench-60K illustrates five Newtonian Motion Primitives.Figure 2 pairs free-body diagrams of dominant forces and accelerations with rendered trajectories from a Kubric-based [18] simulator.Source: §2 · p. 3, Figure 2 · p. 3
3. Newtonian Motion Primitives (NMPs)
The paper organizes motion into five Newtonian primitives, relating their image-plane accelerations to gravity, applied forces, and ramp friction. It derives a shared constant-acceleration structure under stated projection and camera assumptions.
3. Newtonian Motion Primitives (NMPs)
- The five primitives are free fall, horizontal and parabolic throw, and ramp sliding down or up, each associated with a characteristic image-plane acceleration.The paper frames them as distinct combinations of forces.Source: §3 · p. 3
3.1. Background: Newton’s Laws of Motion
- Newton’s laws connect net force, image-plane acceleration, and mass, while ramp contact adds a normal force and opposing tangential friction.In the stated free-fall and throw cases, no horizontal force means zero horizontal acceleration and constant horizontal velocity.Source: §3.1 · p. 3
3.2. Forces and Accelerations of NMPs
- Under gravity, free fall and both throw primitives share constant downward acceleration; their initial velocities distinguish the three cases.The paper also approximates image-plane coordinates as proportional to world coordinates over short intervals.Source: §3.2 · p. 3
- For ramp sliding, gravity and kinetic friction determine constant tangential acceleration, whose sign distinguishes downhill from uphill motion.For tilted cameras, the stated constant apparent acceleration relies on weak perspective and a static camera.Source: §3.2 · pp. 3–4
4. Method: NewtonRewards
NewtonRewards post-trains video generators using physics penalties built from differentiable proxies extracted from generated clips. Optical flow approximates velocity, visual embeddings proxy mass-related appearance, and their rewards enforce kinematic consistency and mass preservation.
4. Method: NewtonRewards
- NewtonRewards fine-tunes video generators with reward functions that quantify whether generated sequences adhere to physical constraints.Its method computes measurable proxies from outputs, then uses those proxies to define the rewards.Source: §4 · p. 4
4.1. Measurable Proxies
- The proxy extractor maps generated videos to differentiable quantities, using optical flow for velocity and visual appearance embeddings for mass-related properties.These quantities support constant-acceleration residuals and mass-consistency rewards.Source: §4.1 · p. 4
4.1.1. Optical Flow as Velocity Proxy
- An optical-flow model estimates per-frame displacement from adjacent video frames, which is used to approximate image-plane velocity.The predicted flow is measured in pixels per frame.Source: §4.1.1 · p. 4
4.1.2. Visual Features as Mass Proxy
- A pre-trained video encoder’s appearance embeddings serve as a differentiable proxy for mass-related properties and visual mass consistency.The paper motivates this proxy by noting that appearance and texture cues can correlate with identity, material, and effective mass.Source: §4.1.2 · p. 4
4.2. Physics Constraints and Rewards
- The physics penalty applies a loss to extracted proxy quantities according to the physical constraints they should satisfy.The loss may use a penalty or norm, with weighting coefficients for the constraints.Source: §4.2 · p. 4
4.2.1. Discrete Constant-Acceleration Constraint
- A discrete second-derivative constraint on optical flow enforces constant acceleration under time-invariant external forces across the five motion primitives.The resulting residual is presented as a differentiable, video-aligned form of Newtonian dynamics.Source: §4.2.1 · p. 5
4.2.2. Mass Conservation via Visual Features
- The mass-conservation residual compares generated and simulated visual features to encourage consistent object appearance and mass-dependent dynamics.The feature-level similarity constraint is intended to preserve appearance through a sequence and across domains.Source: §4.2.2 · p. 5
4.2.3. Post-Training Objective
- The final post-training objective combines the kinematic constraint and mass-matching term, using whole-frame optical flow under a fixed-camera, static-background dataset assumption.It requires no explicit acceleration or depth supervision and applies the constant image-plane acceleration principle across the five primitives.Source: §4.2.3 · p. 5
5. NewtonBench-60K
NewtonBench-60K is a controlled benchmark for five Newtonian motion primitives, with held-out in-distribution and out-of-distribution test videos. Its simulation setup, split protocol, and physics and visual metrics support evaluation of motion realism and consistency.
5. NewtonBench-60K
5.1. Simulation Pipeline and Canonical Setups
- The simulation pipeline uses a fixed side-view camera and provides RGB frames, instance masks, depth maps, and scene metadata for each clip.It combines Kubric scene orchestration, PyBullet dynamics, and Blender rendering; primitives are parameterized using physically meaningful parameters.Source: §5.1 · pp. 5–6
5.2. Benchmark Protocol: ID & OOD
- The held-out protocol tests 1K ID and 1K OOD videos per primitive, with OOD ranges holding out parameter bands for height, speed, angle, and ramp conditions.OOD evaluation can also perturb friction by ±25%.Source: §5.2 · p. 6
5.3. Evaluation Metrics
- Evaluation pairs velocity and acceleration RMSE with visual metrics L2, CD, and IoU on ID and OOD splits.Physics metrics assess constant-acceleration behavior in both image-plane axes; visual metrics capture trajectory position, mask distance, and per-frame overlap.Source: §5.3 · p. 6
6. Experiments
The experiments compare post-training strategies on NewtonBench-60K, then assess out-of-distribution generalization and ablate the kinematic and mass rewards. NewtonRewards improves benchmark metrics and motion consistency, while the ablations show the distinct roles of the two constraints.
6.1. Experimental Settings
- Experiments fine-tune OpenSora v1.2 on 50K NewtonBench-60K videos, using 32-frame clips at 16 fps for both SFT and post-training.RAFT supplies optical flow and V-JEPA 2 visual features; the three PISA-adapted baselines use flow, depth, and segmentation rewards.Source: §6.1 · p. 6, §6.2 · p. 6
6.2. Experimental Results
- NewtonRewards improves all five Table 2 metrics (average +9.75%) and all five primitives, with largest gains of +12.7% on parabolic throws and +11.7% on ramp-up.It also outperforms baselines on simple trajectories.Source: §6.2 · pp. 6–7, Table 2 · p. 7
- Ramp-slide-down comparisons show NewtonRewards maintaining stable grounding and smooth motion, unlike the inconsistent contact and jitter in SFT and PISA variants.The residual analysis also finds the smallest constant-acceleration residuals for NewtonRewards.Source: Figure 5 · p. 7, Figure 6 · p. 8, §6.2 · p. 8
- On the 5K OOD benchmark, NewtonRewards improves all metrics, including +7.01% reduction in L2, +7.38% improvement in CD, and +9.79% reduction in acceleration error despite no OOD post-training exposure.The test shifts include higher drop heights, faster throws, steeper ramps, and perturbed friction.Source: §6.2 · pp. 7–8, Table 3 · p. 8
6.3. Ablation Study
- Removing the discrete kinematic residual leaves only marginal improvement over SFT (+0.25%) and degrades kinematic accuracy despite slightly better spatial fidelity (IoU +3.8%).The ablation identifies the residual constraint as essential for motion stabilization and Newtonian alignment.Source: §6.3 · p. 8
- Removing mass conservation allows reward hacking: optimizing only the kinematic residual can drive velocities to zero and make objects disappear.With the mass reward, object persistence is maintained; without it, the videos show degenerate behavior.Source: §6.3 · p. 8, Figure 7 · p. 8
7. Conclusion
NewtonRewards applies verifiable rewards built from measurable proxies to enforce Newtonian consistency in video generation. The conclusion presents this approach as extensible wherever physical quantities can be estimated.
7. Conclusion
- NewtonRewards uses optical flow and visual features as proxies for velocity and mass to enforce Newtonian dynamics through kinematic and mass-conservation constraints.The paper concludes that estimable physical quantities can support explicit, differentiable constraints for physically valid generative behavior.Source: §7 · p. 8
Supplementary Material
The supplementary document adds real-world, ablation, and qualitative analyses alongside the main paper’s results. It also provides video comparisons across primitives and frame rates.
Supplementary Material
- A real-world evaluation on 361 PISA free-fall videos examines transfer from post-training developed entirely in simulation to natural scenes and real gravitational motion.The evaluation uses everyday dropped objects in varied indoor and outdoor scenes.Source: Supplementary Material · p. 13
- Additional quantitative evidence examines how mass conservation prevents reward hacking by avoiding degenerate zero-motion solutions.Source: Supplementary Material · p. 13
- Supplementary comparisons cover all NMPs and real-world sequences at 16, 8, and 4 fps, with extended frame-level visualizations.Source: Supplementary Material · p. 13
A. Real-World Experiment
The real-world experiment tests whether physics-grounded post-training developed in simulation transfers to natural free-fall footage. NewtonRewards shows the largest and consistent gains across visual and physics metrics in this evaluation.
A. Real-World Experiment
- The test set contains everyday objects dropped in varied indoor and outdoor environments, with variation in texture, lighting, and background clutter.The experiment is intended to assess transfer to real camera imagery and real-world dynamics.Source: A. Real-World Experiment · p. 13
- The comparison evaluates SFT, three PISA post-training variants, and NewtonRewards using L2, Chamfer Distance, IoU, RMSEv, and RMSEa.The same conditions are used across methods.Source: A. Real-World Experiment · p. 13
- NewtonRewards yields the largest and consistent gains across visual and physics metrics on 361 real-world free-fall videos.Source: A. Real-World Experiment · p. 13
B. Reward Hacking Mitigation
The ablation identifies mass conservation as a safeguard against residual-only reward hacking: without it, lower kinematic residuals can reflect frozen motion rather than more accurate Newtonian trajectories.
B. Reward Hacking Mitigation
- Without mass conservation, residual-only optimization drives velocity fields toward zero, producing videos in which objects barely move or disappear.Source: B. Reward Hacking Mitigation · p. 13
- Table 4 pairs residuals, where lower values indicate smoother motion closer to constant acceleration, with velocity magnitude as an indicator of meaningful motion.Source: Table 4 · p. 13
- More than 66%: removing mass conservation drops average velocity magnitude from 0.1017 to 0.0339 while decreasing vertical residual compared with SFT.Full NewtonRewards lowers residuals in both directions while maintaining non-trivial velocity.Source: B. Reward Hacking Mitigation · p. 13
C. Extended Qualitative Results
Extended NewtonBench-60K qualitative comparisons show NewtonRewards producing coherent motion across free fall, ramp sliding up, horizontal throw, and parabolic throw, unlike the inconsistent trajectories seen with SFT and PISA variants.
C. Extended Qualitative Results
- Table 5 covers a real-world experiment, listing visual metrics L2, CD, and IoU alongside physics metrics RMSEv and RMSEa.Source: Table 5 · p. 14
- In free fall, NewtonRewards produces stable vertical paths with consistent acceleration and minimal lateral drift, unlike jittering, hovering, or unstable trajectories under SFT and PISA variants.Source: Figure 9 · p. 14
- On ramp slide up, NewtonRewards yields smooth upward motion with realistic friction and no frame-to-frame jitter, while other strategies show inconsistent contact dynamics or object disappearance.Source: Figure 10 · p. 15
- In horizontal throw, NewtonRewards maintains constant horizontal velocity and stable parabolic descent, while SFT and PISA variants lose velocity, drift, or jitter.Source: Figure 11 · p. 16
- In parabolic throw, NewtonRewards generates a stable arc with consistent horizontal velocity and gravity-governed vertical acceleration, unlike inconsistent or flattened trajectories from other strategies.Source: Figure 12 · p. 17