Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space

1 INTRODUCTION

• High-dimensional learning can capture signal regularities, but causal modeling is also important in physics and may support more robust out-of-distribution predictions.The introduction notes that physics-inspired methods have often imposed causal structure through prior knowledge of physical laws.Source: §1 · p. 1
• The model predicts long-horizon counterfactual video in pixel space, addressing prior work limited to discrete outcomes or supervised 3D trajectories.The target is 150 frames, approximately six seconds.Source: §1 · p. 2
• A hybrid representation combines 2D keypoints with per-point coefficients for complementary shape and appearance information.The model estimates confounders from this representation and forecasts keypoint trajectories from a counterfactual observation.Source: §1 · p. 2
• The proposed model outperforms strong baselines for physics-informed video prediction.The authors identify the representation and higher-dimensional dynamic model as key design choices for long-term prediction.Source: §1 · p. 2
• The new benchmark evaluates pixel-space predictions after interventions on initial conditions and controls causal identifiability and counterfactual variables.Interventions include displacing, re-orienting, or removing objects.Source: §1 · p. 2

2 RELATED WORK

• CLEVRER asks counterfactual questions about observed videos, while CoPhy separates observed and counterfactual experiments but requires supervised 3D object positions.Other cited benchmarks address unknown interaction laws or wind-driven folding and require identifying counterfactual variables and predicting future frames.Source: §2 · p. 2
• Video prediction has used recurrent models and GANs, while integrating causal reasoning and physical priors into these models remains nonstraightforward.Some approaches disentangle physics-informed and residual features or incorporate scene priors.Source: §2 · p. 2
• PhyDNet separates visual and dynamical features intended to follow a PDE, but the authors report that it fails on their benchmark.The cited work achieves strong performance on Human3.6M and Sea Surface Temperature.Source: §2 · p. 3
• Unsupervised keypoints can provide object-centric representations for physical or geometric priors, but prior stochastic keypoint dynamics may be unsuitable for physics counterfactuals over long horizons.The section also situates keypoints in robotics and reinforcement learning, where agents interact with objects.Source: §2 · p. 3

3 THE FILTERED-COPHY BENCHMARK

• Each experiment pairs an observed trajectory AB from initial condition A with a counterfactual trajectory CD from intervention-modified initial condition C.The task is to infer D from AB and C while confounders are generated but not used during training or testing.Source: §3 · p. 3
• Counterfactual prediction follows abduction of unchanged physical parameters from AB, action applying the intervention to replace A with C, then prediction of D.The abduction, action, and prediction stages are specified across the benchmark description.Source: §3 · p. 3
• The benchmark has three rigid-body scenarios: cube towers with mass confounders, moving-sphere collisions, and sphere-cylinder collisions with mass and initial-velocity confounders.The figure depicts 2D or 3D scenes and the intervention from A to C.Source: §3 · p. 3, Figure 1 · p. 4
• Filtered-CoPhy evaluates RGB pixel-space predictions after visible interventions such as moving or removing objects, rather than predicting 3D positions.Confounders require analysis of the full observed trajectory AB; they cannot be identified from the single-frame A.Source: §3 · p. 3
• Experiments are constrained so confounders can be retrieved from AB and do not leave identical observed trajectories paired with different counterfactual outcomes.The benchmark also excludes cases where changing confounder values has no significant effect on the final outcome.Source: §3 · pp. 3–4
• The simulations use 25 FPS rather than CoPhy’s 5 FPS because brief collisions can fall between 5-FPS samples and become difficult to infer.The trajectory figure contrasts black 5-FPS samples with colored 25-FPS samples around cube-ground collisions.Source: §3 · p. 4, Figure 2 · p. 4

4 UNSUPERVISED LEARNING OF COUNTERFACTUAL PHYSICS

• The model predicts long-horizon counterfactual visual sequences from observed and counterfactual experiences, requiring no other supervision.Its three components learn a latent representation and its counterfactual dynamics.Source: §4 · p. 4

4.1 DISENTANGLING VISUAL INFORMATION FROM DYNAMICS

• The encoder combines dense features, 2D keypoints, and per-keypoint coefficients; without state supervision, the representation encodes position in keypoints and appearance and orientation in coefficients.Reconstruction uses source-image dense features with target-image keypoints and coefficients, encouraging static and dynamic information to separate.Source: §4 · p. 5, §4.1 · p. 5

4.2 DYNAMIC MODEL AND CONFOUNDER ESTIMATION

• CoDy uses graph networks and recurrent processing to estimate latent confounders from observed keypoint-and-coefficient trajectories and forecast counterfactual outcomes in a higher-dimensional state space.The dynamics are forecast autoregressively in the projected space before predictions are mapped back to keypoint space.Source: §4 · p. 5, §4.2 · pp. 6–7

4.3 TRAINING

• Training separates representation learning from dynamics learning: the encoder and decoder are trained on reconstruction, then the keypoint detector is frozen while CoDy learns to forecast keypoints.Joint end-to-end training is described as numerically unstable and prone to regression to the mean.Source: §4.3 · p. 7

Experimental setup

• Counterfactual models receive observed sequence AB and counterfactual initial condition C to forecast D, while non-counterfactual baselines predict the video from one frame.Performance is measured with time-averaged PSNR and localized PSNR on regions near moving objects.Source: §5 · pp. 7–8

Comparison to the SOTA

• Filtered-CoPhy consistently and significantly outperforms UV-CDN, PhyDNet, and PredRNN on the counterfactual video-prediction scenarios.PredRNN is competitive on CollisionCF, but localized PSNR indicates that its foreground reconstruction is weaker; Figure 6 visualizes predictions over time.Source: §5 · p. 8, Figure 6 · p. 9

Performance on real-world data

• The appendix reports experiments on 516 videos of real wooden blocks.The videos were introduced in Lerer et al. (2016).Source: §5 · p. 8

Impact of appearance coefficients

• Additional orientation and shape coefficients significantly improve on a keypoint-only representation, which more keypoints cannot compensate for.Table 3 compares the representations at N, 2N, and 4N keypoints.Source: §5 · p. 8, Table 3 · p. 8

Learning filters

• Learning the filter bank from scratch mildly reduces static reconstruction performance compared with the handcrafted bank.The comparison is made without the dynamic model.Source: §5 · p. 9, Table 5 · p. 9

Impact of the high-dimensional dynamic space

• The CoDy encoder’s projection to 256 dimensions performs better than reasoning directly in keypoint-plus-coefficient space.Table 4 reports MSE×10^-3 for prediction of keypoints and coefficients at four points.Source: §5 · p. 9, Table 4 · p. 9

6 CONCLUSION

• The benchmark targets long-horizon pixel-space counterfactual prediction, and the unsupervised model uses keypoints with latent vectors for appearance and shape without object-property supervision.The paper reports persistent failures to maintain rigid object structure over long prediction timescales.Source: §6 · p. 9

A FURTHER DETAILS ON DATASET GENERATION

• Masses are discretized to 1 or 10, and initial velocities are also confounders for BallsCF and CollisionCF.Trajectories for possible mass combinations are simulated from the same initial condition.Source: A FURTHER DETAILS ON DATASET GENERATION · p. 13
• Interventions remove or shift objects in BlocktowerCF and BallsCF, while CollisionCF uses shifts and cylinder-orientation changes.BlocktowerCF shifts keep the cube near the tower to maintain contact.Source: A FURTHER DETAILS ON DATASET GENERATION · p. 13

A.1 ENFORCING THE IDENTIFIABILTY CONSTRAINT

• The identifiability constraint rejects experiments when a different confounder setting yields similar observed AB but a different counterfactual CD; threshold ε must be tuned.The authors choose ε = 100 because it optimizes discrimination and rejects the highest number of unidentifiable trajectories.Source: §A.1 · pp. 13–14, Figure 7 · p. 14
• Adding the identifiability constraint significantly improves mass-estimation accuracy, with corrected accuracy counting only cubes whose masses directly affect trajectory CD.Table 6a presents the sanity check of the constraint.Source: §A.1 · p. 14, Table 6 · p. 14

A.2 ENFORCING THE COUNTERFACTUALITY CONSTRAINT

• The counterfactuality check requires at least one mass change to produce a different counterfactual trajectory.Candidate masses are tested one at a time against the candidate trajectory.Source: §A.2 · p. 14

A.3 ANALYZING TEMPORAL RESOLUTION

• At 25 FPS, trajectory prediction has the reported best trade-off between representing collisions accurately and providing training data.The check predicts the next second of a trajectory from the preceding second; Table 6b reports errors after a one-second rollout, and Figure 9 illustrates temporal-resolution effects.Source: Table 6 · p. 14, §A.3 · p. 15, Figure 9 · p. 15

A.4 SIMULATION DETAILS

• The PyBullet generation process samples physical configurations, searches for initial conditions and valid interventions, and rejects experiments without a valid intervention.Sampling uses 0.04-second steps and 448 × 448 video; BlocktowerCF and BallsCF span 6 seconds, CollisionCF 3 seconds, and BlocktowerCF stable counterfactual towers are kept near 50% per confounder configuration.Source: §A.4 · pp. 15–16, Figure 10 · p. 16

B.1 IMAGE RECONSTRUCTION

• Adding appearance coefficients improves reconstruction by over 2 dB at N keypoints; increasing keypoint count raises PSNR but can degrade forecasting.Doubling the number of keypoints only slightly improves reconstruction accuracy.Source: §B.1 · p. 16, Table 8 · p. 17

B.2 NAVIGATING THE LATENT COEFFICIENT MANIFOLD

• Keypoints control cube position, while unsupervised coefficients encode orientation, camera distance, and size or presence, with some disentanglement.The vertical position also provides a shortcut cue for 3D shape.Source: §B.2 · p. 17

C.1 COMPARISON WITH OUR DE-RENDERING MODEL

• Although Transporter also uses dense target features, this model outperforms it at small keypoint counts; at higher counts Transporter may place keypoints far from objects.Transporter combines target keypoint positions with dense target features, so the models are not directly comparable.Source: §C.1 · p. 18

C.2 ANALYSIS OF TRANSPORTER’S BEHAVIOR

• With keypoints fixed, Transporter still reconstructs substantial image content, indicating that dense target features encode part of the dynamics and can make keypoints unnecessary.The evaluation fixes the source image and substitutes source keypoints for target keypoints; it does not involve dynamic forecasting.Source: §C.2 · p. 19

C.3 TEMPORAL INCONSISTENCY ISSUES

• Transporter keypoints may shift from a cube’s edge to its face over time; the proposed coefficients limit keypoint count and help track unsupervised centers of mass.The reported Transporter inconsistency occurs with as few as four keypoints.Source: §C.3 · p. 20, Figure 15 · p. 21

D.1 DE-RENDERING MODULE

• The de-renderer extracts keypoints and per-point coefficients, maps coordinates to Gaussian maps, and deforms them with coefficient-controlled filters; a zero gate can deactivate a keypoint..Source: §D.1 · pp. 20–21

D.2 CODY

• CoDy estimates confounders from AB states, encodes CD’s initial state in a latent dynamics space, recursively forecasts there, then decodes the keypoint trajectory.Its state representation stacks each keypoint’s coordinates and coefficients.Source: §D.2 · pp. 22–23

E.1 MULTI-OBJECT TRACKING METRICS

• With one keypoint per object, detections naturally land near object centers of mass; MOTP measures paired-location precision and MOTA reflects missed objects and false positives.Effective tracking requires both low MOTP and high MOTA.Source: §E.1 · p. 23
• The method also outperforms CoPhyNet on ballsCF, where the authors suggest its 2D setting removes CoPhyNet’s 3D-position advantage and CoDy’s latent state may ease collision dynamics.CoPhyNet uses ground-truth 3D positions and is described as a soft upper bound; the authors also report superiority over UV-CDN in keypoint space.Source: §E.1 · p. 23, Table 11 · p. 24

E.2 IMPACT OF THE DO-OPERATIONS

• 3 dB favors the Move do-operation; overall, forecasting generalizes across operation types and displacement amplitudes in the balanced benchmark.The plotted results use models with 2N keypoints.Source: Figure 16 · p. 24, §E.2 · p. 24

F EXPERIMENTS ON REAL-WORLD DATA

• Blocktower IRL contains 516 videos of real wooden towers with two to four cubes, and the task predicts their dynamics in pixel space.The dataset is non-counterfactual and contains only a few videos.Source: F EXPERIMENTS ON REAL-WORLD DATA · p. 25
• To adapt to the limited real-world data, the authors fine-tune models pretrained on BlocktowerCF; for CoDy they disable confounder estimation and freeze some dynamic-model layers.The de-renderer uses four keypoints and five coefficients per keypoint.Source: F EXPERIMENTS ON REAL-WORLD DATA · p. 25
• 26.27 dB average PSNR is reported when predicting 20 future frames from a single sampled image in Blocktower IRL.The authors describe this as the first use of the dataset for video prediction and say the PSNR is of the same order of magnitude as their simulation results.Source: F EXPERIMENTS ON REAL-WORLD DATA · p. 25, Figure 17 · p. 25

• Figure 18 presents qualitative model performance on the BlocktowerCF (BT-CF) benchmark.Source: Figure 18 · p. 26
• Figure 19 presents qualitative model performance on the BallsCF (B-CF) benchmark.Source: Figure 19 · p. 27
• Figure 20 presents qualitative model performance on the CollisionCF (C-CF) benchmark.Source: Figure 20 · p. 28