Source-linked AI summary
CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations
Aman Mehta, Riya Baviskar
TL;DR
Pretrained visual encoders are often evaluated for physical competence in clean scenes, but such assessments may not reveal whether representations infer physics or exploit visual shortcuts. CALIPER tests this by calibrating on two observed collisions, predicting a third truncated collision, and swapping calibration evidence; nuisance variation separates representations while clean scenes do not, motivating empirical benchmark checks.
Problem
Clean fixed-camera perturbation benchmarks and linear probes cannot reliably distinguish physics inference from visual shortcuts or mere property decodability.
Method
CALIPER shows two calibration collisions for an object with unknown mass and friction, truncates a third collision at contact, and tests evidence use by swapping calibration clips across objects.
Results
+0.50 R2 from calibration is removed by swapping, while nuisance variation spreads representations across 0.498 R2 and V-JEPA 2 misses goal distances by 4.0 mm versus 19.6 mm for a random ViT.
Takeaways & Limitations
Whether a benchmark can rank representations must be demonstrated with effect-size audits, baseline spreads, and swap or erasure tests rather than assumed from clean-scene scores.
Takeaways & Limitations
The study uses a small simulated setting with rigid boxes and one contact event, and nuisance variation changes several visual factors together, preventing attribution to any single factor.
Abstract
from arXiv · showhide
How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are increasingly used as the perception front end of world models for manipulation, and their physical competence is assessed with perturbation benchmarks and linear probes, almost always in a clean, fixed-camera scene. We show that these assessments cannot distinguish an encoder that infers physics from one that does not. CALIPER (calibrate, then predict) is a direct test: an object of unknown mass and friction is struck twice at known speeds, a third strike is shown only up to the moment of contact, and a linear readout on frozen features must predict how far the object slides. Swapping in another object's calibration clips checks that the evidence is actually used. Across 2,000 simulated episodes and eight representations, from V-JEPA 2 to a randomly initialised ViT and raw pixels, calibration adds +0.50 R^2 and the swap removes it. Yet in the clean scene every representation lands within 0.02 R^2 of the ceiling set by true simulator state, because a fixed camera exposes the object's displacement directly in pixel coordinates. Resampling camera, lighting, and clutter for every clip spreads the same representations across 0.50 R^2; when the readout chooses a push speed for a goal distance, V-JEPA 2 misses by 4 mm and the random ViT by 20 mm, no better than ignoring the object. Linear probes track none of this: a change in frame aggregation moves a probe more than pretraining does, and erasing the probed mass direction from the same representation costs nothing in one scene and 0.35 R^2 in the other. Whether a benchmark can rank models is an empirical property, and we give three checks that establish it.
1 Introduction
CALIPER argues that common clean-scene evaluations cannot distinguish physics-inferencing encoders from untrained or shortcut-based representations. It tests this with calibration-conditioned prediction, swapped evidence, and scene variation.
- Clean-scene perturbation tests and linear probes give nearly identical scores to encoders, including one with never-trained weights.
- CALIPER shows two full calibration collisions before truncating a third query collision at contact, requiring prediction of the resulting slide.
- Swapping in another object’s calibration clips tests whether prediction uses the target’s own interaction history rather than a surviving shortcut.
- +0.50 R2 from calibration and complete removal of that gain by swapping show that the readout uses object-specific interaction history.
- 0.020 R2 is the entire clean-scene spread across representations, because fixed-camera displacement becomes a recoverable pixel-coordinate pattern.
- The paper concludes that benchmark ranking ability must be demonstrated empirically, using three checks that the starting evaluations lacked.
2 Related Work
Related work frames CALIPER as an evaluation of visual physics inference rather than downstream control or property decodability alone. Its calibration clips provide an identification opportunity, while swapping tests whether that opportunity was used.
- Downstream control success mixes perception, physical inference, and policy optimisation, whereas frozen linear readouts isolate only linearly available information.
- Existing physics benchmarks report prediction correctness without checking whether perturbations change outcomes or whether trivial baselines could rank the compared models.
- CALIPER turns inferring object properties from motion into an evaluation: calibration clips provide the identification opportunity, and swapping tests whether it was taken.
- Visual physics erasure extends decodability-versus-use analyses by showing that the same property’s erasure cost changes with the scene.
- Figure 1 depicts calibration collisions revealing unknown physics, followed by a contact-truncated query whose displacement must be predicted.
3 Calibration-Conditioned Prediction
The task uses simulated collisions to expose mass and friction through calibration while preventing the query outcome from appearing in the input. Clean and nuisance scenes keep physics and readout fixed while changing visual conditions.
- Targets vary in mass and sliding friction while appearance variables are sampled independently, preventing visual identity from directly revealing the physical parameters.
- Two collisions at 0.45 and 0.75 m/s are shown fully, while a query collision at 0.40–0.85 m/s is truncated at first contact.
- The regression target is log displacement during the query.
- The clean scene fixes camera, lighting, and floor, whereas the nuisance scene resamples viewpoint, lighting, floor tint, and distractors for every clip.
- Physics, parameter distributions, and readout are identical across scenes, so changes between them are attributed to scene conditions rather than task design.
- The frozen-encoder readout uses PCA, standardisation, concatenation with query speed, ridge regression, and shared component budgets across nested input conditions.
4 Results
In the clean scene, calibration verifies that readouts use interaction history but cannot distinguish representations because fixed-camera displacement is directly recoverable from pixels. Nuisance variation separates representations, preserves their ordering across training sizes, and exposes differences in control error and in whether decoded properties are used.
- Clean scene: 0.020 R2 is the clean-scene spread of the +calibration column, with raw pixels at 0.996, V-JEPA 2 at 0.995, and the random ViT at 0.977.The clean-scene ceiling arises because fixed-camera target displacement is a fixed pixel-coordinate pattern recoverable by linear readouts.
- Clean scene: 13 of 20 perturbation cells change the outcome by less than 10^-4 without a model, so the audited sensitivity benchmark fits integrator noise.This failure could have been detected before evaluating any representation.
- Nuisance scene: 0.498 R2 is the nuisance-scene spread, where V-JEPA 2 reaches 0.974 and the random ViT reaches 0.476 against a speed-only floor of 0.514.The swap still removes every calibration gain, and the ordering holds at every training-set size.
- Data efficiency: 75 episodes preserve the full-data nuisance ordering, led by V-JEPA 2 at 0.716 and ending with the random ViT at 0.017.The clean-scene differences at 75 episodes are not treated as a claim because they come from small-sample fits, whereas nuisance ordering matches the full-data ordering.
- Control-shaped evaluation: 4.0 mm is V-JEPA 2’s nuisance-scene median goal miss, compared with 19.6 mm for the random ViT and 19.0 mm when the object is ignored.In the clean scene, every representation lands within a few millimetres of the goal, at the readout floor of 2.9 mm with true (m, µ).
- Probes and use: 0.000 is the random ViT’s prediction change when frame concatenation raises its friction probe from 0.361 to 0.672.Erasure also shows scene dependence: raw-pixel mass erasure costs −0.001 in the clean scene but −0.350 under nuisance variation.
- Benchmark checks: Three checks are proposed: audit no-model effect sizes, report spread against random features and raw pixels, and pair property claims with swaps or erasures.These checks test whether perturbations matter, whether scenes can rank representations, and whether decodability reflects use.
5 Limitations
The evaluation uses a deliberately small simulated setting and shared frozen-encoder readout, limiting how broadly its numerical findings and model rankings should be interpreted.
- The experiments use rigid boxes, one contact event per clip, and a simulator, so the reported numbers characterize this restricted setting.
- A frozen encoder with ridge regression measures only linearly available information, providing a lower bound on what fine-tuning could extract.
- The nuisance scene varies viewpoint, lighting, and clutter together while also making targets smaller, so the performance drop cannot be attributed to one factor.
- The study compares eight representations under a shared readout rather than native rollout machinery, so models using physics differently could score differently.
- The erasure intervention removes only linearly decodable structure, and the audited perturbation benchmark may not represent defects in published benchmarks generally.
A Effect-size audit
The effect-size audit shows that many perturbations barely change benchmark outcomes, while implementation failures can silently substitute random features for intended encoders.
- 13 of 20 perturbation cells are inert, with outcome changes below 10^-4 in three task-specific or implementation-related cases.
- Free-fall outcomes are geometrically fixed, so mass and friction do not affect the recorded distance to the goal.
- The rigidity perturbation was discarded because a mixed MuJoCo parameterization caused the simulator to use its default value.
- Stacking was scored by final upper-object height, which geometry fixes despite changes in bounce, settling, or toppling dynamics.
- A failed checkpoint load substituted randomly initialized networks in another codebase, making a reported physics-aware model random features.
B Environment design details
The environment and readout are designed to make friction-related sliding observable, preserve answerability under nuisance variation, and support controlled linear evaluation.
- Collisions avoid the force-threshold problem that would leave some independently sampled mass-force pairs stationary or out of frame.
- Boxes are used instead of spheres because matched-physics spheres travelled 5.6× farther and showed much weaker friction-displacement correlation.
- Impact speeds and friction are chosen so slides last about 0.4 seconds across roughly 13 rendered frames, with displacements constrained to 12–280 mm.
- Camera pose, lighting, floor tint, and distractors are resampled per clip, while a wider field of view keeps the workspace visible.
- Features are reduced to a shared PCA component budget, standardized, and fit with ridge regression under repeated 5-fold cross-validation.
- Changing the cross-validation seed moves every reported quantity by at most 0.01, supporting seed robustness.