Source-linked AI summary
Teaching Vision-Language Models to Use the Scale They Are Given: Label-Free Equivariance Training for Metric Physical Reasoning
Kaizhen Tan, Yang Feng, Heqing Du, Siru Tao, Xin Xu, Hanzhe Hong
TL;DR
Vision-language models only partially use supplied metric scale, despite recovering scale-free physics laws. EquiSD projects their own predictions onto the exact scale-equivariant relation and fine-tunes without annotations, improving scale use and transfer to real video.
Problem
Vision-language models underuse supplied metric scale in physical video questions, while scale-free controls indicate that their physical mechanism knowledge remains intact.
Method
EquiSD projects a model’s prediction onto the scale-equivariant family and fine-tunes on the resulting self-generated targets without ground-truth answers or annotations.
Results
EquiSD recovers 93% of what exact simulator answers buy when trained on synthetic clips and evaluated on real video, compared with 66% for simulator supervision.
Takeaways & Limitations
The scale relation provides label-free supervision that transfers from rendered blocks to real footage and generalizes to world scales absent from training.
Takeaways & Limitations
The equivariance relation applies only when video fixes the time base, supplied anchors are lengths or length rates, and the target has dimension L T −k.
Abstract
from arXiv · showhide
Metric questions about video require vision-language models to use supplied real-world references to convert visual measurements into physical units. Yet we find that current models use this scale information only partially. When every world-space quantity in a prompt is rescaled by a common factor, the video remains equally valid and the correct answer changes by exactly that factor, but model predictions move only part of the way and accuracy remains concentrated near the familiar scale of the depicted objects. Across eight vision-language models, this under-response persists over four orders of magnitude. The same models recover the correct closed-form scaling laws when the identical physics is asked in a scale-free form, indicating that the main deficit lies in metric grounding rather than physical mechanism knowledge. We use this exact scaling relation as supervision without requiring metric annotations. Under a common rescaling of the supplied world-space quantities, the correct metric answer must change by the same factor. EquiSD exploits this constraint by projecting a model's own prediction onto the scale-equivariant family and fine-tuning the model on the resulting targets. It requires no ground-truth answers and only one model query per training video. On held-out simulated videos, EquiSD increases a 3B model's median response slope from 0.66 to 0.94 and improves mean relative accuracy by 9.2 points across scales. The learned relation generalizes to unseen world scales and transfers without adaptation to real QuantiPhy videos, where accuracy increases by 6.4 points. These results show that an exact physical symmetry can provide label-free supervision for improving metric grounding in vision-language models.
1. Introduction
Metric video questions require models to convert visual measurements into physical units using supplied references, yet models only partially use that scale. The paper identifies an exact scaling symmetry and turns it into label-free supervision through EquiSD.
- Metric video questions require answers in metres and seconds, while the gap to human performance remains wide.
- A supplied real-world reference converts pixels into metres, making the correct answer proportional to that reference.
- Across eight vision-language models, predictions under-react when all supplied world-space quantities are rescaled, while accuracy remains concentrated near familiar object scales.
- A scale-free control shows the deficit reflects metric grounding rather than general physical reasoning, because models reproduce closed-form laws when physics is expressed as ratios.
- The relation y(V, Sαq) = α y(V, q) provides an exact annotation-free constraint because the answer is homogeneous of degree one under common rescaling.
- EquiSD projects the model’s own answer onto the scale-equivariant family and fine-tunes on the transported target without ground-truth answers.
- EquiSD improves transfer from synthetic training to real video, recovering 93% of what exact simulator answers buy against 66% on simulated video.
2. Related work
Related work reconstructs executable worlds, studies interventions and counterfactuals, or imposes units consistency in structured models. This paper instead repairs the mapping from supplied references to metric answers using a scale relation as a black-box training signal.
- Reconstructing an executable world: Executable-world systems reconstruct scenes or simulators before answering metric physical questions.
- Interventions and counterfactuals: Intervention and counterfactual benchmarks usually evaluate categorical or trajectory-level outcomes after changing physical conditions.
- Interventions and counterfactuals: This paper uses interventions as a control separating mechanism knowledge recovered by scale-free questions from metric grounding in metre-based questions.
- Units equivariance and consistency: Classical units-equivariant methods build symmetry into white-box models, whereas this work measures black-box departures and uses them as training signals.
- Units equivariance and consistency: Unlike ordinary self-distillation or agreement-based consistency, the rewriting fixes the ratio between answers, allowing one query to determine a target.
3. Metric scale as a constraint
Metric physical reasoning is scale-ambiguous without a world-space reference, but projective geometry imposes an exact degree-one scaling relation. Experiments show models only partially use supplied scale, despite retaining scale-free physics knowledge.
- Scale ambiguity: A monocular video needs a world-space reference because identical pixels can arise from globally rescaled scenes.The reference and, for depth-aware questions, camera distances convert visual measurements into world units.
- Homogeneity relation: Under Sα, which scales every supplied world-space quantity while preserving the video, targets with dimension L T^-k satisfy y(V, Sαq) = α y(V, q).The relation holds at a fixed video time base for the target classes studied, while dimensionless outcomes and pure times are excluded.
- Measurement: A consistent rescaling probe gives a median response slope of 0.76 and per-item R2 of 0.90, versus 0.14 and 0.46 for changing the reference alone.Every reported probe number uses the consistent rewriting, which scales camera distances together with the reference.
- Model behavior: Across eight models, log-linear response fits achieve median per-item R2 values from 0.81 to 0.96, with slopes lowest on in-the-wild footage and below 0.4 for six models on speed-to-size questions.The slope equals 1 when the model fully uses the supplied reference and 0 when answers rely entirely on remembered object magnitudes.
- Controls: Changing notation from 1 cm to 10 mm leaves the scene unchanged, yet 13%–30% of answers remain intact and 40%–58% move by more than twofold.The scale-free output also remains reference-dependent, indicating that the deficit is not confined to the metric output format.
4. Equivariance self-distillation
EquiSD turns exact scale equivariance into label-free self-distillation by projecting model responses onto the equivariant family and fine-tuning on those projections. Using K=1 requires one query per video while preserving the projected scale relation.
- E-step: project onto the equivariant family: EquiSD first projects the model’s own responses onto functions satisfying the scale-equivariance constraint, then fine-tunes the model on that projection.The method uses no ground-truth answers or annotations.
- E-step: project onto the equivariant family: The equivariant family is determined by a scale-free ratio R, estimated robustly as the median over a log-symmetric scale grid.The median is the ℓ1 projection in log space and tolerates occasional format or magnitude failures.
- M-step: fit the projection: The projected target at scale αρ is αρ R̂, and fine-tuning uses prompts rewritten with the same Sα while restricting loss to answer tokens.Camera distances move with the reference, so the training prompt remains a consistent description of the video.
- Choosing K: K=1 reduces the projection to transporting the nominal-scale answer, with the grid and single-query estimates differing by a median of 0.003 dex.The procedure therefore costs one model generation per training video.
- Implementation: The reported implementation fine-tunes Qwen2.5-VL-3B-Instruct with LoRA adapters for one epoch after one generation per training video.It uses six video frames and 4-bit NF4 quantization.
5. Experiments
Experiments show that EquiSD improves scale use on held-out simulated videos, generalizes to unseen scales, transfers to real QuantiPhy video, and depends on transporting targets with the prompt.
- 5.1. Setup: The study renders 1,000 rigid-body videos across five dynamics families and constructs 3,000 QuantiPhy-style questions, evaluating held-out scenes across a dynamics-family boundary.Evaluation uses paired items, with mean relative accuracy (MRA) as the accuracy metric.
- 5.1. Setup: EquiSD queries each training question once, projects the nominal answer onto the scale-equivariant family, and fine-tunes on targets transported across five scales.The main scale grid is α ∈ {0.05, 0.2, 1, 5, 20}; the supervised comparison uses identical target placements with simulator answers.
- 5.2. Label-free equivariance training: The median response slope rises from 0.66 to 0.94 on held-out simulated video, while equivariance error falls by two fifths.A slope of 1 represents faithful scale response and zero equivariance error is exact.
- 5.2. Label-free equivariance training: 9.2 MRA points are gained across a 400-fold world-scale range, with the trained accuracy profile becoming close to flat instead of peaking near the familiar scale.The held-out evaluation uses sliding-block scenes, a dynamics family absent from training.
- 5.2. Label-free equivariance training: Unseen-scale evaluation supports a continuous relation: gains are at least as large outside the trained range as on trained scales.The checkpoints are tested at six disjoint scales, including 0.02 and 50, extending beyond the training range.
- 5.3. Transfer to real video: Without adaptation, transfer to real QuantiPhy video raises accuracy by 6.4 MRA and median slope from 0.33 to 0.57, but leaves more than a third of supplied scale unused.Training used rendered blocks and balls, while QuantiPhy contains real cars, people, billiard tables, and boats.
- 5.4. Ablations: Target transport, rather than self-distillation alone, accounts for the gain: fixed targets drive the slope to 0.06, while ordinary self-distillation transfers only 1.4 MRA versus EquiSD’s 6.4.Exact labels show the same dependence on transport, isolating the target-moving rule from the number source.
- 5.4. Ablations: The nominal-scale single-probe E-step is preferred because it is cheaper and retains most of the gain, although the five-point-grid variant reaches slope 1.00 versus 0.94.The two ratio estimates agree to 0.003 dex; the difference comes from the selected training pool.
6. Limitations
The scaling relation applies only when video-fixed time bases, supplied length or length-rate anchors, and targets with compatible dimensions are present. Equivariance alone does not ensure correctness, and the repair evaluation covers limited models, data, and dynamics.
- Scope of the scaling relation: The relation holds only for a video-fixed time base, length or length-rate anchors, and targets with dimension L T −k.Questions outside these dimensional and anchoring conditions require a different relation.
- Scope of the scaling relation: Pure-time or dimensionless-ratio answers have no such constraint, nor do answers relying on an unsupplied dimensional constant.Extending the method beyond this boundary requires a different relation rather than a wider α grid.
- Accuracy versus consistency: Exact equivariance can coexist with uniformly wrong answers, so the paper reports accuracy alongside response slope.Inference-time constraint recovery alone does not necessarily improve accuracy.
- Evaluation scope: The repair is evaluated on one 3B student trained on one synthetic corpus, while simulated testing uses one sliding-block dynamics family held out from four others.The authors present this as a bounded evaluation rather than a within-distribution estimate.
7. Conclusion
The paper treats proportional scaling in monocular metric questions as both a diagnostic and a source of label-free supervision. EquiSD uses projected model responses to improve scale consistency without annotations and transfers from rendered blocks to real footage.
- Conclusion: Models pass between 0.69 and 0.94 of supplied scale into their answers, with accuracy concentrated near familiar object scales.The paper first uses the scaling symmetry as an instrument for measuring metric grounding.
- Conclusion: EquiSD trains on model responses projected onto the scale-equivariant function family, requiring no annotations.The projected targets encode the symmetry rather than externally supplied magnitudes.
- Conclusion: The procedure runs in 32 minutes on a laptop GPU and transfers from rendered blocks to real footage.Its targets carry the scaling constraint without simulator-specific exact answers.
A. Full result tables
The appendix documents evaluation protocols, model-query settings, response parsing, and full-sweep comparisons. It also distinguishes valid joint rescaling from invalid reference-only changes and reports a ratio-prediction control.
- Full result tables: Table captions in Tables 7 and 8 provide the additions that resolve Tables 1 and 6, including the transfer block underlying Section 5.3 retention figures.Each caption specifies what its table contributes.
- Full result tables: The eight models use OpenRouter at temperature 0 with 8192-token budgets and 16 frames, while the local 3B student uses six frames at a 280-pixel longest side.These settings support comparability across the authors’ own student arms, not necessarily across external systems.
- Full result tables: Response slope is the ordinary least-squares coefficient of log10 ŷ on log10 α, fitted on positive parsed answers with at least two distinct scales.Associated R2 and residual σ come from the same per-item fit; paired differences use 6,000 bootstrap resamples.
- Full result tables: The shared system prompt instructs models to trust supplied references even when they differ from typical real-world magnitudes.This makes rescaling a faithfulness probe rather than a plausibility judgment.
- Full result tables: Direct prompting supplies frame count, duration, reference units, optional depth context, the question, and a required Final answer line.The protocol follows the QuantiPhy format.
- Full result tables: The No reference condition removes the numeric reference and asks for the model’s best estimate, measuring the magnitude inferred from video and prior knowledge.This value is used by the test-time correction procedure.
- Full result tables: Ratio prompting requests answer divided by the reference value and states that the ratio is fixed by the video alone.The units are named explicitly to disambiguate the requested quantity.
- Full result tables: Responses cut off because of length are scored as failures rather than parsed, because extracted numbers may be intermediate quantities.Otherwise, parsing takes text after the last Final answer marker, with fallbacks to the last line or number.
C.2. Asking for the ratio instead
Asking for the scale-free ratio and multiplying afterward performs worse than requesting the metric answer directly, while the ratio itself still varies with the reference.
- C.2. Asking for the ratio instead: For Qwen3-VL-8B, ratio prompting scores 14.0 macro-MRA versus 27.2 for the metric answer.The reported ratio also moves with the reference by a median of 0.72 dex across the probe grid.
D. Correcting a frozen model at test time
Test-time correction uses the model’s own scale-response estimates to adjust predictions toward scale-consistent answers, with selective correction reducing the accuracy cost of intervention.
- Estimator: The estimator fits responses as ŷ(α) = c α^s and uses the model’s own answers to infer the scale where remembered and supplied magnitudes agree.The ratio is exact when the response is log-linear.
- Test-time correction: Correcting every IVP-Sim question costs 4.7 MRA relative to a single direct call.
- Test-time correction: Selective correction leaves the corrected model 0.2 MRA below a single direct call, and no sweep threshold exceeds direct-call performance.The threshold is chosen on one half of the scenes and evaluated on the disjoint other half.
- Estimator: The estimate’s variance grows as 1/s^2, making low-slope estimates less stable.
E. IVP-Sim
IVP-Sim is a randomized simulated corpus designed to test physical scaling laws across varied dynamics, rendering conditions, and world scales, while excluding one uncertifiable outcome.
- Corpus design: Each IVP-Sim scene randomizes dynamics, camera pose, field of view, lighting, floor appearance, object colour, and overall world scale over 10^±0.5.Scenes render at 640 × 480 and 25 fps for 1.6 to 3.0 seconds; derived outcomes come from a 200 Hz state log.
- Validation boundary: One ramp-block acceleration outcome is excluded because its estimated median recovers only 0.71 of the closed-form expression and cannot be certified.Every remaining outcome reproduces its closed form.
- Question construction: Questions are restricted to quantities with dimensions L T^−k, following the rationale in Sec. 3.1.Excluded outcomes remain available for intervention records.
- Question construction: Degenerate items are filtered using thresholds on speed, length, displacement, and near-identical reference-target quantities.
F. Training details
Training fits Qwen2.5-VL-3B-Instruct with parameter-efficient, quantized fine-tuning under a constrained laptop-scale configuration.
- Optimization: Training uses LoRA rank 16 with αLoRA = 32 and dropout 0.05 on all attention and MLP projections.
- Optimization: The model is trained in 4-bit NF4 with double quantisation using AdamW at 10^−4 for one epoch.Weight decay is 0.01, with gradient clipping at 1.0 and a onecycle schedule.
- Resource constraints: Six frames per video are sampled at a maximum of 100,352 pixels, producing sequences near 650 tokens and a peak memory use of 5.85 GB.The configuration fits an 8 GB laptop GPU.