Source-linked AI summary
New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models
Sourajit Saha, Shubhashis Roy Dipta, Nobin Sarwar, Shaswati Saha, Yuxuan Jiang, Siyuan Li, Qiheng Wang
TL;DR
Vision language models must decide whether a physical question is answerable from one measurement and, if not, which additional experiment to perform. The paper introduces paired finite-world evaluations with exact cost-aware reference actions and finds that models usually repeat actions despite changed evidence or questions, with failures also appearing in measurement interpretation and physical reasoning.
Problem
Existing physical reasoning benchmarks usually evaluate final answers without directly testing whether models choose evidence adaptively.
Method
The paper evaluates six open models on paired physical problems with four possible worlds, requiring STOP or the cheapest additional experiment that resolves the question.
Results
Across changed-threshold image pairs, direct responses repeat the same action 95.5% to 97.9% of the time, while at most 0.3% have both actions correct.
Takeaways & Limitations
Evaluating evidence selection separately from final answers exposes physical reasoning limitations that conventional answer accuracy can overlook.
Takeaways & Limitations
The study covers specific open checkpoints, three idealized systems, discrete properties, and at most one additional experiment, not natural video, robots, or open-ended experiment design.
Abstract
from arXiv · showhide
A model first sees an image from one physical measurement experiment, such as how far a block coasted, and must answer a question about a new trial, such as whether the block will pass a target after a fixed push. The initial experiment may provide enough information to answer, or the model may need another measurement, such as the object's mass, friction, restitution, or spring stiffness. We study whether vision language models can decide when to answer immediately and, when more evidence is needed, which experiment to perform. Current physical reasoning benchmarks usually evaluate only the final answer, so they do not directly measure this decision-making ability. We introduce a controlled evaluation where each problem provides one measurement image and four possible physical worlds created by combining two possible masses and two possible values of another relevant property. The model must either stop and answer or select the cheapest additional experiment that can resolve the question. We construct matched problem pairs where changing either the observed measurement or the question changes the optimal action. Since all possible worlds and experiment costs are known, we can explicitly determine the optimal choice. Across six open models and 144 physical parameter sets, direct responses repeat the same action for 95.1% to 100% of image pairs even when the correct action changes. Brief reasoning improves action switching, but the best model makes both decisions correctly for only 5.9% of image pairs. Additional analysis reveals failures in measurement interpretation, physical reasoning, and response formatting. By evaluating evidence selection separately from final answers, our benchmark reveals limitations in physical reasoning that conventional answer accuracy can overlook.
1 Introduction
The paper asks whether vision language models can adapt physical experiment choices to the evidence and question, rather than relying on fixed decision rules. It introduces paired problems that expose this ability and finds varied failures across selection, interpretation, and answering.
- 1 Introduction: Average answer scores can conceal fixed decision rules that ignore whether the available evidence supports answering.Always stopping can earn half the available credit when half of the questions are already answerable.
- 1 Introduction: Paired physical problems reveal whether models change actions when measurements or requested thresholds change.One comparison changes the observed measurement while another changes the question, with the other prompt elements held constant.
- 1 Introduction: Failures span unnecessary measurements, premature stopping, incorrect answers after useful observations, reading, and calculation.These findings indicate that experiment selection alone does not account for the observed errors.
- 1 Introduction: Six open models rarely handle both sides of the constructed comparisons correctly.The evaluation determines when another experiment is needed and which test has minimum cost.
2 Related Work
The paper builds on work in physical prediction, active experimentation, selective prediction, and structured model behavior. It shifts attention from inferring outcomes from observations to choosing which observation should come next.
- 2 Related Work: Prior physical reasoning benchmarks primarily study conclusions drawn from observations, whereas this work examines which observation should come next.The related tasks include future-event prediction, hidden-property inference, and simulator-based parameter estimation.
- 2 Related Work: Active perception and experimental-design research treats sensing as an action and often values observations by their information or downstream objective.Related work also considers cost-aware planning and policies that choose questions or executable actions.
- 2 Related Work: Selective prediction studies when to abstain, but this task additionally requires choosing among missing measurements.Related benchmarks pair sufficient and insufficient evidence or corrupt modalities to test calibrated abstention.
3 Method
The method gives a model one physical measurement and asks it to stop or select one further experiment using a finite set of candidate worlds. Reference actions choose the cheapest measurement that makes the answer certain across all remaining worlds.
- 3 Method: The model receives an image and physical question, then chooses STOP or one further experiment.The construction first defines candidate worlds, creates paired inputs, and computes exact reference decisions.
- 3 Method: Each problem defines four possible worlds by combining two mass values with two values of friction, restitution, or spring stiffness.The evaluation covers sliding distance, post-bounce energy, and oscillation period.
- 3 Method: Matched histories and thresholds switch whether the answer is known or unresolved while preserving the relevant comparison conditions.One comparison exchanges the image at a fixed question; the other exchanges the threshold at a fixed image.
- 3 Method: The reference stops when all compatible worlds agree and otherwise selects the cheapest action that resolves every remaining case.A test costs one unit, a combined report costs two, and the missing property is the unique cheapest useful measurement for an unresolved base question.
- 3 Method: Evaluation separates minimum-cost action choice, pairwise consistency, evidence sufficiency, and final-answer correctness.Resolved-and-correct scoring excludes lucky guesses based on insufficient observations.
4 Experiments
The experiments evaluate six unchanged open vision language models across 144 physical families using direct and brief-reasoning protocols. The study is exploratory because planned capability controls failed, and its confidence intervals are pointwise rather than jointly covering all comparisons.
- 4 Experiments: Six open models are tested across 144 independent physical families spanning three systems and paired histories and thresholds.The models include Qwen2.5 VL, SmolVLM2, Idefics3, and Pixtral at the listed parameter sizes.
- 4 Experiments: The protocols compare short direct option outputs with brief reasoning responses under different token limits and response formats.Both use greedy decoding and randomized option codes aligned within pairs.
- 4 Experiments: The study is exploratory because planned capability requirements failed, and its 95% intervals provide pointwise rather than joint coverage.Intervals are based on 2,000 seeded bootstrap resamples of complete families.
5 Results
Across paired physical-selection problems, models often repeat actions despite changed evidence or thresholds, and additional reasoning only partly improves switching. Failures also arise after evidence selection, including measurement interpretation, answer use, visual reading, calculation, and response validity.
- 5.1 Different Observations Often Receive the Same Action: 95.5% to 97.9% of direct-response image pairs repeat the same action when only the threshold changes, while Qwen 32B has 49.5% direct decision accuracy but almost no complete pairs correct.At most 0.3% of image pairs have both actions correct; Pixtral stops on 97.6% of decisions and matches no complete pair.
- 5.1 Different Observations Often Receive the Same Action: Brief reasoning lowers action repetition to 70.8%–93.8%, but Qwen 7B reaches only 5.9% correct decisions on both members of an image pair.For Qwen 32B and Pixtral, greater action variation accompanies worse decision accuracy, and some changes reflect invalid responses or parser errors.
- 5.2 Obtaining Evidence and Using It Can Fail Separately: Useful selected evidence does not ensure correct answers: supplied-test accuracy reaches only 34.6% at best, while several models answer UNDETERMINED after informative tests.Examples include correctly selecting stiffness but withholding a definite answer despite sufficient information.
- 5.2 Obtaining Evidence and Using It Can Fail Separately: 83.5% of Qwen 3B’s 278 single-property purchases on unresolved cases repeat the earlier test, with SmolVLM2 and Idefics3 showing similar preferences.Qwen 32B makes this particular repetition error rarely but stops prematurely on 75.7% of unresolved questions.
- 5.2 Obtaining Evidence and Using It Can Fail Separately: No model reaches the planned 75% accuracy requirement when both physical properties are supplied, showing that selection results remain entangled with reading and calculation.Visual measurement recovery and numerical mechanics therefore contribute independently to the observed failures.
- 5.3 The Capability Controls Remain Weak: Prompt changes provide no common remedy: text readings help four models by 9.0–14.2 points, equations help SmolVLM2 by 5.2 points, and Pixtral declines by 35.8 points.A minimum-cost reference reaches 31.7% final success at cost 0.50 for Qwen 3B, versus 18.8% at cost 0.99 for its own choices.
6 Conclusion
The paper introduces a paired evaluation of whether vision language models know when physical evidence is sufficient and which additional experiment to choose. Across six open models, aggregate scores conceal repeated actions despite changed evidence, while experiment selection and result interpretation remain separate challenges.
- The paired evaluation checks each model choice against an exact reference because controlled changes in observations or questions alter the correct action.
- Across six open models, models often repeat the same action even when new evidence should change the decision.The paired construction makes this behavior visible rather than allowing aggregate scores to conceal it.
- Selecting a useful experiment and correctly interpreting its result are separate challenges.The analysis identifies both decision adaptation and measurement use as requirements for progress.
7 Limitations
The study is bounded to controlled settings involving selected open checkpoints, idealized systems, discrete properties, and at most one additional experiment. Its analyses also limit causal interpretation and direct comparison with the deterministic reference.
- The findings cover specific open checkpoints, three idealized systems, discrete disclosed properties, and at most one additional experiment.Natural video, sensor noise, robots, and open-ended experiment design are not tested.
- Selection and answering use separate calls, so the answering model never sees the selection rationale.Several analyses followed output inspection, and some Qwen 32B brief-reasoning conditions are missing.
- Image ablation and extended generation alter multiple factors, limiting causal attribution.The reported systematic failures therefore do not identify a unique internal cause.
- The deterministic reference receives additional metadata and images, so its perfect score validates the procedure but is not directly comparable.
A.1 Physical Assumptions and Derivations
The benchmark derives controlled outcomes for sliding, bouncing, and spring systems under idealized assumptions, then constructs matched histories and thresholds that switch whether evidence suffices. Rendering and reference checks validate construction, but not learned models’ visual interpretation.
- Sliding: Sliding assumes Coulomb friction with constant µ, no drag or obstacles, and a known-speed coasting measurement paired with a known-impulse target trial.The target speed depends on mass through v0 = J/m, while coasting distance reflects friction and initial speed.
- Bouncing: Bouncing assumes vertical motion without rotation or drag and a constant restitution coefficient e shared by measurement and target trials.Impact speed is √2gh, rebound speed is e√2gh, and post-impact kinetic energy is mge2h.
- Spring motion: Spring motion uses m¨x = −kx with a massless spring, negligible friction and damping, and linear-regime motion.The stiffness measurement uses the same spring under a known static load.
- Matched construction: The generator varies two disclosed properties independently within fixed ranges and creates ordered outcome intervals whose thresholds switch histories between known and unknown answers.The construction uses four worlds per family and thresholds placed inside outcome intervals.
- Validation: Rendered measurements use marked ruler positions and deterministic checks, while the reference program obtains all 9,216 records correctly with information unavailable to models.These checks validate implementation and geometric encoding but cannot establish that learned models read the images correctly.
B.2 Output Generation and Parsing
The study contrasts short direct option responses with brief reasoning under randomized menus and parses generated outputs using fallbacks. Reported scores therefore depend partly on formatting, extraction, and validity decisions in addition to physical reasoning.
- Generation: Direct requests use a 16-token cap, whereas reasoning allows at most two short sentences followed by an exact ANSWER: <option letter> line.The reasoning protocol uses a 256-token limit, with a recorded maximum of 200 tokens for Qwen 32B.
- Generation: Randomized option codes remain aligned within matched pairs, and answer menus preserve YES, NO, and UNDETERMINED ordering across histories.
- Parsing: The parser prioritizes final option letters but falls back to standalone letters, answer words, and action names, so malformed reasoning outputs can be misdecoded.A reasoning answer counted as correct may not express the correct action.
- Parsing: Reasoning ruler scores combine measurement reading, output formatting, and extraction because the stopping rule includes the ANSWER: prefix.Direct ruler error is normalized by the true reading and differs from the renderer’s geometric check.
- Validity: Direct selection validity is 100% for all six models, while SmolVLM2’s reasoning selection validity is 78.0% and other models range from 99.8% to 100%.Invalid-action handling affects reasoning comparisons but not direct selection.
- Study execution: The executed direct study missed some planned accuracy and validity requirements, while added checkpoints, probes, parsing changes, and a shortened Qwen 32B run constrain interpretation.
- Uncertainty: Bootstrap intervals resample 144 families with 2,000 resamples and describe variation across parameter families under one run configuration.They do not measure variation across checkpoints, decoding seeds, or data sources.
C Complete Decision Results
The evaluation reports choice accuracy, pairwise correctness, and action stability across sliding, bouncing, and spring tasks. These measures show whether models select the required action consistently across matched problems.
- The benchmark separately reports individual choice accuracy, pair scores, and unchanged choice rates.Individual scores use 576 decisions, whereas pair scores use 288 pairs.
- Both choices in a matched pair are seldom correct across the three physical systems.Changing the image or threshold creates paired decisions whose joint correctness is measured directly.
- Most pairs retain the same decoded action even when the reference action changes.The reference unchanged-choice rate is zero, so repetition alone does not indicate correctness.
- Minimum-cost choice accuracy is reported separately for sliding, bouncing, and spring motion.The task targets rebound energy in bouncing and period thresholds in spring motion.
D Evidence Selection and Answering
The paper separates evidence sufficiency from answer correctness because a selected measurement can settle the question without producing a correct final answer. It also shows that repeated unknown responses and invariant outputs can obscure these distinctions.
- Evidence sufficiency and answer correctness are distinct quantities evaluated on different answer sets.Selection sufficiency uses 576 decisions, while answer accuracy uses 1,152 informative measurement answers; final success requires both.
- 4,537 to 4,608 of 4,608 direct answers per model repeat UNDETERMINED, producing near-zero accuracy when evidence is sufficient.The counts are reported for Qwen 7B, Qwen 32B, Idefics3, and Pixtral respectively.
- All six models give identical decoded responses on hidden-alternative pairs, and definite binary answers are exactly 50% correct on average.Pixtral never gives a definite answer in this group, so its conditional binary rate is undefined.
- Action agreement is not equivalent to accuracy because saying undetermined and buying a test are scored separately.The benchmark distinguishes known-answer and unknown-answer questions and compares action agreement across all decisions.
E Which Experiments Are Chosen?
The paper analyzes which measurements models choose, how cost changes affect policy scores, and how reasoning, presentation, and formatting variants alter decisions. These analyses distinguish unnecessary testing, premature stopping, and failures in downstream answers.
- Which experiments are chosen?: When answers are known, only stopping has minimum cost; when they are unknown, measuring the missing property costs less than buying both measurements.The benchmark therefore separates unnecessary measurement from the choice among informative tests.
- Which experiments are chosen?: Unneeded tests, useless tests, and early stopping are scored as different errors.Each condition contains 288 decisions, distinguishing testing a settled case from repeating known evidence or stopping too early.
- Which experiments are chosen?: Several models favor measuring the already known property again, while repetition leaves unresolved questions unanswered.The relevant rates have different denominators, and Pixtral’s direct repeated-test rate is based on only seven choices.
- Policy comparisons: The exact selector stops when possible and otherwise obtains the cheapest sufficient evidence, with mean cost 0.5 in the balanced base task.A uniform test has mean cost 1.25, while the model-selector analyses reuse saved answers after each measurement.
- Cost and response variants: Repricing the combined report changes credited choices without changing the recorded actions.For example, Qwen 7B’s direct choice score rises from 36.5% to 46.5% at price one.
- Controls and limitations: Neither response protocol reaches the planned calculation accuracy, and generation, parsing, and response validity affect the reported scores.Additional variants show mixed effects for text readings, renderings, wording, equations, and remaining-possibility prompts.