PhysAlign: A Benchmark for Evidence-Grounded Role Alignment in Multimodal Physics Reasoning

1 Introduction

• Physics setups depend on diagram relations and quantity-to-entity associations, so evaluation must assess recovered premises as well as final answers.Source: §1 · p. 1
• PhysAlign contains 3,341 localized probes from 986 physics problems, linking entity-reference and quantity-ownership questions to explicit source evidence and human-reviewed targets.Probes retain the original statement, diagram, and physical conditions.Source: §1 · pp. 2–3
• The benchmark scores content recognition and role grounding separately, while joint correctness requires both targets to be correct on the same probe.A 553-probe subset also compares original inputs with versions supplying correct local readings without revealing the queried role.Source: §1 · p. 2
• Grounding errors persist after correct local recognition, with the lowest conditional grounding error approximately 13.8% and paired results showing both repairs and harms.The highest observed +GT accuracy is 84.9%.Source: §1 · pp. 2–3
• PhysAlign identifies recognized content assigned to incorrect roles, including on probes where a frozen proximity rule fails; final-answer accuracy alone cannot localize these response-level errors.Source: §1 · p. 3

2 Related Work

• Physics and science benchmarks assess final solutions or broad reasoning stages, where perception, entity correspondence, and derivation remain coupled.Source: §2 · p. 3
• Controlled vision–language evaluations isolate compositional, attribute, relational, and ordering errors, while scientific diagnostics vary visual evidence or modality.Source: §2 · p. 3
• PhysAlign fixes the original problem and evidence anchors, separately measures content recognition and role grounding, and supplies local readings without revealing the grounding target.Source: §2 · p. 3

3.1 Task Definition and Scope

• Each probe anchors a local question to an image region or exact text span and asks for the associated role, with local content reported when a separate reading target exists.Probe types cover entity reference, image quantity transcription and ownership, and textual quantity ownership; probes without independent reading targets receive grounding scores only.Source: §3.1 · p. 3

3.2 Evidence-Linked Annotations and Probe Construction

• Reviewed annotations linked to original evidence supply probe targets, while construction includes human review, screening, and verification of exported probes.The annotation passes are answer-blind; reference solutions are aligned with evidence separately for analysis, and ambiguous or unsupported records are withheld pending resolution.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Dataset Scale and Coverage

• PhysAlign contains 3,341 unique probes from 986 parent problems across six sources; 553 probes from 385 parents also support paired +GT inputs.The source problems span seven named physics domains and a cross-disciplinary group, with mechanics and electromagnetism accounting for most parents under estimated assignments.Source: §3.3 · p. 4, Figure 3 · p. 5

4.1 Paired Evaluation with Ground-Truth Local Content

• Matched +GT inputs supply verified local content while withholding the queried role, and the paired comparison measures how grounding changes under that condition.The control does not provide a complete physical description or isolate the causal effect of correcting OCR errors.Source: §4.1 · p. 5

4.2 Deterministic Scoring and Balanced Aggregation

• Recognition, grounding, and joint correctness use deterministic target matching; joint correctness requires both recognition and grounding to be correct for the same local premise.Recognition and joint metrics use the 412-probe joint set, while overall grounding uses all 3,341 grounding probes.Source: §4.2 · p. 6
• Aggregation averages probes within parent–task-type groups before averaging parents and applying prespecified task-type weights, preventing annotation-rich parents from dominating.The C+G− weighted share is CAcc − JAcc, and JAcc is not the product of marginal accuracies.Source: §4.2 · p. 6

5.1 Experimental Setup and Main Results

• The evaluation uses six MLLMs, release-partition pools, and independent solving across all 986 parents, with local metrics weighted differently from solving.Local metrics use balanced weights, while solving weights parents equally.Source: §5.1 · p. 6
• Within Qwen3.5, scaling from 4B to 27B coincides with CAcc increasing from 53.97% to 84.14% and JAcc from 32.09% to 68.12%, though recognition does not eliminate the grounding gap.Source: §5.1 · p. 6

5.2 Recognition–Grounding Decomposition and Local-Reading Controls

• Across five models, 12.32%–28.26% of weighted events involve correctly read content assigned to the wrong role, while this full-set share is not a conditional error rate.Figure 4 decomposes recognition and grounding outcomes; the event share also depends on how often content is read correctly.Source: Figure 4 · p. 7, §5.2 · p. 7
• Supplying local readings yields positive Base-to-+GT grounding point estimates of 1.32–8.10 percentage points, but net gains do not imply uniform item-level benefits.Even under +GT, Qwen3.5-27B, GPT-6 Astra, and Gemini-3.8-flash retain grounding error rates of 18.00%, 15.10%, and 19.14% on their observed Pm.Source: §5.2 · p. 7

5.3 Validity: Visual Evidence, Geometry, and Interface Robustness

• Image removal reduces grounding accuracy for Qwen3.5-27B, GPT-6 Astra, and Gemini-3.8-flash by 20.39, 6.37, and 13.81 points, but improves it for two weaker models.On a fixed applicable subset, Qwen3.5-27B scores 86.35% against 76.82% for nearest-region; candidate reordering changes aggregate GAcc by at most 2.50 points for five models.Source: §5.3 · p. 8

5.4 Association with Original-Problem Solving

• All six models have higher grounding point estimates on correctly solved problems, but differences span only 0.28–3.94 points and neither task certifies the other.Qwen3.5-27B and GPT-6 Astra retain local grounding error rates of 12.11% and 14.47% on correctly solved problems.Source: §5.4 · p. 9, Figure 5 · p. 9

6 Conclusion

• Correct recognition does not guarantee grounding, and correct final answers can coexist with incorrect reported premises; PhysAlign therefore evaluates explicit reports under fixed anchors and candidates.The conclusion limits its claims to static diagrams and does not establish complete scene understanding, internal model states, or premises used in a particular solution.Source: §6 · p. 9

A.1 Source Problems and Provenance

• PhysAlign reuses original problems from six source datasets and preserves their images, statements, questions, answer options when present, reference answers or solutions, and provenance identifiers.The merge deduplicates identical public inputs and propagates splits through image hashes and recorded groups, but does not establish the absence of semantic near-duplicates or prior model exposure.Source: §A.1 · p. 12

A.2 Observational Annotation and Probe Compilation

• Four answer-blind passes record diagram primitives and text, link them to entities, annotate textual quantities and constraints, and consolidate supported observations into Gobs.A fifth pass aligns reference-solution steps for analysis only; it neither re-solves problems nor supplies probe targets or corrections to Gobs. Quantity ownership remains distinct from the entity on which a force acts.Source: §A.2 · p. 12

A.3 Quality Control and Admission Rules

• Construction review checks source annotations and exported probes, while blind-first human review requires independent agreement or third-reviewer adjudication and rechecking after corrections.GPT-6 Astra revised 1,089 further-review candidates, which then received item-by-item human verification; these supplementary candidates were not used in reported model evaluation.Source: §A.3 · pp. 12–13, §A.4 · p. 13

A.4 Candidate Screening and Release Retention

• The reported benchmark retains 3,341 probes from 986 parents, or 59.72% of 5,594 generated candidates; 184 parents retain all generated probes and 802 retain an approved subset.The accounting does not attribute the 98 parents without retained probes to a single exclusion cause.Source: §A.4 · p. 13

A.5 Post-construction Quality Audit

• 2,332 of 2,400 audited probes passed without revision (97.17%), while 68 (2.83%) required correction for transcription, localization, ownership, or consistency errors.These semantic errors required comparison with source evidence and could not be detected by file-existence, checksum, or schema checks alone.Source: §A.5 · p. 14
• After correction, 66 of the 68 flagged probes passed reinspection (97.06%); two remained unresolved and were excluded from the subsequent release candidate.The audited subset contained 2,398 accepted probes after correction and reinspection (99.92%).Source: §A.5 · p. 14
• The audit rates are descriptive, not sampling-weighted estimates for the full inventory; 941 probes were unaudited, and the audit does not establish effects on model results.The 97.17% rate is pre-correction, while 99.92% is post-correction acceptance within the audited subset.Source: §A.5 · p. 14

A.6 Composition, Problem Content, and Evaluation Coverage

• The retained release contains 3,341 probes from 986 parents, with 553 paired +GT variants from 385 parents; the joint reading–grounding set instead contains 412 probes from 311 parents.Paired-input and joint-set eligibility differ; the sets must not be conflated.Source: §A.6 · pp. 14–15
• Mechanics and electromagnetism account for approximately 75.6% of retained parents and 83.1% of parents with paired +GT variants, so neither the full inventory nor paired-input coverage is domain-balanced.Domain assignments are estimated, and the residual labels are classifications rather than additional physics domains.Source: §A.6 · p. 15
• Figure 6 complements coarse domain labels with recurring vocabulary families and prominent physics-related terms; this lexical view does not assign domains, construct probes, determine eligibility, or score the benchmark.The figure is descriptive of the retained source problems' text.Source: §A.6 · p. 15, Figure 6 · p. 16

B.1 Public Inputs, Evidence Anchors, and Candidates

• A public probe retains the original text and diagram and adds a local question, evidence anchor, neutral candidate aliases, and output requirements; reference mappings remain on the scoring side.T02-single resolves an anchored mention to an entity; T03-image reads an anchored quantity and identifies its owner; T03-text identifies the owner of a textual quantity.Source: §B.1 · pp. 17–18

B.2 Ground-Truth Local Content and Evaluation Pools

• The sets G, L, and P respectively cover eligible grounding tests, joint reading–grounding tests with independent reading targets, and probes with valid, permitted +GT content.Their eligibility is determined separately, and an item without valid +GT content may remain in G but cannot enter the paired comparison.Source: §B.2 · p. 18

B.3 Deterministic Scoring, Aggregation, and Uncertainty

• Local scoring normalizes only declared format equivalences and scores fields independently; aggregation weights parent problems equally within type before applying fixed type weights.A missing reading does not zero a correct owner field, and parent problems with many probes do not dominate the aggregate.Source: §B.3 · pp. 18–19

L + JAcc. (7)

• Conditional grounding describes performance only among correctly read items and should be reported with CAcc and parent support, not used alone to rank models.The four same-item outcome quadrants form a decomposition and do not assume recognition and grounding are independent; uncertainty intervals use parent-level paired cluster resampling.Source: §B.3 · p. 19, L + JAcc. (7) · p. 19

B.4 Original-Problem Solving Score

• Original-problem solving is scored independently across 986 retained parents, using binary answer matching for choice and fill-in tasks or expert-assigned partial credit for worked solutions.SolveAcc is mean normalized credit per parent, not the fraction of entirely correct solutions.Source: §B.4 · pp. 19–20

B.5 Observed Paired Coverage and Repeated Base Scores

• For the four open-weight models, the paired subset equals the joint set, so repeated Base grounding scores follow by construction rather than indicating independent-task agreement or added robustness.The equality holds because both summaries use the same probes, Base predictions, and aggregation weights.Source: §B.5 · p. 20
• For GPT-6 Astra and Gemini 3.8 Flash, paired analyses use 304 parents and 397 probes, versus 311 parents and 412 probes for the joint set; Base-score differences are 0.2740 and 0.6697 percentage points, respectively.These differences reflect support changes, not a change in the Base input condition; within each model, Base and +GT use identical probes and weights on its observed paired subset.Source: §B.5 · p. 20
• Paired results describe observed T03-image subsets, not all 553 paired-eligible probes; cross-model comparisons with unequal coverage are descriptive, and full-release paired performance cannot be inferred.Equal coverage counts for API models do not establish identical probe membership.Source: §B.5 · p. 20

C.1 Model Configurations and Execution

• Model-specific controls, image policies, output caps, aliases, and run dates are recorded; API rows use requested aliases, not observable provider-side revisions or exact weight snapshots.Output caps map to backend-specific fields, and differing accounting precludes an equal-compute interpretation.Source: §C.1 · p. 21
• Each Base/+GT pair uses a fresh conversation and holds the source problem, prompt, candidates, images, and settings fixed; only the local-reading field changes.Responses are not resampled, and retries apply only when infrastructure failures yield no usable response.Source: §C.1 · p. 22, §C.2 · p. 23
• The run manifest links requests to model or API identifiers, run time, settings, status, and raw response; supplementary rows summarize settings but use distinct prompts and inputs.The no-image control sends no image attachment, not a blank image.Source: §C.1 · p. 22

C.2 Prompt Templates

• Transcription requests require visible text without conversion or inference, while grounding queries return an anchored referent or the owner of supplied text.T03-image requests both the literal reading and owner; reading and ownership are scored separately.Source: §C.2 · p. 22
• Local probes share a system instruction and user-template structure, with task instruction and JSON schema varying across probe types.R locates evidence, E denotes a candidate entity, and V denotes a locator view; multiple views of one entity remain one candidate.Source: Figure 7 · p. 22, §C.2 · p. 22
• The +GT condition supplies local text without the grounding answer, while independent solving uses the original problem in a fresh context without probe metadata.The paired inputs otherwise retain the same question, candidates, images, output schema, and settings.Source: §C.2 · p. 23, Figure 8 · p. 23
• Choice and fill-in solutions receive binary boxed-answer matching, whereas short-answer and calculation solutions receive normalized expert stepwise credit.SolveAcc averages one normalized score per original problem across all 986 problems.Source: §C.2 · p. 23, Figure 8 · p. 23

D.1 Full Results and Error Decomposition

• Paired Base/+GT results use model-specific observed subsets, while the joint analysis uses L; the populations differ and their metrics should not be directly subtracted.Table 10 partitions L into four joint reading-grounding outcomes, with rows summing to 100% before rounding.Source: §D.1 · pp. 23–24, Table 11 · p. 25
• The model with the highest overall grounding accuracy differs from the joint-set leader: Gemini-3.8-flash reaches 90.71% GAcc on G, while GPT-6 Astra leads L with 89.44% CAcc, 77.13% JAcc, and 86.23% conditional grounding accuracy.GPT-6 Astra’s conditional grounding error is 13.77%; Qwen3.5-27B’s is 19.03%.Source: §D.1 · p. 23, Table 9 · p. 24
• Qwen3.5-9B reads more accurately than Qwen3.5-4B (65.81% versus 53.97%) but has higher conditional grounding error (42.94% versus 40.54%).InternVL3.5-8B-HF’s GAcc is 33.36%, while its CAcc and JAcc are 2.12% and 1.05%.Source: §D.1 · p. 23
• Reading-correct but grounding-wrong cases comprise 1.07%–28.26% of weighted mass on L, reaching 28.26% for Qwen3.5-9B and 21.88% for Qwen3.5-4B.InternVL3.5-8B-HF has 50.63% conditional grounding error within its small content-correct subset, while most of its mass reflects reading and grounding errors together.Source: §D.1 · p. 24, Table 10 · p. 24
• All four open-weight models show both repairs and harms after +GT; Qwen3.5-4B has a net gain of 8.10 points [3.88, 12.50], while the other models’ net-change intervals include zero.For Qwen3.5-4B, +GT repairs 13.35% and harms 5.25% of weighted probe mass; supplied readings do not reveal the correct owner or isolate a pure perception effect.Source: §D.1 · p. 25, Table 12 · p. 25

D.2 Validity and Robustness Checks

• Qwen3.5-27B exceeds the nearest-region rule by 9.53 points [6.10, 13.06] on matched support and reaches 73.22% on Hhard versus 18.76% random accuracy.The result shows performance beyond this particular proximity rule but does not identify the model’s internal strategy or exclude other shortcuts.Source: Table 14 · p. 26, Table 13 · p. 26, §D.2 · p. 27
• On the nearest-region failure subset, Qwen3.5-27B has a 29.71% reading-correct/grounding-wrong event share and 34.75% grounding error conditional on correct reading.These measures use the same intersection support but have different denominators; the subset is defined by reference-target rule failures.Source: §D.2 · pp. 26–27, Table 15 · p. 27
• Removing images lowers reference GAcc for the three models above 80% by 20.39 points for Qwen3.5-27B, 6.37 for GPT-6 Astra, and 13.81 for Gemini-3.8-flash.The control retains text, coordinates, and candidate geometry, so these changes measure sensitivity to image input rather than isolated visual-reasoning causality.Source: §D.2 · pp. 27–28, Table 16 · p. 28
• Candidate reordering changes GAcc by at most 2.50 points for five models, but InternVL3.5-8B-HF falls from 23.52% to 7.24%.Together with its improvement without images, this exception indicates pronounced sensitivity to input presentation.Source: §D.2 · pp. 27–28
• Across the controls, random selection and general candidate-order bias are insufficient explanations, but the experiments do not identify internal strategies or exclude shortcuts outside the tested controls.Spatial proximity remains a strong but incomplete baseline, and interface robustness should be measured rather than assumed.Source: §D.2 · pp. 25, 26, 28

D.3 Grounding and Original-Problem Solving

• Grounding differences between solve-correct and solve-wrong groups are descriptive: group membership varies by model and may reflect problem difficulty or other unobserved factors.Independent probing cannot establish which local evidence was used during solving or whether correcting a mismatch would alter the answer.Source: §D.3 · pp. 28–29, Table 17 · p. 28

Across cases

• The cases are diagnostic rather than estimates of error frequency, and separate probe and solution contexts establish behavioral non-equivalence rather than a causal account of internal reasoning.The controlled construction shows the result of substituting reported ownership into a fixed derivation, not that the model used that premise in its independent solution.Source: §D.4 · pp. 29, 31

Case 1

• All three Qwen3.5 models correctly read D but assign it to E1 rather than its correct owner, E2, and the mismatch persists under +GT.GPT-6 Astra and Gemini-3.8-flash recover both reading and owner from the same evidence.Source: §D.4 · p. 29, Figure 9 · p. 29

Case 2

• InternVL3.5-8B misreads the v–t graph’s vertical-axis label under Base and selects E1, then changes its owner prediction to E2 after receiving the correct reading.The paired outcome is consistent with a reading-linked error but does not establish causal mediation by recognition; its independent solving response is incorrect.Source: §D.4 · p. 29, Figure 10 · p. 30

Case 3

• A correct original-problem answer can coexist with persistent incorrect grounding, while correct local reading and grounding can coexist with an incorrect solution.The cases also include a +GT repair and a scene with correct recognition, grounding, and solving.Source: §D.4 · pp. 29, 31, Figure 11 · p. 30

Case 4

• In the controlled substitution, assigning wedge mass M to the descending cylinder yields Mgh instead of the reference mgh, although the valid relation Eloss = (descending mass)gh stays fixed.This construction isolates the consequence of the observed ownership response; it is not the model’s observed solution trace.Source: §D.4 · p. 31, Figure 12 · p. 31

• Unlike problem- and step-level physics benchmarks, PhysAlign probes local content–role correspondences without requiring complete solutions or using solution-derived facts as targets.Source: E Comparison with Related Benchmarks · p. 32
• Compared with diagram-structure, scene-graph, and image-text benchmarks, PhysAlign fixes anchors and candidates and scores literal recognition separately from physical-role correspondence.Source: E Comparison with Related Benchmarks · p. 32
• Its controlled variant supplies a verified local reading while holding the problem, anchor, candidates, and scoring fixed; it neither removes all perceptual uncertainty nor reveals internal reasoning.Source: E Comparison with Related Benchmarks · p. 32
• Table 18 compares diagnostic protocols, not benchmark quality; its criteria include fixed anchors, separate recognition and grounding scores, paired reading controls, and links to problem solving.Source: Table 18 · p. 32, E Comparison with Related Benchmarks · p. 32

• PhysAlign measures anchored local content-to-role mapping under candidate-constrained outputs, not holistic diagram understanding; +GT supplies transcription only, leaving entity boundaries, spatial relations, and physical semantics uncertain.Source: F Scope, Limitations, and Release… · p. 32
• Base/+GT measures grounding change after verified transcription, not an upper bound under perfect perception.Source: F Scope, Limitations, and Release… · p. 32
• Local-probe and original-solving results come from separate contexts, so agreement or disagreement cannot identify which local premise a model used or propagated in a specific solution.Source: F Scope, Limitations, and Release… · p. 33
• Evaluation covers static educational physics diagrams, skewed toward mechanics and electromagnetism; probes cover reviewed local facts, and paired +GT inputs cover only a subset.Source: F Scope, Limitations, and Release… · p. 33
• Public-source exposure cannot be excluded despite split propagation and exact-input deduplication; experiments combine development and test, so they are release-derived diagnostics, not held-out test estimates.Source: F Scope, Limitations, and Release… · p. 33
• Release plans provide probe inputs, metadata, prompts, code, and permitted run configurations; source problems and images are redistributed only where upstream licenses allow.Source: F Scope, Limitations, and Release… · p. 33