PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning

Shan Yang

arXiv:2605.14040v1cs.CL

TL;DR

Multimodal-physics benchmarks are rarely audited for contamination, translation effects, or MCQ saturation, limiting defensible comparisons. The paper audits the pipeline, releases audited corpora and a held-out olympiad evaluation, and provides Physics-R1 as a reference training recipe. Across tasks, the findings expose substantial measurement gradients, while Physics-R1 improves the audited held-out evaluation over its 8B base.

  • Problem

    Benchmark construction is rarely audited for train-eval contamination, translation drift, and MCQ saturation, despite their relevance to multimodal-physics measurement.

  • Method

    The paper performs an end-to-end audit and releases PHYSCORP-A, PHYSR1CORP, PHYSOLYM-A, and a GSPO+DAPO Physics-R1 recipe cold-started from Qwen3-VL-8B-Thinking.

  • Results

    Across tasks, three findings reveal 134 near-duplicates, a 17-pp translation delta, and a 46-point format-and-novelty gradient; Physics-R1 lifts PHYSOLYM-A liberal by +18.3 pp over the 8B base.

  • Takeaways & Limitations

    Defensible physics-VL measurement requires audited training data, original-language gold, and open-ended novel-source evaluation.

  • Takeaways & Limitations

    The cross-lingual finding is Sonnet-4.5-specific on n=59 paired items, and intermediate-length responses can fail to provide a final answer under the evaluation scorer.

Abstract

from arXiv · show

We audit the multimodal-physics evaluation pipeline end-to-end and document three undetected construction practices that distort how the field measures vision-language reasoning: train-eval contamination, translation drift, and MCQ saturation. (1) Public training pools (UGPhysics-Train, SciInstruct, MMK12) pass single-stage 5-gram-Jaccard audits with zero hits across all six public physics evals; a three-stage audit (Jaccard -> mxbai-embed-large cosine -> Haiku-4.5 LLM-judge) surfaces 134 near-duplicates and 4,846 paraphrase candidates in SciInstruct alone. (2) A 17-pp Sonnet 4.5 delta on 59 paired Estonian-English olympiad problems (30.5% vs. 13.6%; sign test p=0.011, McNemar p=0.021, paired bootstrap 95% CI [+5.1, +28.9] pp). (3) A 46-pp format-and-novelty gradient on identical Sonnet weights between MCQ (79.7% on PhyX) and open-ended olympiad evaluation (33.4% on PhysOlym-A). We release four artifacts addressing these gaps: PhysCorp-A (6,432-record three-stage-audited multimodal corpus), PhysR1Corp (2,268-record closed-form RL pool), PhysOlym-A (500-problem, 99.8% novel-source held-out olympiad eval with native difficulty labels and an EN/ET bilingual subset), and Physics-R1, a reference GSPO+DAPO recipe cold-started from Qwen3-VL-8B-Thinking. Across 3 seeds, Physics-R1 lifts the audited corpus over the 8B base by +18.3 pp on PhysOlym-A liberal (8.0 -> 26.3 +/- 1.7; 7.1 pp behind Sonnet 4.5), +15.7 pp on PhysReason (23.9 -> 39.6 +/- 6.4; ahead of Qwen3-VL-32B and Gemini 2.5 Pro), +6.9 pp on OlympiadBench-Physics (46.2 +/- 1.5), and +4.1 pp on PhyX MCQ (77.8 +/- 0.3).

1 Introduction

The paper audits multimodal-physics evaluation and identifies contamination, translation drift, and format saturation as construction practices that distort measurement. It releases audited data, a held-out olympiad benchmark, and a reference RL recipe to address these gaps.

  • Measurement findings: 134 near-duplicates emerged from 4,846 SciInstruct paraphrase candidates under a three-stage audit, despite zero hits from single-stage 5-gram-Jaccard checks.The stages are Jaccard, mxbai-embed-large cosine, and Haiku-4.5 LLM judging.
  • Measurement findings: 17 pp separated Sonnet 4.5 performance on 59 paired Estonian originals and English translations, scoring 30.5% versus 13.6%.The sign test was p=0.011, McNemar exact p=0.021, and the paired-bootstrap 95% CI was [+5.1, +28.9] pp.
  • Measurement findings: 46 points separated identical Sonnet 4.5 weights across 4-way MCQ and open-ended evaluations, from 79.7% on PhyX to 33.4% on the held-out audited eval.OlympiadBench-Physics scored 50.4% liberal in the same comparison.
  • Released artifacts: The paper instantiates three-stage auditing, original-language gold, and open-ended novel-source evaluation through four released artifacts.The artifacts are PHYSCORP-A, PHYSR1CORP, PHYSOLYM-A, and Physics-R1.
  • Released artifacts: The released comparison framework distinguishes audit status, train-test leakage, difficulty, cross-lingual pairing, usage, and RL readiness across benchmarks.After cleanup, PHYSCORP-A and PHYSR1CORP are clean against all six public physics evaluations.

2 Related Work

Related work spans rule-based reasoning recipes, canonical physics QA benchmarks, contamination audits, and multilingual evaluation. The paper combines these lines by adding a three-stage audit and paired original-language versus translation testing.

  • Rule-based RL for reasoning: Physics-R1 builds on DeepSeek-R1, GRPO, MM-Eureka, DAPO, and GSPO while retaining binary correctness rewards and VLM difficulty-curriculum structure.GSPO uses sequence-level importance weighting, while DAPO adds decoupled clipping and dynamic sampling.
  • Physics QA benchmarks: Canonical physics benchmarks include PhyX, OlympiadBench-Physics, UGPhysics, PhysReason, MMMU/MMMU-Pro, MMK12, PHYBench, and PhysUniBench.Prior benchmark coverage varies across format, contamination auditing, and evaluation use.
  • Contamination and multilingual evaluation: Existing benchmarks often cluster near the MCQ ceiling, while the paper introduces jointly audited, open-ended, and bilingual alternatives.The paper reports that only a subset of prior benchmarks publish contamination protocols, and none publish its three-stage pairwise audit.
  • Contamination and multilingual evaluation: The paper extends prior contamination-audit templates with embedding cosine because physics statements are especially sensitive to paraphrases involving units, vectors, and figure references.Its cross-lingual comparison instead pairs identical Estonian originals and English translations on the same closed model.

3 Data: The Audited Corpus and Held-Out Olympiad Eval

The data contribution combines audited training pools with a held-out olympiad evaluation designed around novel sources, native difficulty signals, and contamination control. The corpus and audit pipeline separate high-recall candidate discovery from precision filtering.

  • Released data: PHYSCORP-A contains 6,432 audited records, while PHYSR1CORP contains 2,268 closed-form RL records.The released artifacts include both training pools, PHYSOLYM-A, the Physics-R1 recipe, and the audit pipeline.
  • Training corpus composition: The corpus draws from nine source families, including five repackaged benchmark or textbook sources and four first-ML-format olympiad sources.The Estonian collection contributes organizer-issued difficulty labels and a bilingual EN+ET subset.
  • Training corpus composition: Approximately 73 of 2,268 PHYSR1CORP records have LLM-touched statements, while approximately 2,195 retain unmodified source statements.The touched subset includes synthetic MCQ augmentation and numeric-variation paraphrases, documented per distribution record.
  • Held-out olympiad evaluation: PHYSOLYM-A contains 500 problems, with 499 novel-source records under the four-corpus audit and native difficulty signals from Estonian organizers and Kevin Zhou.The benchmark has 0 Stage-3 near-duplicate overlaps with the audited training pool; one non-novel EuPhO 2020 record is disclosed.
  • Three-stage audit pipeline: The audit uses Stage 1 n-gram Jaccard, Stage 2 embedding cosine, and Stage 3 Haiku-4.5 judging, removing only Stage-3 close duplicates.The candidate set is the high-recall union of Stage-1 and Stage-2 matches.
  • Three-stage audit pipeline: The three-stage audit raises detected leakage from 3.3% with Stage 1 alone to 8.8% at the joint operating point on a 1,679-record researcher-curated sample.The detected rate ranges from 4.7% to 27.1% as the cosine threshold moves from 0.90 to 0.80.

4 Physics-R1: A Multi-Model RL Recipe

Physics-R1 applies an unmodified GSPO+DAPO recipe with binary or dense reward variants to train Qwen3-VL-8B-Thinking on closed-form physics rollouts. The paper presents binary reward as the deployable choice because it performs better on open-ended evaluation while dense reward offers only a small saturated-MCQ advantage.

  • Recipe: Physics-R1 cold-starts Qwen3-VL-8B-Thinking with GSPO+DAPO on a closed-form physics pool under a difficulty curriculum and binary correctness reward.The recipe samples rollouts, scores them, normalizes group advantages, and optimizes a clipped sequence-level objective.
  • Reward design: The dense reward adds format, dimensional, symbolic, and conservation components to binary answer matching.Binary answer matching accepts MCQ-letter equality, ±1% numeric tolerance, or symbolic equivalence.
  • Reward design: Dense reward gives only small saturated-MCQ gains, leading binary by at most 0.6 pp on the reported MCQ splits.The dense variant is tied with binary on PUB-OE within 0.7 pp.

5 Evaluation

The evaluation combines audited, open-ended, multilingual, and format-diverse tests to characterize Physics-R1 and the measurement properties of PHYSOLYM-A. Physics-R1 improves the 8B base most on the audited held-out olympiad split, while benchmark format, novelty, language, and judging protocol materially affect scores.

  • Evaluation protocol: The evaluation uses problem-level liberal Sonnet-as-judge scoring for open-ended benchmarks, with every sub-part required correct on multi-part problems.PHYSOLYM-A has a 13.9% unjudgeable rate, and intermediate-length Sonnet responses can fail by omitting a boxed final answer.
  • PHYSOLYM-A as a Measurement Instrument: 46 points separate identical Sonnet 4.5 weights on PhyX MCQ at 79.7%, OlympiadBench-Physics at 50.4%, and PHYSOLYM-A at 33.4%.The reported gradient reflects format, genre, and contamination-removal differences; a controlled 2×2 attribution study is left for follow-up.
  • PHYSOLYM-A as a Measurement Instrument: Native difficulty labels show Sonnet 4.5 strict accuracy declining from 62.5% at difficulty 1 to 0% at difficulties 3, 6, 8, and 10.The labels are organizer-issued for 131 Estonian problems, supporting PHYSOLYM-A as a non-saturating stopping signal.
  • PHYSOLYM-A as a Measurement Instrument: 17 percentage points separate Sonnet 4.5 strict accuracy on 59 paired Estonian originals at 30.5% from English translations at 13.6%.The paired tests report sign test p=0.011, McNemar exact p=0.021, and bootstrap 95% CI [+5.1, +28.9] pp.
  • Physics-R1 results: +18.3 pp is Physics-R1’s largest 3-seed-mean lift over the Qwen3-VL-8B-Thinking base, reaching 26.3 ± 1.7% on PHYSOLYM-A liberal.The same 3-seed mean reaches 39.6 ± 6.4% on PhysReason, 46.2 ± 1.5% on OlympiadBench-Physics, and 77.8 ± 0.3% on PhyX-mini.
  • Physics-R1 results: Physics-R1 mitigates missing final commitments, dimensional shortcuts, and first-panel-only visual attention, but not graduate-level perturbation-theory gaps.Hand inspection of 30 flips identified these recurring base-model failure modes and linked them to specific recipe levers.

6 Discussion and Limitations

The audit pipeline is presented as robust across embedder and judge choices, but the cross-lingual result is limited to Sonnet 4.5 on 59 paired items.

  • ρ=0.78 between mxbai and text-embedding-3-large rankings supports directional robustness of the audit findings.GPT-4o produced a more lenient cross-judge pattern, while Sonnet-as-judge had a disclosed 13.9% unjudgeable rate.
  • The cross-lingual finding is Sonnet-4.5-specific and uses n=59 paired items, with 65.7% MC power.All three preregistered tests reject the null, while the expected direction is documented to reverse for cross-lingual-weak models.

7 Conclusion

The paper identifies contamination, translation, and format-related weaknesses in multimodal-physics evaluation, then releases audited corpora and evaluation artifacts to address them.

  • 134 near-duplicates in SciInstruct surfaced only through the three-stage Jaccard, cosine, and Haiku-4.5 audit.The audit also identified 4,846 paraphrase candidates.
  • 17 pp separated Sonnet 4.5 performance on identical Estonian-English olympiad problems.The paired results were 30.5% versus 13.6%.
  • 46 pp separated MCQ and open-ended olympiad performance on identical Sonnet 4.5 weights.The format-and-novelty gradient was measured across PhyX and PhysOlym-A.
  • The released artifacts include PHYSCORP-A, PHYSR1CORP, and PHYSOLYM-A for audited training and held-out olympiad evaluation.The supplied passage lists a 6,432-record corpus, a 2,268-record RL pool, and a 500-problem evaluation.

A.1 Threshold-Sensitive Leakage Finding on a Researcher-Curated Baseline

This section evaluates threshold-sensitive leakage detection on a researcher-curated baseline and documents the audit corpus, annotations, reproducibility choices, and reward-design limitations.

  • Threshold-sensitive leakage finding: Stage-1 Jaccard leakage is bimodal, with researcher-curated leaks at J=1.0 and distinct records below J=0.3.The passage attributes this pattern to aggressive normalization and identifies cosine as the paraphrase detector in this corpus.
  • Artifacts and annotations: The released artifacts comprise PHYSCORP-A, PHYSR1CORP, and PHYSOLYM-A with a shared annotation schema.PHYSOLYM-A preserves source-native difficulty labels separately from aggregated difficulty.
  • Artifacts and annotations: Expected-solution-length labels have κ ∼0.6 and should be treated as a noisy proxy when stable labels are required.The ambiguity primarily concerns the medium-versus-long boundary.
  • Reward-design limitation: Dense reward shaping can reorder 14.3% of within-group rollout pairs, with 87% of those flips inside all-wrong subgroups.The section links these flips to formatting proxies that can be satisfied without solving the physics.

C.2 Dense five-component physics-native reward (ablation)

The dense five-component reward combines correctness with formatting, dimensional, symbolic, and consistency signals, but its ablation exposes a Goodhart channel relative to binary reward.

  • Reward construction: The dense reward sums answer, format, dimensional, symbolic, and consistency components, then clips the result to [−1, 1].The implementation exposes dense reward as an ablation alongside the recommended binary mode.
  • Goodhart behavior: Dense advantages reorder wrong rollouts by LaTeX surface form in a worked eight-rollout kinematics example.A well-formatted but incorrect rollout receives a less-negative dense advantage than some other wrong rollouts.
  • Goodhart behavior: Binary reward assigns equal positive advantages to correct rollouts and equal negative advantages to wrong rollouts after group normalization.Clipping removes dense variation among the correct rollouts in the example.
  • Goodhart behavior: A within-wrong spread of σ=0.42 absorbs gradient capacity into formatting-related rank differences under dense reward.Binary reward spends no capacity ranking rollouts within the same correctness class.
  • Ablation results: At matched step 60, binary exceeded dense reward by 8.9 pp on PhysReason, 4.9 pp on OlymBench-Phys, and 6.4 pp on PHYSOLYM-A.PUB-OE was effectively tied, with binary at 37.0 versus dense at 37.7.

D LLM-judge details: three judges, scoring conventions, and reproducibility

The paper uses three Sonnet-based judges matched to evaluation structure, with explicit equivalence rubrics, problem-level scoring conventions, and released audit materials for reproducibility.

  • Three judges: Three judges serve different evaluation structures: problem-level grading for PHYSOLYM-A and OlympiadBench-Physics, and per-subpart grading for PhysReason and PhysUniBench-OE.The paper states that multi-subpart problems require all subanswers to be correct, while problem-level evaluations use one verdict per problem.
  • Scoring conventions: Problem-level accuracy requires every subpart to pass for multi-subpart tasks, whereas PHYSOLYM-A and OlympiadBench-Physics receive one problem-level verdict.A softer partial-credit variant is 4–17 pp higher across multi-subpart rows but is not the headline metric.
  • Scoring rubric: YES/NO prompts compare candidate answers with gold answers while allowing equivalent forms, unit or format differences, trailing prose, and bounded rounding.Problem-level judging uses a 2% relative tolerance; the per-subpart rubric allows 1–2% tolerance depending on the judge.
  • Diagnostics: The PHYSOLYM-A evaluation partitions 500 problems into five source-stratified chunks and reports strict-correct counts using judgeable-only denominators.The reported chunk counts are 38, 20, 35, 30, and 40, with denominators 98, 100, 99, 98, and 96.
  • Reproducibility: Reproducibility materials include judge prompts, rubrics, per-chunk verdicts, human-graded samples, agreement matrices, and per-record verdicts.Two judge seeds yield preliminary κ ≥0.8 on PHYSOLYM-A; a 100-problem human subset is described as calibration rather than ground truth.
  • Bias checks: Self-grading checks report a 4.7-pp strict-versus-liberal Sonnet gap and 88% raw agreement with GPT-4o on a 50-problem subsample.The cross-vendor comparison has Cohen’s κ=0.44, and the reported asymmetry is not significant at n=6 discordant pairs.

E Per-source license and provenance log

The provenance log records source identity, acquisition details, licensing, redistribution terms, and attribution practices across the corpus’s source families.

  • Repackaged sources: UGPhysics contributes 5,520 EN/ZH undergraduate physics problems under CC BY-NC-SA 4.0.The source is Xu et al. (2025), and the same license is carried through in redistribution.
  • Repackaged sources: OpenStax contributes 2,381 end-of-chapter records under CC BY 4.0, while Physics Stack Exchange contributes 2,291 olympiad-style records under CC BY-SA 4.0.The records are harvested from College Physics, University Physics, and the Stack Exchange data dump.
  • First-ML-format sources: The Estonian Physics Olympiad collection contains 418 problems with native 1–10 difficulty labels and a 201-problem bilingual EN/ET subset.The collection covers archived rounds from 2004–2018 and is released under competition-policy educational-use terms.
  • First-ML-format sources: Kevin Zhou’s handouts contribute 692 problems with native point values and an advanced [A] flag under CC BY-NC 4.0.Written redistribution permission is documented in the supplementary archive.
  • International olympiads: The APhO, USAPhO, and INPhO scrape recovers 241 problems after splitter fixes, including 33 records newly recovered from INPhO.These sources are released as public-domain material with per-record attribution.
  • Licensing: Each released record carries its source license through the dataset, with academic-use restrictions for non-commercial sources and attribution for public-domain olympiad scrapes.UGPhysics and Kevin Zhou’s materials are non-commercial; olympiad sources are distributed under competition-policy public-domain terms.

F Reproducibility checklist

The paper provides pinned environments, deterministic seeds, audit scripts, hosted artifacts, and maintenance commitments to make data construction and experiments reproducible.

  • Seeds and selection: The headline Physics-R1 binary checkpoint uses seed 42, while the reported three-seed mean aggregates seeds 42, 17, and 23.Checkpoint selection uses a held-out PhyX-mini-MC early-stop signal, and data construction uses numpy.random.default_rng(42).
  • Versioning: The paper pins transformers 4.57.0, vLLM 0.11.0, verl 0.6.1, SymPy 1.13.3, sentence-transformers 5.4.1, and PyTorch 2.8.0+cu128.Re-evaluation with transformers 4.57.6 is reported to drift results by approximately 1.5 points.
  • Hosting: Code and datasets are hosted on GitHub, Hugging Face, and Zenodo, with stated hosting of at least five years.The release includes Croissant metadata augmented with Responsible AI fields and validator checks.
  • Audit reproducibility: The two-stage audit script accepts training and evaluation JSONL files with Jaccard and cosine thresholds, emitting per-record and aggregate reports.A 5,000-record audit against four held-out splits takes approximately 3 minutes on MPS or a CUDA GPU.
  • Release package: The released package includes four datasets, audit code, reward implementation, LLM-judge artifacts, license confirmations, and Croissant 1.0 plus RAI metadata.The supplementary index names the principal files and archives shipped with the release.
  • Maintenance: The recommended configuration uses binary correctness by default, while dense reward is reported as an ablation rather than a tuned standard.The paper identifies reward-component dropout, embedder sensitivity, and paraphrase- or translation-aware audits as follow-up work.

G Datasheet for Physics-R1

The datasheet defines the corpus’s motivation, composition, collection and audit procedures, intended uses, evaluation diagnostics, and explicit scope boundaries.

  • Motivation: The dataset was created for contamination-audited evaluation and post-training of multimodal vision-language models on visual physics reasoning.Its stated gaps concern three-stage contamination auditing, novel-source olympiad evaluation, and related measurement practices.
  • Composition: The release contains 6,432 audited PHYSCORP-A records, 2,268 PHYSR1CORP closed-form RL records, and 500 held-out PHYSOLYM-A problems.Records include problem statements, optional images, gold answers, optional solutions, and 14-field annotations.
  • Collection process: The corpus combines five repackaged benchmark releases with four first-ML-format olympiad scrapes collected from public archives.The author handled scraping, parsing, and auditing, while Sonnet 4.5 generated approximately 3,900 annotation labels.
  • Evaluation diagnostics: A taxonomy of 100 OlympiadBench-Physics errors finds wrong_subpart dominant at 30%, followed by missing_physics at 22% and different_question at 10%.The taxonomy is presented as a methodological diagnostic rather than a calibrated human grade.
  • Evaluation diagnostics: Per-category reporting is motivated by a +34.5pp strict accuracy gap between electromagnetism at 38.4% and astrophysics at 72.9% on identical Sonnet weights.The table caption identifies this comparison as the reason for reporting physics-category breakdowns.
  • Uses and scope: PHYSOLYM-A is intended for test-only use, and the artifacts are out of scope for general physics ability, human grading substitution, and experimental-physics evaluation.The stated use cases include visual physics reasoning, contamination auditing, cross-lingual studies, native-difficulty calibration, and VLM RL post-training.

H.7 Corpus composition, hyperparameters, and reward ablation (extended)

The Physics-R1 configuration combines GSPO+DAPO implementation choices with a dense physics-native reward and held-out early stopping, while related tables report difficulty and cross-lingual evaluation analyses.

  • Hyperparameters and configuration: FSDP1 is required for Qwen3-VL under verl 0.6.1 because FSDP2 fails on the multimodal projector path.The configuration uses vLLM 0.11.0 with TP=4 for rollouts.
  • Hyperparameters and configuration: The reference recipe combines recipe-marked GSPO+DAPO settings with a dense physics-native reward, an audited training pool, and held-out early stopping.These components jointly define the Physics-R1 reference recipe.
  • Evaluation analyses: Table 12 reports Sonnet 4.5 strict accuracy by organizer-issued native difficulty on 131 Estonian Physics Olympiad problems.Accuracy decreases nearly monotonically with difficulty and reaches 0% at difficulties 3, 6, 8, and 10.
  • Evaluation analyses: Table 13 evaluates cross-lingual Sonnet performance on 59 identical Estonian-English problems using strict and liberal scoring.Strict scoring requires numeric or symbolic matching, while liberal scoring uses an LLM-judge threshold of at least 0.5.
  • Evaluation analyses: The EN/ET agreement matrix shows a 4.3:1 asymmetry between ET-correct/EN-wrong and EN-correct/ET-wrong cases.The table states that this asymmetry rules out a noise explanation.
Loading 2605.14040v1…