PaperlayerView exact revision on arXiv

Source-linked AI summary

Axiomatic World Modeling for Physics Reasoning

Xinye Yang, Zhenyang Liu, Yuxuan Wang, Yuanyuan Lei

arXiv:2610.24519v1cs.AIcs.CL

TL;DR

Research-level physics reasoning remains difficult partly because LLM rewards do not directly supervise the physical world a model constructs. RLAWM rewards both world modeling and reasoning within that world, and the paper reports gains across benchmarks and zero-shot distribution shifts.

  • Problem

    Physics reasoning requires inferring relevant laws and assumptions, but common outcome- and step-level rewards rarely supervise whether the inferred physical world is correct.

    Source: §1 · pp. 1–2

  • Method

    RLAWM uses modeling and solving rewards to compare an externalized axiomatic world model with the stated physics and check reasoning execution for physical consistency.

    Source: §1 · p. 3, §3.3 · p. 5

  • Results

    RLAWM improves performance across backbones and three distribution-shift axes, with gains continuing as the SubPass@k sample count grows.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The reported gains extend zero-shot across difficulty, source, and domain without test-time checker support.

    Source: §5.3 · pp. 8–9

    RLAWM requires extra models during training, raising training cost, and its static checkers do not adapt to rollouts.

    Source: A Limitations · p. 15

Abstract

from arXiv · show

Recent research-level physics-reasoning benchmarks for LLMs reveal a failure mode that lies not in knowledge or execution, but in modeling the world being reasoned about. Models often silently alter given quantities, fabricate laws, or inject unstated assumptions to force a problem into familiar patterns, and then reason inside this corrupted world. We define this premise-level divergence as modeling drift: the gap between a model's internal axiomatic world model (AWM) and the true physical world specified in the problem. Although RLVR and PRM have yielded substantial gains in mathematics and code, their reward shapes do not directly supervise world modeling in axiomatic domains such as physics. We propose Reinforcement Learning with Axiomatic World Modeling (RLAWM), which optimizes the policy by penalizing divergence between its inferred axiomatic world model and the true physics world. RLAWM probes physics reasoning at two hierarchical levels (modeling and solving), each scored by a physically grounded reward. The modeling reward targets abstract knowledge formulation, evaluating whether the model captures the true axiomatic world before attempting calculation. The solving reward targets instantiated knowledge reasoning, checking whether the model's concrete derivation remains consistent with the axiomatic world. This two-level structure separates what world the model believes it is reasoning in from how it reasons within that world. The framework operates exclusively during training, with no extra cost during inference. RLAWM improves performance by up to +12.2 points on PhysReason and +5.2 on PHYSICS over the baseline. It also generalizes zero-shot across difficulty, source, and domain, with a 2.45x gain and +10 points over the strongest baseline on olympiad-level physics and math.

1 Introduction

Research-level physics benchmarks expose errors in how LLMs model the stated physical world, not merely in their algebra. RLAWM rewards both faithful world construction and reasoning within that world; the paper reports gains across benchmarks and distribution shifts.

1 Introduction

  • Modeling drift is premise-level divergence between a model’s inferred axiomatic world model and the physical world specified in the problem.The representative example shows an incorrect geometry assumption propagating into an unphysical result, followed by changing a stated field condition to restore consistency.Source: §1 · p. 2
  • Existing outcome-only and step-level rewards rarely supervise whether a physics solver has constructed the correct physical world.Outcome rewards assess final answers, while step-level signals often assess textual plausibility or local correctness, leaving the inferred world model unchecked.Source: §1 · p. 2
  • RLAWM separates modeling and solving rewards, scoring the inferred world against the stated physics and probing reasoning within that world.The approach builds on the premise that physical claims can often be checked more easily than derived, and turns interactive checks into training signals.Source: §1 · pp. 2–3
  • RLAWM improves results across backbones and three distribution-shift axes, while its SubPass@k performance continues improving as more samples are considered.The reported shifts cover difficulty, source, and domain; the paper describes the gains as expanding the reasoning boundary and transferring beyond training distributions.Source: §1 · p. 3

2 Related Work

Prior reward methods assess answers, textual steps, or derivation soundness against adopted axioms, but do not directly evaluate whether the physical scenario was modeled accurately. RLAWM addresses that gap with separate modeling and reasoning rewards.

RL Reward Signals Do Not Address Modeling Drift

  • Existing outcome, judge-based, and theorem-prover rewards do not directly assess whether the policy constructed an accurate model of the physical scenario.A theorem prover can verify a sound derivation against axioms the policy adopted, even when those axioms describe a drifted world.Source: §2 · p. 3
  • RLAWM decomposes physics solving into modeling and reasoning, rewarding faithful representation of the setup and reasoning within the constructed model.Figure 2 depicts the modeling reward scoring a five-field world representation and the solving reward testing pseudocode with a frozen checker.Source: §2 · p. 3, Figure 2 · p. 4
  • Text-based step rewards can miss premise errors, while theorem-prover verification checks soundness only relative to the policy’s adopted axioms.Source: §2 · p. 3

3 Reinforcement Learning with Axiomatic World Modeling

RLAWM externalizes the policy’s implicit axiomatic world model and applies complementary rewards to its abstract construction and executed reasoning. The modeling reward compares the stated world with the inferred one, while the solving reward checks physical consistency in execution.

3.1 Physical Reasoning through a Belief-State Lens

  • RLAWM separates abstract world construction from instantiated reasoning, with execution feedback also refining which world representation the policy constructs.Different structural commitments can be physically valid yet vary in usefulness for downstream deduction; a poor choice can make derivations harder and increase execution errors.Source: §3.1 · p. 4

3.2 Modeling Reward: What World Does the Policy Believe In?

  • The modeling reward compares an externalized five-field AWM with the problem’s ground-truth axioms, using motif-based F1 to score fidelity.The fields cover objects and constraints, representation, mapping, and plan; the motifs check quantity-law-context, variable coupling, and assumptions.Source: §3.2 · p. 5

3.3 Solving Reward: Did Reasoning Stay Within That World?

  • The solving reward checks executed reasoning for physical consistency, using tests of invariants, trace consistency, and responses to perturbations.A frozen checker runs programmatic tests on an executable instance; each test counts equally, and the reward reflects the fraction passed.Source: §3.3 · p. 5

3.4 Joint Training Objective

  • The joint training objective combines modeling and solving rewards with final-answer correctness.Source: §3.4 · p. 6

4 Experimental Setup

The experiments evaluate in-distribution performance and zero-shot transfer across difficulty, source, and domain. They compare three multimodal base policies with untrained, outcome-reward, and judge-based baselines under shared training and evaluation protocols.

4.1 Datasets

  • Five benchmarks cover matched in-distribution tests on PhysReason and PHYSICS-text_only, plus zero-shot shifts in difficulty, source, and domain.The OOD benchmarks are OlympiadBench-Phys, PHYBench, and AIME 2026; policies train only on PhysReason for these evaluations.Source: §4.1 · p. 6

4.2 Models and Baselines

  • The study uses three multimodal base policies across two model families and three parameter scales, and compares them with five baselines.Baselines span no training, final-answer correctness, and judge-based reward signals.Source: §4.2 · pp. 6–7

4.3 Training and Evaluation Protocol

  • Except for the AIME cross-domain experiment, methods share an SFT-then-GRPO protocol and hyperparameters, differing in reward source and RLAWM’s SFT restructuring.For AIME, all methods instead apply GRPO directly to the base policy without an SFT cold start; RLAWM’s checker components operate only during training.Source: §4.3 · p. 7
  • Evaluation scores the fraction of sub-questions judged correct, then reports SubPass@1 under greedy decoding and SubPass@k as the best-of-k score.SubPass@k gives partial credit and reduces to Pass@k when a problem has one sub-question.Source: §4.3 · p. 7

5 Main Results

RLAWM outperforms the strongest baselines across the evaluated backbones and datasets, with gains across difficulty levels and subdomains. Its advantage also appears in SubPass@k and zero-shot tests across difficulty, source, and domain.

5.1 SubPass@1 Across Baselines, Difficulties, and Subdomains

  • On Easy and Medium problems, RLAWM's margin is 11–20 points and narrows on Difficult problems.The authors interpret this pattern as consistent with modeling drift being a primary bottleneck when reasoning ability is sufficient, while the lower ceiling on Difficult problems leaves less room for improvement.Source: §5.1 · p. 7
  • Performance gains vary across PHYSICS subdomains, which the authors attribute to how explicitly each field exposes physical structure to the checker.They say PMF more readily captures explicit coordinate frames in E&M and Optics than abstract operator algebra in Quantum and Statistical Mechanics.Source: §5.1 · p. 7

5.2 SubPass@k: Expanding the Reasoning Boundary

  • On PhysReason SubPass@k, RLVR and Reference-Likert fall below the base model as k grows, while RLAWM improves monotonically.The authors describe the former pattern as diversity collapse and the latter as an expansion of the reasoning boundary.Source: §5.2 · p. 7, Figure 3 · p. 8

5.1 SubPass@1 Across Baselines, Difficulties, and Subdomains

  • RLAWM outperforms the strongest baselines across all backbones and datasets, gaining up to +12.2 points on PhysReason and improving consistently on PHYSICS-text_only.Table 1 reports SubPass@1 (%); the caption specifies that best-per-column results are selected within each backbone.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 SubPass@k: Expanding the Reasoning Boundary

  • RLAWM's SubPass@k scaling is attributed to its rewards anchoring abstract axioms and filtering for physical consistency without prescribing a unique reasoning trace.The authors say this lets the policy explore diverse representation choices and calculation paths.Source: §5.2 · p. 8

5.3 Out-of-Distribution Generalization

  • RLAWM generalizes zero-shot from PhysReason across difficulty, source, and domain, including OlympiadBench-Phys, PHYBench, and AIME 2026.Reported comparisons include 15.2% versus RLVR's 6.2% on OlympiadBench-Phys, 10.5% versus the strongest baseline's 8.7% on PHYBench, and a 10.0-point lead over RLVR on AIME 2026.Source: §5.3 · p. 8, Table 3 · p. 9
  • For the cross-domain AIME gain, the authors identify reduced modeling drift and the shared axiomatic core of physics and mathematics as the basis.They state that both learned behaviors transfer without test-time support.Source: §5.3 · p. 9

6 Ablations and Additional Analyses

The ablations show that modeling and solving rewards address distinct failure modes, while the solving reward adds graded training signal and the modeling reward remains comparatively robust to a weaker scorer.

6.1 Necessity of Each Reward Component

  • RS+RA leads on difficult problems, RM+RA leads where choosing the governing law is the bottleneck, and the joint reward performs best when both failure modes co-occur.On regime-extreme cells, the joint reward stays close to the dominant configuration.Source: §6.1 · p. 9

6.2 What Training Signal Does the Solving Reward Add?

  • 29.9% of GRPO groups have zero reward variance under outcome reward alone, falling to 9.3% with the checker; 64.1% of all-wrong rollouts still receive nonzero RS.The checker averages 7.34 assertions per rollout; 4.9% of correct, successfully executed rollouts are fully rejected, and false positives reach 15.8%.Source: §6.2 · p. 9

6.3 Sensitivity to the Modeling-Reward Scorer

  • PMF outperforms Judge by +0.8 points on both backbones with GPT-4o-mini and by up to +2.8 with Qwen3-VL-4B, while Judge’s 4B-policy performance drop is roughly twice PMF’s.PMF uses schema-bounded motif extraction and deterministic F1 scoring; Judge requires end-to-end physics reasoning by the scorer.Source: §6.3 · p. 10

6.4 Disentangling Reward Signal from SFT Restructuring

  • Zero-RL RLAWM reaches 37.3 Overall on PhysReason, beating Reference-Likert (31.4) by +5.9 points, while structured SFT improves Reference-Likert mainly on Easy and Knowledge.The structured SFT addition changes Easy and Knowledge by +2.1 and +2.6, while Medium and Difficult scores barely change.Source: §6.4 · p. 10

7 Conclusion

RLAWM uses hierarchical modeling and solving rewards to mitigate modeling drift in physics reasoning. Its gains span backbones, seeds, and three distribution-shift axes, and the authors point to formal mathematics as a possible extension where verification is cheap.

7 Conclusion

  • RLAWM mitigates modeling drift in physics reasoning with two hierarchical rewards for modeling and solving.Together, the rewards turn the verification-derivation asymmetry of axiomatic domains into a trainable signal.Source: §7 · p. 10
  • RLAWM’s gains hold across backbones, seeds, and three axes of distribution shift.The authors identify formal mathematics with cheap verification as a possible extension, citing the cross-domain AIME 2026 gain.Source: §7 · p. 10
  • The authors identify replacing static checkers with agentic probes and developing modeling rewards without external scorers as future work.Source: §7 · p. 10

A Limitations

RLAWM requires additional training-time models and uses static checkers. The paper defines AWM as the policy’s implicit belief about a problem’s physical structure, which is externalized for reward scoring rather than represented by a separate predictive network.

A Limitations

  • RLAWM requires extra training-time models, and its static checkers do not adapt to rollouts.A frozen LLM scores the modeling block, a frozen translator compiles pseudocode, and another LLM synthesizes checkers from a fixed theorem library; these components raise training cost.Source: A Limitations · p. 15

B.1 AWM is the Policy’s Internal Belief, not a Separate World Model

  • The policy itself carries the AWM; the structured modeling object is an externalization of that implicit belief, not a separate trainable world model.The externalization lets reward reach the belief, and RLAWM adds no parameter budget.Source: §B.1 · p. 16

B.2 What “Axiomatic” Means

  • In this paper, an AWM represents the problem’s axiom set, including laws, symmetries, boundary conditions, regime assumptions, and valid approximations.Modeling drift means placing a problem in the wrong axiom set, not making a prediction error about future states.Source: §B.2 · p. 16

D Dataset Details

The appendices provide dataset and asset details, then illustrate how different representations and reward levels distinguish valid modeling from correct execution. The case study shows that correctly identifying physical premises does not ensure they are applied consistently during solving.

D Dataset Details

  • The appendix consolidates dataset-preparation details summarized in §4.1, and Table 8 reports per-split sizes and composition for the five datasets used in the paper.Source: D Dataset Details · p. 17

Dataset case: PHYSICS-text_only

  • The appendix gives a Thomas–Fermi atom problem as a representative PHYSICS-text_only example, asking for relations among Fermi momentum, charge density, and electrostatic potential.Source: Dataset case: PHYSICS-text_only · p. 17

Dataset case: PhysReason

  • The representative PhysReason problem involves charged sliders, a spring, friction, an electric field, and staged motion questions.Source: Dataset case: PhysReason · p. 18

E Model Assets and Licenses

  • The appendix lists pretrained models, their pipeline roles, and licenses, with open-weight assets and the frozen translator obtained from public Hugging Face releases and GPT-4o-mini accessed through the API.Source: E Model Assets and Licenses · p. 18

F Case Study: Why We Need Rewards at Different Hierarchical Levels

  • The case study compares two rollouts from the same training problem and policy: both identify relevant premises, but one omits nonzero work during solving, which a work–energy invariant check can detect.Source: F Case Study: Why We Need Rewards at… · p. 18

Case 2: interaction_graph construal →low reward

  • In the interaction-graph rollout, force cancellation is incorrectly used to infer unchanged kinetic energy and v_O = √gR; the two rollouts share modeling commitments but differ in their work accounting.The appendix explains that RM evaluates the declared setup, while RS checks selected physical relations in its executable instantiation.Source: Case 2: interaction_graph construal… · pp. 20–21

G Hyperparameters

H Robustness to Random Seeds

Across three Qwen3-VL-8B-Instruct seeds, RLAWM’s PhysReason SubPass@1 Overall scores show less than one point of seed-to-seed spread, and even the worst run beats Direct-Likert.

H Robustness to Random Seeds

  • 48.1 ± 0.9 SubPass@1 Overall across three seeds leaves the worst run 10.2 points above Direct-Likert on PhysReason.The runs retrain RLAWM end to end with other settings held identical to the main experiments.Source: H Robustness to Random Seeds · p. 22, Table 12 · p. 22

I Modeling Schema and SFT Data Synthesis

The SFT phase trains the policy to emit structured modeling and solving blocks, synthesized from two GPT-4o-mini calls per problem conditioned on the reference solution and target answers.

I Modeling Schema and SFT Data Synthesis

  • The SFT corpus is synthesized with two GPT-4o-mini calls per problem, both conditioned on the ground-truth solution and target final answers.A post-processing script parses their JSON outputs and assembles the modeling and solving blocks used as SFT targets.Source: I Modeling Schema and SFT Data Synthesis · p. 22
  • The modeling JSON has five fields—objects, constraints, representation, mapping, and plan—and the modeling reward scores all five.Mapping and plan also bridge modeling and solving.Source: I Modeling Schema and SFT Data Synthesis · p. 22
  • The solving JSON organizes givens, shared variables, global equations, and subproblems into a required structural skeleton.Each subproblem includes an index, emitted quantities, a final answer, and at least one non-empty mapping, equation, or step field.Source: I Modeling Schema and SFT Data Synthesis · pp. 22–24
  • The modeling instructions ask for explicit objects and constraints, one selected representation, equations mapped from constraints, and a short plan for each sub-question.The plan identifies the target quantity and main law, and flags dependencies on earlier intermediate results.Source: I Modeling Schema and SFT Data Synthesis · pp. 22–23
  • The solving instructions require reuse of the modeling model and treat target answers as alignment hints rather than substitutes for self-consistent reasoning.The output is intended to expose concrete computed or threshold quantities that support verification.Source: I Modeling Schema and SFT Data Synthesis · pp. 23–24

J PMF Motif Taxonomy and Error-Mode Diagnosis

PMF evaluates modeling fidelity through three structured motif types—law-and-quantity choice, equation coupling, and assumption validity—and reports their scores separately for diagnosis.

J PMF Motif Taxonomy and Error-Mode Diagnosis

  • PMF extracts three motif sets from policy and reference models and computes an F1 score for each type, making its score diagnostic rather than merely scalar.The three components are QEL, VCL, and AEB.Source: J PMF Motif Taxonomy and Error-Mode… · p. 24
  • QEL checks whether the selected physical principle and equation contain the required quantity in the correct physical context.It can identify wrong-law choices, omitted quantities, or a law assigned to the wrong stage.Source: J PMF Motif Taxonomy and Error-Mode… · p. 24
  • VCL checks whether equations communicate through the right shared variable, exposing broken chains and inconsistent intermediate quantities.It targets cases where locally correct equations fail to work together globally.Source: J PMF Motif Taxonomy and Error-Mode… · p. 24
  • AEB checks whether an equation’s required simplifying or boundary assumptions are present, catching formulas applied outside their valid regime.Examples include frictionless motion, constant volume, and small-angle approximation.Source: J PMF Motif Taxonomy and Error-Mode… · p. 24

K PMF Extraction Prompt Templates

  • PMF uses the same schema-bounded extraction prompt for policy and reference outputs, then computes motif-level F1 over the extracted structures.The prompt requests only explicitly stated or clearly implied motifs and leaves a motif set empty when no instances are present.Source: K PMF Extraction Prompt Templates · p. 25

L LLM-Judge Prompt Templates

  • The LLM-as-Judge baseline scores objects, constraints, representation, and mapping separately, then returns one integer score on a 1–5 scale with a short justification.Its rubric rates work with a clear hole as partial (3), while a conceptual error is weak (2).Source: L LLM-Judge Prompt Templates · p. 26

M Checker Implementation

The checker implementation binds problem-specific audits to a reusable theorem library and emphasizes local physical relations over final-answer matching.

M Checker Implementation

  • The checker uses reusable theorem templates, extracted and normalized from PhysReason reference solutions, with 147 templates covering laws and applicability conditions rather than problem-specific answers.The synthesis pipeline instantiates templates for each problem and validates assertions in a sandbox.Source: M Checker Implementation · p. 26
  • For each training problem, PhysicsChecker binds a fixed variable contract to retrieved theorems and runs independent interface, invariant, trace-consistency, and perturbation tests.Missing values are represented as None; final-answer matching is only a weak sanity check, while the primary reward comes from local physics relations re-derived from givens.Source: M Checker Implementation · p. 26
  • $5.35 in API calls synthesizes checkers for all 1,080 PhysReason training problems, about half a cent per problem.The pipeline extracts entities and attributes, binds them to library theorems, emits assertions, and validates each class in a sandbox.Source: M Checker Implementation · p. 27
  • The checker prioritizes theorem-based physical sanity and modeling-quality checks, while limiting answer-sanity tests to weak conditions such as sign or broad magnitude.The instructions reject hardcoding long lists of solved constants into a single consistency test.Source: M Checker Implementation · pp. 27–28

N Checker Reliability Diagnostics

Checker diagnostics show that graded rewards reduce outcome-reward sparsity, while disagreements with answer correctness arise from identifiable coverage and interface issues.

N Checker Reliability Diagnostics

  • 29.9% of groups have zero-variance outcome rewards versus 9.3% with the checker, which recovers 69.0% and gives 64.1% of all-wrong rollouts nonzero RS.The checker runs 7.34 assertions per rollout on average, allowing hard problems whose rollouts all fail to re-enter training.Source: N Checker Reliability Diagnostics · p. 30
  • 19.4% of rollouts receive RS=0, mainly from sandbox errors and variable-contract mismatches; translation failure and missing solving code together account for under 1%.Contract mismatch means pseudocode compiles but freely named variables cannot bind to checker fields.Source: N Checker Reliability Diagnostics · p. 30
  • False positives (15.8%) exceed false negatives (4.9%); uncovered answer errors motivate retaining the final-answer reward RA.Among correct answers with successful execution, 89.7% receive full credit, 5.4% partial credit, and 4.9% are rejected.Source: N Checker Reliability Diagnostics · pp. 30–31

Failure case 1: false negative (all sub-answers correct, RS = 0.56)

  • A consistent work-on-gas convention earns RS=0.56 because the checker binds opposite-sign fields; four of nine tests fail.The rollout answers all three sub-questions correctly, and the penalty remains bounded because the tests are independent.Source: Failure case 1: false negative (all… · pp. 31–32

Failure case 2: false positive (RS = 1.0, one sub-answer wrong)

  • RS=1.0 can coexist with a wrong bandwidth because the checker audits emitted quantities, not final sub-answer algebra.The checker passes the emitted bandwidth difference, while the written closed form mistakenly gives R/(2L) instead of R/L.Source: Failure case 2: false positive (RS =… · p. 32

O Baseline Implementation Details

  • Judge baselines use the same GRPO hyperparameters and holistic five-point rubrics, with Direct-Likert and Reference-Likert variants.Reference-Likert additionally receives reference answers and, when available, reference reasoning; both variants allow valid derivations that differ from the reference.Source: O Baseline Implementation Details · p. 32
Loading 2610.24519v1…