Solving Physics Olympiad via Reinforcement Learning on Physics Simulators

Mihir Prabhudesai; Aryan Satpathy; Yangmin Li; Zheyang Qin; Nikash Bhardwaj; Amir Zadeh; Chuan Li; Katerina Fragkiadaki; Deepak Pathak

Source-linked AI summary

Solving Physics Olympiad via Reinforcement Learning on Physics Simulators

Mihir Prabhudesai, Aryan Satpathy, Yangmin Li, Zheyang Qin, Nikash Bhardwaj, Amir Zadeh, Chuan Li, Katerina Fragkiadaki, Deepak Pathak

arXiv:2604.11805v1Apr 13, 2026cs.LGcs.AIcs.CVcs.RO

TL;DR

Physics reasoning lacks scalable, diverse supervision compared with mathematics. Sim2Reason generates verified QA pairs from procedurally constructed physics simulations and trains LLMs with reinforcement learning, improving zero-shot performance on real-world physics benchmarks, including IPhO mechanics.

Problem
Physics lacks the large-scale, systematically varied QA datasets available for mathematics, limiting supervision for training models in physical reasoning.
Method
Sim2Reason procedurally generates diverse physics scenes and verified numeric, reverse, and symbolic QA pairs, then trains LLMs with reinforcement learning using verifiable rewards.
Results
5–10 percentage points: training solely on Sim2Reason data improves zero-shot IPhO mechanics performance across 3B–32B models, with similar gains on JEEBench and PHYSICS.
Takeaways & Limitations
Physics simulators can provide scalable reasoning supervision that transfers zero-shot to multiple real-world physics benchmarks.
Takeaways & Limitations
Simulator-generated supervision may encode modeling assumptions and failure modes that do not hold in the real world, with limited fidelity and coverage.

Abstract

from arXiv · show

We have witnessed remarkable advances in LLM reasoning capabilities with the advent of DeepSeek-R1. However, much of this progress has been fueled by the abundance of internet question-answer (QA) pairs, a major bottleneck going forward, since such data is limited in scale and concentrated mainly in domains like mathematics. In contrast, other sciences such as physics lack large-scale QA datasets to effectively train reasoning-capable models. In this work, we show that physics simulators can serve as a powerful alternative source of supervision for training LLMs for physical reasoning. We generate random scenes in physics engines, create synthetic question-answer pairs from simulated interactions, and train LLMs using reinforcement learning on this synthetic data. Our models exhibit zero-shot sim-to-real transfer to real-world physics benchmarks: for example, training solely on synthetic simulated data improves performance on IPhO (International Physics Olympiad) problems by 5-10 percentage points across model sizes. These results demonstrate that physics simulators can act as scalable data generators, enabling LLMs to acquire deep physical reasoning skills beyond the limitations of internet-scale QA data. Code available at: https://sim2reason.github.io/.

1. Introduction

Sim2Reason addresses the scarcity and uneven distribution of physics QA data by using physics simulators to generate verified synthetic supervision for RL training. Training solely on this data improves zero-shot performance across real-world physics benchmarks and supports scalable evaluation of physical reasoning.

Motivation: Physics lacks large-scale, systematically varied QA data: less than 1% of DeepSeek-R1’s 800K QA pairs involve STEM topics, contributing to poor physics-benchmark generalization.Internet QA data is sparse, unevenly distributed, and difficult to scale beyond a few million examples.
Motivation: Directly using simulators as external tools is difficult because LLMs must generate executable, physically accurate code through complex simulator-specific APIs.Early experiments with this paradigm were unsuccessful, motivating simulator-based QA generation instead of requiring initial coding competence.
Method: Sim2Reason procedurally constructs diverse physical systems, simulates their dynamics, and automatically generates verified numeric, reverse, and symbolic QA pairs for RL post-training.The pipeline uses a DSL, compiles scenes into MuJoCo, records grounded state and force traces, then filters degenerate, shortcut, and unstable examples.
Results: 5–10 percentage points: Sim2Reason improves zero-shot performance on IPhO mechanics problems across 3B to 32B model scales, with +17.9% on JEEBench for 32B models.Similar gains on PHYSICS support robust sim-to-real transfer beyond memorizing simulator dynamics.
Results: Sim2Reason QA accuracy correlates highly with real-world physics-benchmark performance, enabling scalable automated testing across specific physical domains.Quality filtering is critical to obtaining the reported gains.

2. Method

The method generates physically grounded synthetic QA data with MuJoCo, filters examples that permit shortcut solutions, and trains LLMs with verifiable-reward reinforcement learning. The pipeline spans procedural scene construction, simulation, question generation, filtration, and dynamic sampling.

Synthetic data generation: MuJoCo generates QA pairs covering kinematics, rotational mechanics, orbital motion, variable-mass systems, and basic electromagnetism.The simulator records physical quantities from generated scenes before converting time-series data into natural-language questions and answers.
Scene generation: The scene-generation DSL composes reusable bodies and entities through physically meaningful connections, enabling scalable randomization without a human in the loop.The DSL separates meaningful parameters, such as mass, from parameters that typically do not change system dynamics, such as pulley-string length.
Question generation: Questions test numeric forward reasoning, reverse inference over masked scene parameters, or symbolic reasoning with numeric values replaced by symbols.Each question selects a body, recorded quantity, and timestep from the simulated scene.
Data filtration: Approximately 15% of generated QA pairs are discarded when entity-removal or joint-removal ablations leave the ground-truth answer unchanged.This filtration removes examples solvable through shortcuts that ignore part of the scene or collapse multi-body dynamics into an oversimplified system.
Reinforcement learning: RLVR assigns positive reward when the final answer is within 5% relative error of the simulator value, while DAPO-style sampling resamples uninformative groups.The method optimizes GSPOZheng et al. (2025a) using group-relative advantages and dynamic sampling for sparse-reward training.

3. Experiments

SIM2REASON uses reinforcement learning on simulator-generated physics questions to improve LLM physical reasoning, transferring to real-world physics and math benchmarks without real-world physics QA post-training data. Across experiments, gains are strongest with filtered numeric data and remain robust against real-world QA and generic math-RL baselines.

Main Results: Up to 7 percentage points of IPhO Mechanics improvement occurs across model sizes, including a +4.4-point gain for Qwen3-30B-Instruct.These improvements arise even though post-training uses no real-world physics QA data.
Sim-to-Real Transfer: Qwen2.5-32B gains consistently across JEEBench, OlympiadBench, PHYSICS, AIME 2025, and MATH 500, with the largest improvement on JEEBench at +17.9 points.The JEEBench gain is attributed to mechanics questions aligned with simulator-covered phenomena, while math gains indicate broader transfer.
Training Comparison: RLVR improves both synthetic and real-world benchmarks more reliably than SFT, whose 200,000 teacher-generated trajectories produce only modest in-distribution gains and -3.9% on IPhO Mechanics.RLVR uses sparse, verifiable final-answer rewards, allowing diverse solution strategies while staying closer to the base policy.
Data Ablations: Filtered numeric training yields 13.15% versus 7.14% without filtering and 5.68% for the baseline, while numeric QA transfers best to IPhO.Shortcut filtering removes questions solvable through shortcuts, and Table 1 also shows improvements across reverse and symbolic synthetic evaluation modes.
Real-World Baselines: 40.0% on IPhO lets Sim2Reason outperform Prime P1 30B’s 38.6% despite using only simulator-generated synthetic post-training data.Prime P1 30B was trained on more than 5,000 curated physics QA pairs, whereas Sim2Reason used no real-world physics QA data during post-training.
Scalable Evaluation: Synthetic simulator accuracy correlates strongly with IPhO Mechanics accuracy across models, with Spearman ρ = 0.79, supporting simulator evaluation as a proxy for model and ablation comparisons.The pipeline also supports scalable mechanically grounded question generation with automatic verification.

4. Conclusion

SIM2REASON uses procedurally generated physics scenes and verifiable simulator-derived QA pairs to post-train LLMs with RLVR, yielding consistent zero-shot sim-to-real gains on real-world benchmarks. Future work includes combining synthetic and curated real-world QA and extending beyond classical mechanics.

4. Conclusion: SIM2REASON generates diverse physics scenes, converts simulated traces into verifiable QA pairs, and post-trains LLMs with RLVR.The pipeline uses simulators as a source of reasoning supervision.
4. Conclusion: Models trained only on synthetic simulator supervision show consistent zero-shot sim-to-real gains across multiple real-world benchmarks, including IPhO mechanics.These results suggest simulators can provide scalable reasoning supervision.
4. Conclusion: Future work should combine simulator-generated data with curated real-world QA to improve robustness and coverage.The proposed extension targets complementary strengths of synthetic and real-world supervision.
4. Conclusion: The approach could also extend beyond classical mechanics to electromagnetism, thermodynamics, and other physical sciences.These domains are identified as promising directions for broader application.

Impact Statement

The work aims to expand access to scientific tutoring and problem-solving tools while reducing reliance on scraped internet QA data, but it also raises misuse and simulator-fidelity risks requiring safeguards and real-world validation.

The primary expected benefit is broader access to high-quality scientific tutoring and problem-solving tools, with less dependence on scraped internet QA data.
Potential risks include misuse of stronger reasoning models for harmful engineering and over-reliance on simulator supervision that embeds unrealistic assumptions or failure modes.
Mitigations include evaluation on real-world benchmarks, transparent reporting of simulator fidelity and coverage limits, and safeguards, monitoring, and domain-specific validation in deployment.

A. Related Work

Prior work has used verifiable rewards, symbolic regression, and synthetic data or simulator interaction to scale reasoning and discover physical structure. This work instead targets natural-language questions about continuous physical dynamics by turning physics simulators into generators of structured post-training data.

Reinforcement Learning from Verifiable Feedback: RLVR trains reasoning-capable language models with automatically verifiable signals instead of relying solely on human preference annotation DeepSeek-AI et al. (2025); Shao et al. (2024); Yang et al. (2025); Yu et al. (2025).Examples of verification include exact-answer matching, program execution, theorem proving, and symbolic checks.
Symbolic Regression: Symbolic regression recovers interpretable physical laws using genetic programming Schmidt and Lipson (2009), sparse regression Brunton et al. (2016), and neural approaches Raissi et al. (2019); Udrescu and Tegmark (2020).These methods are often most successful in relatively simple, low-dimensional settings and can become brittle.
Synthetic Data training: Synthetic-data methods have scaled supervision in mathematics and robotics, including procedurally generated Olympiad geometry data and simulator training with automatic domain randomization Trinh et al. (2024).Other approaches use synthetic interaction data or self-play to create scalable training curricula without relying exclusively on human-written supervision Liu et al. (2025); Tajwar et al. (2025).
Novelty: Our work addresses a distinct physics setting where models answer natural-language questions about systems governed by continuous dynamics rather than tasks with clean symbolic structure and canonical supervision.It proposes turning physics simulators into generators of structured post-training data.

B. Domain-Specific Language and Timestep pruning strategy

The training-data pipeline adds a YAML-based domain-specific language for generating MuJoCo scenes and prunes unstable simulation-trace suffixes before question-answer generation.

Scene-generation DSL: A YAML-based scene-generation DSL is compiled into MuJoCo XML to produce rendered simulation scenes.Figure 4 presents the DSL and an example MuJoCo rendering.
Timestep pruning: Timestep pruning keeps the stable trace prefix and discards suffixes flagged by sliding-window deviations before QA generation.The heuristic addresses unmodelled transitions, such as contact between a block and pulley, that create unstable recorded signals.

C. Additional Results

Additional results show that performance on SIM2REASON synthetic questions correlates with IPhO mechanics accuracy, supporting the synthetic QA suite as a lightweight proxy for real-world physics reasoning.

C. Additional Results: Figure 10 shows that higher SIM2REASON synthetic-question accuracy tends to coincide with higher IPhO mechanics accuracy across models and runs.This supports using the synthetic QA suite as a lightweight proxy for real-world physics reasoning performance.

D. Bodies and their parameters

This section defines the bodies used in the simulations and identifies their randomizable parameters, listed in Table 8.

D. Bodies and their parameters: The simulation setup specifies a list of bodies together with their randomizable parameters in Table 8.

E. Recorded physical quantities

The simulator logs time-series data for every scene to generate question-answer pairs, organizing recorded quantities into mass-related, string-related, and contact categories.

Recorded data: Time-series data are logged for each simulated scene to support synthetic question-answer pair generation.
Recorded categories: Mass-related quantities capture body state and dynamics.
Recorded categories: String-related quantities cover length and tension, while contact quantities capture interaction forces.

F. Entities and their Connections

The section defines a library of physics entities with configurable parameters, connection points, and connection modes, spanning pulley, plane, collision, orbital, rotational, rolling, and electromagnetism setups. It also illustrates how these entities support simulated Olympiad problems and can be extended with novel entities for more complex scenes.

Entity vocabulary: The DSL defines entities through randomizable parameters and explicit connection points or modes, visualized across Figures 11–25.Table 8 summarizes the bodies and their corresponding randomizable parameters.
Pulley entities: Pulley entities support fixed, movable, and reverse-movable configurations, including three mass-system variants and connections that pull the movable pulley down or up.The movable pulley connects on both sides to mass_with_fixed_pulley variants, while the reverse variant reverses the pull direction.
Mechanics entities: Other mechanics entities model two-sided and stacked mass planes, suspended masses, inclined-plane systems, box-plane systems, and one-dimensional collision scenes with objects such as spheres, blocks, walls, and springs.These entities expose connection points for composing multi-body systems, while complex_collision_plane uses a frictionless plane and initial velocities to reduce problem complexity.
Advanced physics entities: The library also includes orbital, rocket, rotational, rolling, and electromagnetism entities, covering fuel-burning rockets, rolling 3D shapes with automated cutouts, and charged particles in varying electric and magnetic fields.The rocket tracks dry and initial mass plus fuel-loss rate; rolling entities use Blender-generated arbitrary cutout shapes; electromagnetic fields vary with position, velocity, and time.
DSL scaling: The DSL can be scaled by having LLMs invent new entities: they support simulations of some target Olympiad scenes even when raw simulator-code generation fails.For the F=MA 2024 and JEE Advanced 2019 examples, extending the DSL succeeds where raw code generation fails; the USA PhO 2019 example succeeds with both approaches.

MUJOCO OMNIVERSE

LLMs successfully ported a subset of DSL entities from MuJoCo to Omniverse, demonstrating portability across simulators and the potential to broaden simulator-based synthetic data generation.

MUJOCO OMNIVERSE: LLMs successfully transferred DSL entities from MuJoCo to Omniverse, demonstrating cross-simulator portability that could expand synthetic data generation.The evaluation used a subset of entities and assessed whether the DSL could be ported between the two simulators.