Source-linked AI summary
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Tarjei Paule Hage, Markus J. Buehler
TL;DR
Can hard, verifiable rewards teach compact language models transferable physics reasoning? This paper trains a compact model on beam mechanics with parameter-efficient RLVR and finds improved accuracy but brittle generalization under structural shifts, indicating template learning rather than robust equation internalization.
Problem
Whether compact language models can learn structured engineering reasoning from targeted, verifiable signals rather than relying on model scale remains insufficiently established.
Method
The study trains a distilled language model with parameter-efficient RLVR on synthetic beam-mechanics question-answer pairs using binary format and accuracy rewards, then evaluates in-distribution and shifted configurations.
Results
RLVR improves task accuracy and generalizes to additional loads, but performance degrades under changed support configurations and continued optimization despite maintained reward.
Takeaways & Limitations
Outcome-level verifiable rewards induce distribution-dependent procedural solution strategies, not reliable internalization of governing equations for robust engineering reasoning.
Takeaways & Limitations
The binary, sparse reward provides no graded signal for partially correct solutions, which may hinder learning first-principles reasoning.
Abstract
from arXiv · showhide
Can reinforcement learning with hard, verifiable rewards teach a compact language model to reason about physics, or does it primarily learn to pattern-match toward correct answers? We study this question by training a 1.5B-parameter reasoning model on beam statics, a classic engineering problem, using parameter-efficient RLVR with binary correctness rewards from symbolic solvers, without teacher-generated reasoning traces. The best BeamPERL checkpoint achieves a 66.7% improvement in Pass@1 over the base model. However, the learned competence is anisotropic: the model generalizes compositionally (more loads) but fails under topological shifts (moved supports) that require the same equilibrium equations. Intermediate checkpoints yield the strongest reasoning, while continued optimization degrades robustness while maintaining reward. These findings reveal a key limitation of outcome-level alignment: reinforcement learning with exact physics rewards induces procedural solution templates rather than internalization of governing equations. The precision of the reward signal - even when analytically exact - does not by itself guarantee transferable physical reasoning. Our results suggest that verifiable rewards may need to be paired with structured reasoning scaffolding to move beyond template matching toward robust scientific reasoning.
1 Introduction
The introduction motivates compact language models for structured engineering reasoning and proposes parameter-efficient reinforcement learning with verifiable rewards as a way to improve mechanics performance without teacher-generated reasoning traces. It frames the study around whether outcome-level alignment produces transferable understanding of governing equations or reward-driven specialization.
- Motivation: AI models are increasingly positioned to support engineering workflows, with human experts validating and refining their outputs for efficiency, reliability, and scale.The paper situates engineering reasoning within broader scientific and agentic AI workflows.
- Motivation: LLMs are being applied across scientific tasks including structural analysis and mechanical design, making reasoning processes central to their usefulness.These applications span idea generation, materials discovery, structural analysis, and mechanical design.
- Motivation: Structured engineering domains such as beam mechanics may not require massive models because they rely on physical principles, symbolic manipulation, and verifiable mathematical relationships.This motivates targeted adaptation of compact reasoning models rather than broad general-purpose scaling.
- Research questions: The study hypothesizes that parameter-efficient reinforcement learning with verifiable rewards can improve dense distilled models on mechanics problems without teacher-generated reasoning traces.It further asks whether such alignment internalizes governing equations and generalizes across problem configurations rather than producing reward-driven pattern specialization.
- Method: The proposed setup uses Group Relative Policy Optimization with accuracy and format rewards to elicit structured reasoning traces and correct equilibrium calculations without auxiliary supervision.This isolates outcome-level alignment while evaluating both in-distribution performance and generalization across configuration variations.
2 Results and Discussion
The study develops a synthetic beam-mechanics question-answer dataset, fine-tunes a distilled language reasoning model with parameter-efficient RLVR, and evaluates it on in-distribution, out-of-distribution, and mathematical reasoning tasks.
- The study first constructs a synthetic dataset of beam-mechanics-specific question-answer pairs.
- It then applies parameter-efficient RLVR fine-tuning to a dense, distilled language reasoning model using the constructed dataset.
- Evaluation covers held-out in-distribution examples, out-of-distribution cases testing generalization beyond the training support, and standard mathematical reasoning benchmarks.
- The beam setup uses a simply supported one-dimensional beam with distinct pinned and roller supports and multiple transverse point loads at distinct positions.
2.1 Problem Definition and Dataset Generation
This section defines simply supported beam reaction-force problems and constructs a symbolic training dataset generated and solved through equilibrium equations. The dataset varies beam lengths, load magnitudes, and load positions within a controlled configuration subset.
- Problem Definition: The task calculates support reactions for a simply supported beam modeled as a one-dimensional continuum with pinned and roller supports.The pinned support supplies horizontal and vertical reactions, while the roller supplies a vertical reaction.
- Problem Definition: Beam reactions are determined by enforcing horizontal-force, vertical-force, and moment equilibrium equations.With no horizontal loads, RH,pin = 0; the two vertical reactions are solved from force and moment equilibrium.
- Dataset Generation: Training beams use lengths L ∈ {l, 2l, 3l}, supports at xpin = 0 and xroller = L, and single loads P ∈ {−p, −2p, −3p}.Load positions are sampled at xP ∈ {0.05 k L | k = 0, 1, 2, ..., 20}.
- Dataset Generation: A modified SymBeam solver produces analytical and numerical beam responses, while an LLM generates self-contained configuration questions.The solver evaluates reaction forces, internal force and moment distributions, deflections, and rotations.
- Dataset Generation: A representative example asks for reactions under a −3 ∗P load at x = 0.9 ∗L on a beam of length 2 ∗L and returns [1.65P, 1.35P].The example uses a pin at x = 0 and a roller at x = 2 ∗L.
2.2 Training Strategy
The training strategy combines GRPO-based RLVR with LoRA-based parameter-efficient adaptation of a distilled 1.5B reasoning model, using verifiable beam-statics rewards. Training reward rises sharply early and then plateaus, with initial gains linked primarily to improved response formatting.
- RLVR optimization: GRPO-based RLVR adapts a dense distilled reasoning model to beam equilibrium problems without an explicit value function.GRPO uses policy-gradient optimization by comparing generated outcomes, while binary rewards guide sequences toward correct final answers.
- Initialization: The base model’s prior beam-statics reasoning capacity is critical because insufficient knowledge may prevent informative rewards from initiating learning.The study therefore explored strategies to ensure the starting model could successfully begin reinforcement learning.
- Parameter-efficient adaptation: LoRA adapters provide parameter-efficient adaptation while all pretrained backbone weights remain frozen.The pipeline builds on Tina, which extends Open R1 for GRPO-based PERL and LoRA-based adaptation.
- Training dynamics: 0.2 to 0.8: average training reward increases over the first 120 training examples before approximately plateauing for the remainder of reinforcement learning.Reward scores range from 0 to 1, with intermittent spikes after the plateau.
- Training dynamics: Early performance gains are primarily driven by improved output formatting, as completion length decreases toward a task-appropriate token length.The model learns to structure responses in ways that enable successful problem solving during the initial training phase.
2.3 Evaluation Strategy
The evaluation tracks Pass@1, Pass@7, and Majority@7 across checkpoints to characterize training dynamics and generalization. It separates in-distribution beams from multi-load and varying-support out-of-distribution configurations, revealing distinct transfer behavior.
- Evaluation metrics: Evaluation tracks Pass@1, Pass@7, and Majority@7 across evenly spaced checkpoints to measure one-shot accuracy, best-case performance, and output consistency.These metrics characterize how performance evolves during fine-tuning and identify when it is most effective.
- Generalization partitions: Evaluation partitions examples into end-supported beams with one load, end-supported beams with multiple loads, and beams with varying support locations.Each partition is evaluated using Pass@1, Pass@7, and Majority@7 averaged over the corresponding example types.
- Multiple-load OOD: The base model produces no correct outputs on the multiple-load OOD example, whereas the best-performing BeamPERL checkpoint generates a fully correct response.This qualitative result demonstrates generalization to an unseen structural configuration.
- Varying-support OOD: On varying-support OOD examples, the best-performing checkpoint generalizes successfully, but the final checkpoint produces no correct outputs after later-training degradation.The failure preserves the required format while becoming incoherent and semantically meaningless under distribution shift.
- Training dynamics: Training progresses from learning output structure and improving ID performance to intermediate-checkpoint generalization on OOD configurations, followed by degraded later-stage reasoning.Required reasoning tags and boxed final answers emerge early, while peak intermediate checkpoints solve configurations the base model fails.
2.4 Discussion
PE-RLVR-FT improves beam-mechanics performance but produces anisotropic generalization and peaks at intermediate checkpoints. These dynamics suggest sparse, binary outcome rewards can specialize compact models without reliably producing transferable engineering reasoning.
- 2.4 Discussion: PE-RLVR-FT improves standardized beam-mechanics performance, but generalization is anisotropic and performance peaks at intermediate checkpoints.Increasing policy divergence is not uniformly associated with improved reasoning robustness.
- 2.4 Discussion: Accuracy on multi-load OOD beams continues improving even as format reward declines, indicating gains beyond improved output formatting.The contrast suggests substantive beam-mechanics improvement for multi-load problems rather than merely betterstructured outputs.
- 2.4 Discussion: The format-and-accuracy reward structure induces a two-stage dynamic: rapid formatting alignment followed by task-specific specialization.Reward design therefore shapes both the training trajectory and downstream evaluation performance.
- 2.4 Discussion: Support-location shifts remain brittle: semantic correctness degrades during later training while ID and multi-load OOD performance continue improving.Different OOD configurations preserve different amounts of the structure learned from simply supported beams with end supports.
- 2.4 Discussion: Domain-specific RL improves compact-model performance with low scale and computational overhead, but this efficiency reduces robustness.The findings frame outcome-level alignment as a diagnostic of limitations in structured scientific reasoning.
3 Conclusion and Future Directions
RLVR with LoRA adapters improves compact language models on standardized beam mechanics without teacher-generated reasoning traces, with intermediate checkpoints performing best on in-distribution and selected out-of-distribution cases. However, outcome-level rewards can produce result-oriented, template-based competence, motivating process-level verification, broader topology coverage, and structured reasoning scaffolding.
- Findings: PE-RLVR-FT improves dense, distilled language models on standardized mechanics problems without teacher-generated traces or full-parameter updates.Frozen base models with tunable LoRA adapters improve format adherence and task accuracy during training.
- Findings: Intermediate checkpoints yield higher in-distribution and selected out-of-distribution performance, while parametric generalization indicates that outcome-level supervision can induce structured solution strategies.The reported generalization is along parametric, linear problem dimensions.
- Future directions: The composite reward assigns one third to format and two thirds to accuracy, but these weights were selected empirically rather than through controlled sensitivity studies.Future sweeps should measure effects on in-distribution performance and out-of-distribution anisotropic generalization.
- Future directions: Process-level verifiable rewards could reduce reward gaming by evaluating intermediate logic rather than relying primarily on correct final answers.The current outcome-level reward may permit flawed reasoning to produce correct beam-mechanics answers.
- Future directions: Future studies should expand support configurations, load types, boundary conditions, task scope, base-model coverage, and multi-agent workflows to test robustness and scalability.The proposed extensions include internal forces, displacements, frames, trusses, minimal reasoning priors, and collaborative output refinement.
- Future directions: A hybrid pipeline could combine structured thought integration with RLVR refinement using deterministic, verifiable rewards from symbolic solvers.This synthesis is proposed to address procedural template learning by providing richer initial reasoning structure alongside precise physics-based correctness signals.
4 Materials and Methods
The study trains DeepSeek-R1-Distill-Qwen-1.5B with GRPO and parameter-efficient LoRA adapters, using binary format and accuracy rewards for verifiable beam-mechanics outputs. Checkpoints are evaluated against the unmodified base model with pass@1, pass@7, and majority@7, alongside mathematics benchmarks.
- RL training: GRPO samples multiple outputs per prompt and updates the policy to increase the probability of higher-ranked outputs.Advantages are computed from relative rewards within each sampled output group.
- Model and parameter-efficient training: DeepSeek-R1-Distill-Qwen-1.5B is fine-tuned with frozen base weights and trainable LoRA adapters.The adapters use rank 32, scaling factor 4, dropout 0.05, and target attention and feed-forward projection modules.
- Reward design: Binary format and accuracy rewards jointly enforce machine-readable structure and verifiable physical correctness, prioritizing correctness over formatting.The format reward requires exactly one <think> opening tag, one </think> closing tag, and at least one non-empty \boxed{} expression; each reward returns 1 or 0.
- Evaluation: Nine checkpoints are merged into the base model for evaluation, while the unmodified base model serves as a baseline.Each checkpoint is evaluated on 24 beam examples, with seven completions generated per prompt.
- Evaluation: Evaluation reports pass@1, pass@7, and majority@7 to measure one-shot performance, best-case sampled performance, and output consistency.Majority@7 requires at least four of seven completions to receive a positive accuracy reward.
- Evaluation: Selected checkpoints and the base model are additionally evaluated on AIME24, AIME25, and AMC23 using LightEval.These are Hugging Face Hub mathematics benchmarks covering the 2024 and 2025 AIME tests and the 2023 AMC.