BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning

1 Introduction

• Compact, task-specific models may achieve reliable beam-mechanics reasoning through parameter-efficient RL aligned to governing equations rather than massive models or full-parameter training.Beam mechanics provides structured physical principles, symbolic manipulation, and verifiable mathematical relationships suitable for targeted adaptation.Source: §1 · pp. 1–2

1.1 Reinforcement Learning for Incentivizing Reasoning

• Prior work established that outcome-level optimization can enhance compact-model reasoning, motivating comparison with deterministic binary physical-correctness rewards from symbolic solvers.PRefLexOR used masked reasoning tokens and preference-derived quality signals, while this study asks whether stricter rewards produce equation internalization or procedural specialization.Source: §1 · p. 2, §1.1 · p. 3

1 Introduction

• The study tests whether PEFT with GRPO accuracy and format rewards can improve a dense LRM on beam-support reaction calculations without teacher-generated reasoning traces.The setup evaluates both strong in-distribution performance and generalization across beam-configuration variations under outcome-level alignment.Source: §1 · pp. 2–3

1.2 Open-Source Reproductions and Efforts Toward Efficient LRM Development

• Open-source RLVR efforts and PEFT methods motivate applying LoRA-based parameter-efficient reinforcement learning to small distilled LRMs for standardized mechanics problems.These efforts report improved compact-model reasoning while highlighting the roles of distillation, cold-start training, and lightweight trainable adapters.Source: §1.2 · p. 4

1.3 Outline of Paper

• The paper proceeds from synthetic beam-data construction and PE-RLVR-FT training to in- and out-of-distribution evaluation, checkpoint analysis, discussion, and conclusions.It also evaluates selected checkpoints on standard mathematical-reasoning benchmarks and reports experimental methods and materials separately.Source: §1.3 · p. 4

Study design

• The study develops synthetic beam-mechanics question–answer pairs and uses them to PE-RLVR-FT a dense, distilled language reasoning model.Source: §2 · p. 4

Evaluation

• Evaluation covers held-out in-distribution data, out-of-distribution examples beyond training support, and standard mathematical reasoning benchmarks.The benchmarks assess task specialization, general reasoning ability, and potential catastrophic forgetting.Source: §2 · p. 4

Beam mechanics task

• The benchmark task models a simply supported beam with distinct pinned and roller supports and multiple distinct transverse point loads.The beam is defined on x ∈ [0, L], with support locations x_pin and x_roller and load positions x_i.Source: Figure 1 · p. 5

• The beam is a one-dimensional span with distinct pin and roller supports, material properties E and I, and N distinct normally applied point loads.Each configuration is specified by {L, E, I, xpin, xroller, x, P}.Source: §2.1 · p. 5
• Support reactions are determined by enforcing horizontal-force, vertical-force, and moment equilibrium, with RH,pin = 0 because no horizontal loads are applied.The unknown vertical reactions RV,pin and RV,roller are obtained by solving the vertical-force and moment equations after substituting the applied loads.Source: §2.1 · p. 5
• 189 distinct training configurations combine lengths {l, 2l, 3l}, endpoint supports, single loads {−p, −2p, −3p}, and 21 discrete load locations.The load location is sampled as xP = 0.05kL for k = 0, 1, ..., 20.Source: §2.1 · p. 6
• 756 training question–answer pairs are generated by applying four self-contained question formulations to each configuration and solving them with modified SymBeam and SymPy.The solver produces analytical and numerical reaction-force solutions, while also computing internal forces, moments, deflections, and rotations.Source: §2.1 · p. 6

2.2 Training Strategy

• RLVR adapts a distilled reasoning model using GRPO and binary symbolic-solver rewards, allowing it to learn without explicit reasoning-trace supervision.GRPO compares sampled response groups and updates their relative probabilities; rewards cover final-answer correctness and lightweight format adherence.Source: §2.2 · pp. 6–8
• Pure RL from small pretrained LLMs struggled under sparse rewards, motivating initialization from the distilled DeepSeek-R1-Distill-Qwen-1.5B model instead of adding domain-specific SFT traces.The selected distilled model is intended to provide sufficient prior reasoning capacity for informative initial rewards while preserving self-taught learning.Source: §2.2 · p. 7, §2.2.1 · p. 8
• LoRA adapters are trained while the pretrained backbone remains frozen, enabling computationally efficient parameter-efficient post-training.The workflow samples candidate responses, evaluates format and beam-statics correctness deterministically, and uses GRPO advantages to update only adapter parameters.Source: §2.2 · p. 8, Figure 3 · p. 8

2.2.1 Training Results

• Reward increases from approximately 0.2 to 0.8 within the first 120 training examples, then plateaus with intermittent fluctuations between roughly 0.6 and 0.85.Both format and accuracy rewards rise initially, but accuracy later decreases slightly with high variance while format reward remains above 0.8.Source: §2.2.1 · p. 8
• Reward gains occur as completion length decreases to a task-appropriate level and then stabilizes, indicating improved solution efficiency without increased token usage.The authors attribute early gains primarily to better response formatting and task-aligned structure.Source: §2.2.1 · p. 9
• After approximately 120 training examples, sharply increasing KL divergence accompanies diminished marginal reward gains, prompting checkpoint evaluations of strategy evolution and out-of-distribution behavior.The proposed explanation is that optimization shifts from refining baseline response patterns toward greater deviation from the base model.Source: §2.2.1 · p. 9

2.3.1 Evaluation Data

• The evaluation uses 24 deliberately structured samples spanning 4 held-out ID single-load beams, 8 OOD multiple-load beams, and 12 OOD beams with shifted supports.Held-out examples exclude training parameter combinations and use non-grid beam lengths, loads, and load locations while keeping Young’s modulus and area moment of inertia fixed.Source: §2.3.1 · p. 10

2.3.2 Evaluation Results

• 66.7% improvement in Pass@1 and 42.9% improvement in Pass@7 over the base model is achieved by the best BeamPERL checkpoint, while Majority@7 rises from 0% to 4.17%.Performance peaks after 80–120 training examples, plateaus, and then declines at later checkpoints.Source: Table 1 · p. 11, §2.3.2 · p. 11
• Pass@7 increases throughout training for OOD beams with multiple loads, but performance on OOD beams with varying support locations peaks after 80–120 examples and subsequently declines.ID Pass@7 remains approximately 50%, while ID Pass@1 and Majority@7 converge to 50% by the end of training.Source: §2.3.2 · p. 11, Figure 7 · p. 12

2.3.3 Sample Results

• Qualitative samples show retained ID correctness and successful later-stage generalization to multiple loads, but shifted-support outputs can preserve formatting while becoming incoherent and semantically invalid.The base model produces no correct outputs on both OOD examples; the best checkpoint succeeds on each, while the final checkpoint fails on the shifted-support case.Source: §2.3.3 · pp. 12, 15, 17
• Intermediate checkpoints solve out-of-distribution beam configurations that the base model misses, whereas continued training preserves in-distribution performance but causes fragile, sometimes incoherent behavior under distribution shift.The model first learns output formatting and in-distribution performance, then develops out-of-distribution generalization before late-stage reinforcement learning trades robustness for specialization.Source: §2.3.3 · p. 17

2.3.4 Effects of Task Specialization on General Reasoning Ability

• Best-performing BeamPERL modestly improves AMC23 and AIME24 over the base model while matching it on AIME25, indicating no immediate loss of general mathematical reasoning.Performance remains comparable after 80 training examples, but declines after roughly 200 examples and worsens by the final 360-example checkpoint, consistent with catastrophic forgetting.Source: §2.3.4 · p. 17

Core interpretation

• PE-RLVR-FT improves beam-mechanics performance, but anisotropic generalization, intermediate-checkpoint peaks, and later robustness loss reveal limits of sparse outcome-level alignment.Continued optimization can preserve reward while degrading transfer, suggesting procedural solution templates rather than internalized governing equations.Source: §2.4 · pp. 17–18

Reward dynamics

• Sparse binary rewards work most effectively when the base model already produces occasional correct solutions alongside structurally formatted outputs.Format rewards reliably improve output structure, but accuracy may stagnate when the pretrained model lacks sufficient initial solution competence.Source: §2.4 · p. 18

Generalization dynamics

• RL induces selective generalization: accuracy improves for multiple-load beams despite declining format reward, but both accuracy and robustness deteriorate under shifted support locations.This separates genuine task improvement from formatting gains and exposes brittleness under structural shifts.Source: §2.4 · pp. 18–19, Figure 9 · p. 19

Generalization structure

• Multiple loads support compositional reuse of learned equilibrium-solving steps, whereas moved supports alter reaction relationships and lever arms beyond the training structure.Thus, mechanics generalization is hierarchical: shifts preserving core structure are more transferable than shifts changing the governing constraint regime.Source: §2.4 · pp. 19–20

Training dynamics

• Early training improves Pass@7 with minimal KL divergence, whereas later optimization produces sharp, variable distributional drift and can reward superficial alignment over reasoning competence.The two-stage process moves from formatting and efficiency gains toward specialization, reward hacking, and diminishing returns beyond the best checkpoint.Source: §2.4 · p. 20, Figure 10 · p. 20

Implications

• Domain-specific RL efficiently improves narrow engineering performance but can erode general reasoning, while PRefLexOR [27] suggests structured reasoning scaffolding may complement exact verifiable rewards.More training is not inherently better, and reward precision alone does not guarantee transferable reasoning.Source: §2.4 · pp. 21–22

3 Conclusion and Future Directions

• The learned policies generalize along parametric problem dimensions but degrade under topological shifts, indicating procedural templates rather than internalized first-principle equilibrium representations.Outcome-level supervision can induce structured solution strategies without guaranteeing transferable physical reasoning.Source: §3 · p. 22
• PE-RLVR-FT improves dense, distilled models on standardized mechanics problems without teacher-generated traces or full-parameter updates, but its competence degrades under configuration shifts and continued optimization.The resulting method is computationally efficient for task specialization, but not necessarily sufficient for stable, generalizable engineering reasoning.Source: §3 · p. 22
• BeamPERL releases an open-source beam-mechanics dataset-generation and PE-RLVR-FT framework with full training and evaluation datasets, reproducible experiments, and an evaluation protocol.These resources are intended to support transparent benchmarking and future extensions.Source: §3 · p. 22

3.1 Future Directions

• Future reward research should ablate the one-third format and two-thirds accuracy weighting, add verifiable process rewards, and study dynamic weighting, KL regularization, early stopping, and checkpoint selection.These changes aim to reduce reward gaming and catastrophic forgetting while balancing task performance with broader reasoning capability.Source: §3.1 · pp. 22–23
• Future datasets and curricula should diversify support configurations, load types, boundary conditions, internal-force and displacement tasks, frames, and trusses to test whether models learn equilibrium invariants and cross-domain structural principles.Controlled topological variation could distinguish limited training coverage from inherent limitations of outcome-level alignment.Source: §3.1 · p. 23
• Future evaluations should vary base models, reasoning priors, rollout counts, LoRA and sampling settings, reward weights, learning schedules, and batch sizes, while exploring multi-agent, tool-augmented, and structured-knowledge workflows.A hybrid curriculum could first provide reasoning scaffolding through preference-derived signals, then refine it with deterministic symbolic-solver rewards, addressing procedural template learning.Source: §3.1 · pp. 23–24

4.1 Experimental Setup

• Experiments run on two NVIDIA L4 GPUs with fixed seeds, but stochastic sampling, GPU parallelism, and vLLM scheduling prevent full bit-for-bit determinism.Training uses Python 3.10, CUDA 11.8, PyTorch, DeepSpeed, and vLLM; repeated runs produce similar but not identical results.Source: §4.1 · p. 24

4.2 Synthetic Dataset Generation via Automated Prompt Generation for Large Language Models

• Synthetic questions are generated from beam parameters using INT8-quantized DeepSeek-R1-Distill-Qwen-7B, while SymBeam symbolically computes analytical reaction-force answers.Multiple questions are generated per beam configuration, with shared answers stored as ordered reaction-force lists.Source: §4.2 · p. 24

4.3 Parameter-Efficient Reinforcement Learning Fine-Tuning with GRPO

• GRPO trains DeepSeek-R1-Distill-Qwen-1.5B by sampling multiple outputs and increasing the probability of outputs with higher within-group rewards.Each prompt produces G sampled outputs, with advantages computed from their relative rewards; optimization uses an outcome-dependent objective without a clipped surrogate or reference-model constraint.Source: §4.3 · pp. 24–25
• LoRA freezes the 1.5B base model and trains rank-32 adapters, reducing trainable parameters from 1.777 billion to 36.93 million, or 97.9%.Adapters target attention and feed-forward projection modules with scaling factor 4 and dropout 0.05, enabling training on two GPUs.Source: §4.3 · p. 25

4.3.1 Reward Function Design

• Binary format and accuracy rewards enforce machine-readable outputs and exact reaction-force correctness, weighting physical accuracy two-thirds and format compliance one-third.Accuracy requires multiset matching of extracted coefficients with tolerance ε = 10^-4; binary rewards avoid ambiguous partial credit but create sparse learning signals.Source: §4.3.1 · p. 26

4.4 Evaluation Protocol

• Evaluation compares nine adapter checkpoints with the base model on beam examples using pass@1, pass@7, and majority@7, plus selected mathematical benchmarks.Seven temperature-sampled completions are scored using the training rewards; mathematical evaluations cover AIME24, AIME25, and AMC23 with extractive matching.Source: §4.4 · pp. 26–27