PaperlayerView exact revision on arXiv

Source-linked AI summary

CALM Before the STORM: Unlocking Native Reasoning for Optimization Modeling

Zhengyang Tang, Zihan Ye, Chenyu Huang, Xuhan Huang, Chengpeng Li, Sihang Li, Guanhua Chen, Ming Yan, Zizhuo Wang, Hongyuan Zha, Dayiheng Liu, Benyou Wang

arXiv:2510.04204v1cs.CLcs.AIcs.CEcs.LG

TL;DR

Existing adaptation methods can misalign with LRMs’ native reasoning, and direct fine-tuning on non-reflective data shows limitations. CALM uses lightweight expert hints and solver feedback to refine reasoning before supervised fine-tuning and reinforcement learning; the resulting 4B STORM reaches 68.9% average accuracy across five benchmarks, comparable to a 671B model.

  • Problem

    Existing adaptation methods may impose artificial reasoning modes and rely on non-reflective data, limiting how well they exploit LRMs’ native reasoning patterns.

    Source: §1 · p. 2

  • Method

    CALM uses solver execution and concise expert hints to correct reasoning flaws, then uses the resulting trajectories for supervised fine-tuning followed by reinforcement learning.

    Source: §1 · p. 2

  • Results

    68.9% macro-average accuracy across five benchmarks makes 4B STORM comparable to 671B DeepSeek-R1-0528 at 67.5%.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    CALM offers a minimally invasive, reasoning-aligned approach that preserves native reasoning while improving optimization-modeling accuracy.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large Reasoning Models (LRMs) have demonstrated strong capabilities in complex multi-step reasoning, opening new opportunities for automating optimization modeling. However, existing domain adaptation methods, originally designed for earlier instruction-tuned models, often fail to exploit the advanced reasoning patterns of modern LRMs -- In particular, we show that direct fine-tuning on traditional \textit{non-reflective} datasets leads to limited gains. To fully leverage LRMs' inherent reasoning abilities, we propose \textbf{CALM} (\textit{Corrective Adaptation with Lightweight Modification}), a framework that progressively refines LRMs within their native reasoning modes for optimization modeling tasks. In CALM, an expert intervener identifies reasoning flaws and provides concise corrective hints, which the LRM incorporates to produce improved reasoning trajectories. These interventions modify fewer than 2.6\% of generated tokens, but generate high-quality data for soft adaptation through supervised fine-tuning. The adapted model is then further improved through reinforcement learning. Building on CALM, we develop \textbf{STORM} (\textit{Smart Thinking Optimization Reasoning Model}), a 4B-parameter LRM that achieves a new state-of-the-art average accuracy of 68.9\% across five popular optimization modeling benchmarks, matching the performance of a 671B LRM. These results demonstrate that dynamic, hint-based data synthesis both preserves and amplifies the native reasoning patterns of modern LRMs, offering a more effective and scalable path towards expert-level performance on challenging optimization modeling tasks.

1 INTRODUCTION

Existing adaptation methods rely on static, non-reflective data and can impose reasoning modes misaligned with LRMs’ native patterns. CALM uses lightweight, solver-informed interventions to refine those patterns, and STORM reaches state-of-the-art average accuracy across five benchmarks with 4B parameters.

1 INTRODUCTION

  • Existing methods use static problem–solution datasets, while LRMs support iterative, adaptive reasoning within a single inference pass.The paper calls these iterative reasoning capacities native reasoning patterns.Source: §1 · p. 2
  • Prior adaptation methods can impose artificial reasoning modes and use non-reflective data, misalignments that may yield only marginal improvements.The paper identifies preserving native reasoning as a central adaptation challenge.Source: §1 · p. 2
  • The authors evaluate recurring flaws in native reasoning, identifying seven types and grouping them into Code Utilization Distrust and Lack of OR Expertise.They ask whether native LRM reasoning can be leveraged effectively for optimization modeling.Source: §1 · p. 2
  • CALM uses solver-code execution and expert hints to correct reasoning flaws, with interventions modifying fewer than 2.6% of generated tokens.The framework is designed to align LRM reasoning with optimization modeling requirements.Source: §1 · p. 2
  • CALM-generated expert trajectories support supervised fine-tuning for soft adaptation, followed by reinforcement learning to refine skills toward autonomous mastery.The resulting model is named STORM.Source: §1 · p. 2
  • STORM, a 4B-parameter model, achieves state-of-the-art average accuracy across five optimization modeling benchmarks, matching a 671B LRM.The controlled analysis also reports that CALM-based adaptation is crucial for reinforcement-learning success.Source: §1 · p. 3

2 BACKGROUND AND MOTIVATION

The section contrasts one-shot, non-reflective generation with solver-feedback reasoning, then shows that direct fine-tuning trades easier-task gains for weaker complex-task performance. Expert analysis organizes native reasoning flaws into two categories whose prevalence varies with problem difficulty.

2.1 BACKGROUND: LLMS FOR OPTIMIZATION MODELING

  • Automated optimization modeling translates natural-language problem descriptions into mathematical models and executable solver code.The section distinguishes two paradigms by how they obtain the final solution.Source: §2.1 · p. 3
  • Traditional non-reflective generation produces a complete model and solver code in one linear step, without execution feedback or revision.This contrasts with the reflective paradigm described for modern LRMs.Source: §2.1 · p. 3

2.2 PILOT STUDY: ADAPTING LRMS WITH NON-REFLECTIVE DATA

  • Fine-tuning on existing non-reflective data improves accuracy on MAMO-Easy but sharply reduces performance on the more complex IndustryOR and OptMath benchmarks.Table 1 presents base-LRM performance before and after SFT on the existing dataset.Source: §2.2 · p. 4, Table 1 · p. 4
  • The authors suggest static problem–solution pairs may push LRMs toward rigid generation, improving simple cases while undermining complex-task reasoning.They conclude that adaptation should preserve native reasoning patterns.Source: §2.2 · p. 4

2.3 A TAXONOMY OF FLAWS IN LRM’S NATIVE REASONING

  • Human OR experts annotated base-LRM responses through a multi-stage process to identify recurring error patterns and establish the flaw taxonomy.The protocol included annotation, clustering, and refinement.Source: §2.3 · p. 4
  • Expert analysis identifies six major flaws among seven types and groups them as Code Utilization Distrust or Lack of OR Expertise.Code-utilization flaws include manual calculations or fragmented code; OR-expertise flaws include flawed formulations, missed constraints, and implementation errors.Source: §2.3 · pp. 4–5
  • Flaw prevalence varies with difficulty: Code Utilization Distrust dominates easy-to-medium tasks, while Lack of OR Expertise is the main barrier on complex benchmarks.The distribution is shown across evaluated benchmarks in Figure 3.Source: Figure 3 · p. 4, §2.3 · p. 5

3 METHODOLOGY

The methodology formalizes reflective reasoning as an interpreter-mediated trajectory, then uses CALM to correct flawed reasoning with localized hints. Filtered trajectories support supervised soft adaptation followed by reinforcement learning to train STORM.

3.1 PRELIMINARIES: FORMALIZING THE REFLECTIVE GENERATION FLOW

  • The Reasoner’s flow is a sequence of text reasoning, code actions, and execution outputs; the objective is to refine its policy toward trajectories yielding correct solutions.Source: §3.1 · p. 5

3.2 CALM: CORRECTING ADAPTATION WITH LIGHTWEIGHT MODIFICATION

  • CALM pairs flaw-specific hints with intervention at an erroneous step, resumes reasoning, and repeats checks until the trajectory is flaw-free or the intervention cap is reached.The iterative corrections are localized and minimally invasive, preserving the model’s native reasoning.Source: §3.2 · pp. 5–6

3.3 TRAINING PIPELINE: FROM SOFT ADAPTATION TO AUTONOMOUS MASTERY

  • CALM retains only trajectories with correct final answers and flaw-free reasoning for DCALM, then SFT soft-adapts the base LRM without enforcing a rigid non-reflective pattern.Source: §3.2 · p. 6, §3.3 · p. 6
  • After SFT, GRPO optimizes correctness with up to four interpreter executions per rollout and a binary final-outcome reward; execution-output masking improves stability, yielding STORM.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments benchmark STORM across five optimization-modeling benchmarks, then use ablation and behavioral analyses to examine CALM’s contributions and changes to reasoning.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans five benchmarks, with pass@1 averaged over eight independent samples per problem and up to four code executions per LRM trajectory.Baselines include foundation, reasoning, agent-based, and learning-based systems, plus the unadapted Qwen3-4B-Thinking-2507.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 68.9% macro-average accuracy is an 11.8-point gain over the 57.1% base model; the 4B STORM matches 671B DeepSeek-R1-0528 at 67.5%.STORM also reaches 70.3% on MAMO-Complex, a new state-of-the-art for learning-based methods.Source: Table 2 · p. 7, §4.2 · p. 8

4.3.1 ABLATION STUDY: THE TWO-STAGE LEAP TO SOTA

  • CALM-curated SFT raises macro-average accuracy from 57.1% to 58.7%, then RL raises it from 58.7% to 68.9%.SFT gently corrects flaws without overwriting native patterns; Figure 5 tracks performance from the base model through both stages.Source: §4.3.1 · p. 8, Figure 5 · p. 8

4.3.2 DECONSTRUCTING CALM: AN INSIDE LOOK AT THE CURATION PROCESS

  • CALM curates data by diagnosing native flaws, iteratively hinting trajectories, and filtering for correct answers with flawless reasoning.Code Utilization Distrust and Lack of OR Expertise are dominant flaws; hinting boosts success while reducing response length.Source: §4.3.2 · p. 9

4.3.3 BEHAVIORAL EVOLUTION: HOW CALM SHAPES REASONING

  • With CALM-SFT, RL learns faster and more stably, uses more code with shorter responses, and SFT reduces OR-expertise flaws while RL reduces code distrust.Within the same computational budget, RL with CALM reaches a higher performance ceiling than the unguided control.Source: §4.3.3 · pp. 9–10

5 CONCLUSION

CALM adapts LRM reasoning through targeted interventions, followed by supervised fine-tuning and reinforcement learning. The resulting STORM achieves state-of-the-art macro-average performance across five optimization modeling benchmarks.

5 CONCLUSION

  • CALM combines hint-guided supervised fine-tuning with reinforcement learning to adapt a compact LRM into STORM, achieving state-of-the-art macro-average performance across five benchmarks.The framework aligns targeted interventions with reasoning flaws while preserving native reasoning capabilities.Source: §5 · p. 10

A RELATED WORK

  • Earlier OR methods framed modeling as a static, non-reflective generation task, whereas this work aims to preserve and guide LRMs’ iterative, adaptive reasoning.The passage identifies ORLM, LLMOPT, and SIRL as examples of early methods.Source: A RELATED WORK · p. 12
  • CALM’s automated, dynamic Reasoner-Intervener pattern uses an expert model to detect flaws and inject tailored hints, unlike static hint libraries or manual oversight.The related-work comparison cites START [Li et al., 2025b] and CoRT [Li et al., 2025a].Source: A RELATED WORK · p. 12

B ILLUSTRATION: INPUT–OUTPUT STRUCTURE OF TRADITIONAL LLMS FOR OPTIMIZATION PROBLEMS

  • The illustration formalizes a 25-ton transportation problem with binary mode-selection variables, nonnegative shipment volumes, cost minimization, and capacity and demand constraints.The example also prohibits choosing trucks and ships together.Source: B ILLUSTRATION: INPUT–OUTPUT STRUCTURE… · p. 12

C PROTOCOL FOR HUMAN-IN-THE-LOOP FLAW TAXONOMY CREATION

A four-expert, multi-stage protocol converts observed LRM errors into a systematic taxonomy of seven recurring flaw types.

Stage 1: Initial Data Generation and Independent Annotation

  • Four experts independently annotated responses to 50 benchmark problems generated by Qwen3-4B-Thinking-2507, recording reasoning errors with descriptive tags and brief justifications.The resulting annotations were initially unstructured.Source: C PROTOCOL FOR HUMAN-IN-THE-LOOP FLAW… · p. 13

Stage 2: Collaborative Clustering and Taxonomy Refinement

  • The team merged tags, affinity-clustered semantically similar errors, and refined operational definitions until the categories were mutually exclusive and collectively exhaustive for observed phenomena.Ambiguous or overlapping clusters were merged or refined through discussion.Source: C PROTOCOL FOR HUMAN-IN-THE-LOOP FLAW… · p. 13

C PROTOCOL FOR HUMAN-IN-THE-LOOP FLAW TAXONOMY CREATION

D TRIGGERS TYPE

The appendix defines seven LRM error triggers, distinguishes substantive reasoning flaws from a procedural formatting error, and also describes benchmark splitting and a pilot fine-tuning setup.

D TRIGGERS TYPE

  • The seven triggers cover premature natural-language solving, fragmented coding, redundant manual verification, lack of sanity checking, flawed modeling, implementation errors, and protocol violations.Triggers 1–3 indicate Code Utilization Distrust, triggers 4–6 indicate Lack of OR Expertise, and trigger 7 checks formatting; the main analysis focuses on the six substantive flaws.Source: D TRIGGERS TYPE · pp. 13–14, Table 3 · p. 14

E.1 BENCHMARK DATASETS AND SPLITTING STRATEGY

  • Data from eight sources were randomly partitioned into non-overlapping SFT, RL, and test sets; the main evaluation uses five benchmarks for comparability with prior work.Test splits are provided for all datasets to support future research.Source: §E.1 · p. 16, Table 4 · p. 16

E.2 IMPLEMENTATION DETAILS FOR THE PILOT STUDY

  • The pilot fine-tuned Qwen3-4B-Thinking-2507 with standard SFT on OR-Instruct-3K, a non-reflective dataset of 3,000 problem-solution pairs.The training used the same hyperparameters as the main SFT stage.Source: §E.2 · p. 16

E.3 IMPLEMENTATION DETAILS FOR THE CALM & STORM FRAMEWORK

The implementation uses a bounded hinting loop to curate SFT trajectories, then applies supervised fine-tuning followed by reinforcement learning.

CALM Data Curation

  • CALM data curation uses Qwen3-4B-Thinking-2507 as Reasoner and Gemini-2.5-Pro as Intervener, with at most five interventions per problem.Trajectories still flawed after five interventions are discarded from the final SFT dataset.Source: §E.3 · p. 17

Stage 1: Supervised Fine-Tuning (SFT)

  • The SFT stage trains the base Qwen3-4B-Thinking-2507 model on 112 CALM-curated “golden” trajectories using AdamW.Training uses DeepSpeed Stage 3 with bf16 precision.Source: §E.3 · p. 17

Stage 2: Reinforcement Learning (RL)

  • The RL stage starts from the final SFT checkpoint and uses Group Relative Policy Optimization (GRPO) through the Verl framework.The experiments ran on four nodes, each equipped with eight NVIDIA H800 GPUs.Source: §E.3 · pp. 16–17

E.4 IMPLEMENTATION DETAILS FOR THE CONTROLLED EXPERIMENT

The controlled experiment isolates SFT data quality by comparing RL with CALM against a control that differs only in its SFT data. The control uses unguided trajectories and otherwise matches the main model’s training setup.

  • The comparison was designed to isolate SFT data quality, with the control matching the main model in every aspect except its SFT data.Source: §E.4 · p. 17
  • The control model was fine-tuned on 112 original, unguided reasoning trajectories for the same problems as the main model’s SFT stage.Source: §E.4 · p. 17
  • Control-model SFT and RL hyperparameters matched the main model’s, while this RL comparison ran for 30 epochs because of computational resource constraints.Source: §E.4 · p. 17, Table 6 · p. 18

F DETAILED BREAKDOWN OF FLAW FREQUENCY EVOLUTION

Across the per-benchmark breakdown, SFT reduces both flaw categories and RL further reduces the remaining flaws. The stages work synergistically, with RL bringing flaw rates to their lowest levels.

G COMPARISON OF REASONING PARADIGM

A parking-task example contrasts single-step generation, which fails to detect a constraint error, with reflective generation, which uses solver feedback to correct it. A separate example shows guided reasoning employing the solver successfully.

G COMPARISON OF REASONING PARADIGM

H INTERVENTION PROCESS FOR SPECIFIC FLAWS

  • In the Code Utilization Distrust example, unguided reasoning fails by avoiding the solver, while reasoning guided by CALM uses it to reach the correct solution.Source: Figure 10 · p. 21

I ILLUSTRATIVE CASE STUDY OF THE CALM FRAMEWORK

The case study illustrates CALM’s workflow, showing how targeted hints address successive reasoning failures and specifying the roles of its reasoning components. It also evaluates automated flaw quantification by comparing LLM labels with expert labels on test responses.

I ILLUSTRATIVE CASE STUDY OF THE CALM FRAMEWORK

  • A multi-turn Reasoner–Intervener exchange corrects an ILP misclassified as an LP, then a code-trust issue, yielding an expert-aligned solution without further intervention.The case illustrates how lightweight hints address distinct failure modes in sequence.Source: I ILLUSTRATIVE CASE STUDY OF THE CALM… · p. 21, Figure 11 · p. 21

I.1 PROMPT TEMPLATES

  • The framework relies on carefully designed prompts for both the Reasoner’s initial task and the Intervener’s supervisory role.Source: §I.1 · p. 22

Prompt for Reasoner

  • The Reasoner prompt requires a plan before code, complete Python scripts using recommended tools, printed solution results, and a single numerical answer in boxed format.Source: Prompt for Reasoner · pp. 22–23

Prompt for Intervener

  • The Intervener compares responses with an iterative OR-expert workflow and, at the first deviation, inserts a focused hint that guides the next action.Its prompt directs it to replace the relevant response text, or return NO_INTERVENTION when no deviation is found.Source: Prompt for Intervener · pp. 23–24

J FLAW QUANTIFICATION OF NATIVE LRMS

  • Gemini-2.5-Pro statically classified completed LRM responses across thousands of outputs, separately from CALM’s interactive, real-time hinting role.The classifier measured flaws using seven predefined categories; it did not interact with the LRM.Source: J FLAW QUANTIFICATION OF NATIVE LRMS · p. 26

Prompt for Flaw Classification

Related papers

Loading 2510.04204v1…