PaperlayerView exact revision on arXiv

Source-linked AI summary

Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling

Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, Yinyu Ye

arXiv:2505.11792v3cs.AI

TL;DR

LLMs still struggle to generate formally correct, usable optimization models from natural-language descriptions. SIRL uses optimization solvers to verify generated code and models, supplying feedback for reinforcement learning and data synthesis. Experiments report strong benchmark performance, including a 32B model with a higher Macro Average than much larger baselines.

  • Problem

    LLMs can struggle to generate formally correct and usable optimization models from natural-language problem descriptions.

    Source: §1 · p. 2

  • Method

    SIRL uses optimization solvers to verify generated code and instance-level models, providing feedback for reinforcement learning and supporting instance-enhanced self-consistency data synthesis.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 7

  • Results

    The SIRL-trained 32B model achieved a higher Macro Average than the 671B Deepseek-V3.1 and the DeepSeek-R1 and OpenAI-o3 reasoning models.

    Source: §4.1 · p. 9

  • Takeaways & Limitations

    Solver-derived verification signals support both RL training and data synthesis for generating accurate, well-formed optimization models.

    Source: §5 · p. 10

    The benchmark review ignores variable-type errors when time-like decision variables can reasonably be treated as integer or continuous.

    Source: §7 · p. 22

Abstract

from arXiv · show

Optimization modeling is fundamental to decision-making across diverse domains. Despite progress in automating optimization formulation from natural language descriptions, Large Language Models (LLMs) often struggle to generate formally correct and usable models against hallucinations, posing a challenge for reliable automation. Inspired by the success of Reinforcement Learning (RL) in enhancing Large Reasoning Models, we present Solver-Informed Reinforcement Learning (SIRL), a novel framework that significantly improves the authenticity of LLMs for optimization modeling using Reinforcement Learning with Verifiable Reward by leveraging external optimization solvers as verifiers. These verifiers automatically assess the executable code and the instance-level mathematical model represented by the associated LP file, yielding precise and comprehensive feedback signals -- including syntax, feasibility, and solution quality, serving as direct rewards for the RL process. This automated verification process, particularly from classic optimization solvers, also underpins our instance-enhanced self-consistency method to synthesize high-quality training data. Extensive experiments on diverse public benchmarks demonstrate that SIRL achieves state-of-the-art performance, substantially outperforming existing methods in generating accurate and executable optimization models. Our code is publicly available at https://github.com/Cardinal-Operations/SIRL.

1 Introduction

Optimization modeling remains difficult to automate because translating natural-language problems into precise, executable models requires substantial expertise. The introduction positions solver-verified reinforcement learning as a way to provide direct correctness feedback, building on work in LLM modeling, synthetic data, and verifiable rewards.

1 Introduction

  • LLMs can assist with model generation, but prompt- and agent-based approaches do not adapt model parameters and can depend on prompt design and the foundation model.Offline learning from demonstrations or preferences also struggles to guarantee functional correctness or feasibility.Source: §1 · p. 2
  • Optimization-model outputs can be checked by solvers for syntax, feasibility, and objective-value agreement, providing direct reward signals for reinforcement learning.The authors describe the modeling process as a multistep sequence of reasoning, formulation, and code implementation.Source: §1 · p. 2
  • SIRL uses solver outputs and statistics from the generated .lp file to assess both outcome correctness and the instance-level mathematical model.The .lp representation supports a more process-aware reward than outcome-only feedback.Source: §1 · p. 2

2 Related work

  • Prior work on data synthesis addresses correctness through methods including LLM-as-a-judge, rejection sampling, and self-consistency.These methods target formal adherence in synthesized data.Source: §2 · p. 3
  • The paper positions classical optimization solvers as domain-specific objective oracles that can support both training-data synthesis and RLVR feedback.This is framed as analogous to using Lean for proofs and compilers for code.Source: §2 · p. 4

3 Method

The method combines solver-checked data synthesis with reinforcement learning that rewards verifiable optimization outputs. Its pipeline uses instance-level model information to select training answers, structures generated responses into reasoning, modeling, and code, and applies staged rewards with selective KL regularization.

3 Method

  • The method begins with a dataset of problem descriptions and ground-truth optimal values, then maximizes expected verifiable reward using the policy.The reward evaluates the derived objective value against the ground truth.Source: §3 · p. 4

3.1 Data synthesis framework

  • Data synthesis generates and validates augmented problems, then executes generated code to obtain optimal values, feasibility status, and instance-level LP files.Ten LLM roles generate answers, with reflection and refinement addressing execution errors and infeasible solutions; pairs easy for the Qwen-32B-Instruct baseline are filtered out.Source: §3.1 · pp. 4–5
  • Instance-enhanced self-consistency scores role responses by agreement on objective value, optimization direction, and binary and integer variable counts.The selected response supplies the definitive answer, using structural information from the LP file alongside final objective values.Source: §3.1 · pp. 5–6

3.2 SIRL: Solver-Informed Reinforcement Learning

  • SIRL structures each response into reasoning, a mathematical model, and executable code, with code execution deterministically producing the final value.Partial KL omits the penalty for reasoning tokens but applies it to model and code segments, supporting exploration while retaining constraints on those outputs.Source: §3.2 · pp. 6–7

3.3 Reward design and training scheme

  • SIRL uses solver verification and a two-stage reward curriculum to train LLMs for optimization modeling.Stage 1 rewards format, execution, and accuracy; Stage 2 adds a bonus for correct solutions that use advanced modeling strategies.Source: §3.3 · p. 8

4 Experiments

Experiments evaluate SIRL on four optimization benchmarks, alongside instance-enhanced self-consistency and ablations of its surrogate and reward designs. The reported results favor SIRL over existing models, show gains from instance-level voting information, and identify trade-offs between reward stages.

4.1 Main results

  • SIRL-Qwen2.5-7B outperforms existing 7B and 14B offline-learning models and OptiMUS, while SIRL-Qwen2.5-32B leads across evaluated benchmarks and exceeds larger models in Macro Average.Evaluation uses pass@1 on NL4OPT, MAMO, IndustryOR, and OptMATH; Table 1 presents the comparison.Source: Table 1 · p. 8, §4.1 · pp. 8–9

4.2 Instance-enhanced self-consistency

  • Majority-vote self-consistency beats single-pass pass@1, and adding optimization direction and variable counts to objective-value voting improves selection of correct solutions.The instance-enhanced method uses structural information from generated models in its consensus mechanism.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 Ablation Study

  • The ablations examine how surrogate-function design and the two-stage reward mechanism affect performance.The studies compare alternative KL regularization designs and training with either reward stage alone against the integrated reward.Source: §4.3 · p. 9, §4.3.1 · p. 10, §4.3.2 · p. 10

4.3.1 Ablation study on different surrogate function designs.

  • Partial KL performs best across the three challenging datasets, while removing KL sharply reduces execution rate; selective penalties preserve reasoning diversity while improving execution.Partial KL applies KL regularization selectively to mathematical formulation and code segments.Source: Table 3 · p. 9, §4.3.1 · p. 10

4.3.2 Ablation study on reward design.

  • Stage-1-only reward is comparatively strong on NL4OPT, stage-2-only reward performs best on OptMATH, and the integrated reward is superior on most benchmarks except OptMATH.The reported trade-off is between foundational skills on simpler tasks and advanced strategies on the most challenging dataset.Source: §4.3.2 · p. 10, Table 4 · p. 10

5 Conclusion

The conclusion presents SIRL as an RLVR framework for generating authentic optimization models, combining Partial KL with solver-based two-stage rewards. Experiments show improved model quality, while reward hacking and weaker performance on difficult benchmarks remain challenges.

5 Conclusion

  • SIRL combines Partial KL with a two-stage reward system that uses optimization solvers for automated verification.The conclusion identifies these as the framework’s core contributions.Source: §5 · p. 10
  • Experiments report that SIRL-trained models generate more accurate, well-formed optimization models than existing methods.Source: §5 · p. 10
  • The authors identify reward hacking and limited performance on IndustryOR and OptMATH as remaining challenges.Future work is directed toward systematic error analysis and targeted improvements.Source: §5 · p. 10

6 Technical Background for Automated Optimization Modeling

This section introduces the components that connect a natural-language problem to a solver-ready optimization model. A production-mix example traces the process from a problem description through mathematical formulation and Python code to a solver result and an LP artifact.

6.1 Core Components

  • The pipeline translates an unstructured decision problem into a mathematical model, implements it through a modeling library, and uses a solver to obtain a solution.A model specifies decision variables, an objective, and constraints; the solver receives this formal model and returns a solution if one exists.Source: §6.1 · p. 18
  • An LP file is a standardized, solver-independent representation of a model, typically containing its objective, constraints, bounds, and variable types.The section presents the LP file as an artifact for verifying the structure of a generated model.Source: §6.1 · p. 18

6.2 A Concrete Example

  • In the production-mix example, the model maximizes profit from tables and chairs subject to labor, wood, nonnegativity, and integer-variable requirements.The Python code implements the model using GurobiPy, writes an LP file, and invokes the solver.Source: §6.2 · pp. 19–20
  • The example solver reports an optimal solution with objective value 750.0, producing 0 tables and 75 chairs.The LP artifact and numerical solver output are presented as complementary feedback signals in the framework.Source: §6.2 · pp. 19–20

7 Benchmark dataset

The evaluation uses four public optimization-modeling benchmarks, with dataset descriptions spanning LP through nonlinear and other formulations. The authors reviewed and corrected benchmark data and describe their evaluation metric and treatment of variable-type errors.

Benchmark selection and curation

  • The evaluation covers NL4OPT, MAMO, IndustryOR, and OptMATH, and the authors report reviewing and correcting test-set errors, especially in NL4OPT and IndustryOR.They say the corrections addressed contradictory or incomplete conditions, flawed logical premises, and incorrect ground-truth answers.Source: §7 · pp. 20–21

Benchmark characteristics

  • NL4OPT has 245 filtered instances; MAMO contains 652 EasyLP and 211 ComplexLP instances; IndustryOR has 100 scenarios; main results use OptMATH's 193-instance variant.The benchmarks vary in scope, including LP, MILP, IP, MIP, NLP, and SOCP tasks.Source: §7 · p. 21

Evaluation metric

  • The authors assess correctness using a strict accuracy metric based on the difference between predicted and ground-truth objective values.The supplied passage does not state the formula's threshold.Source: §7 · p. 21

Evaluation assumption

  • Variable-type errors are ignored, with the authors noting that reviewed cases typically involve quantities such as time that may reasonably be integer or continuous.This evaluation choice follows prior work [26].Source: §7 · p. 22

8 Seed data

The seed data comprise real-world industry cases collected from operations-research textbooks and industry scenarios. The section illustrates this material with a restaurant seating-allocation problem and its stated formulation and answer.

Seed data

  • The seed data comprise 686 real-world industry cases collected from operations-research textbooks and a list of 100 industry scenarios [25].Source: §8 · p. 22

Example problem

  • The example allocates restaurant seats across two periods, with a 100-seat capacity and period-specific demand bounds.Its stated objective combines customer satisfaction and revenue, assigning each a weight of 1.0 per allocated seat.Source: §8 · p. 22

Example formulation

  • The example formulation gives an expected optimal answer of 260.0 for the seat-allocation problem.It uses period-wise allocation variables and capacity and demand constraints.Source: §8 · p. 22

9 Prompt templates

The appendix describes prompt templates for candidate generation, problem validation, error recovery, and infeasibility repair. Its system prompt structures model generation into analysis, mathematical formulation, and solver-code steps.

9 Prompt templates

  • The appendix collects prompts used across the framework's modeling and validation procedures.Source: §9 · p. 23

9.1 Ten roles for self-consistency

  • Self-consistency generates multiple candidate solutions by prompting the LLM with ten expert roles emphasizing operations research, Python, and Gurobi expertise.Source: §9.1 · p. 23

9.2 LLM-as-a-judge for generated problem evaluation

  • The LLM-as-a-judge prompt checks generated problems for operations-research structure, practical relevance, and semantic consistency.It asks for a step-by-step analysis and a yes-or-no judgment.Source: §9.2 · pp. 23–24

9.3 Refine and regenerate for the error case

  • For code execution errors, the refinement prompt asks the LLM to diagnose the cause and return corrected Gurobi code.It supplies the problem, previous code, and execution error as inputs.Source: §9.3 · p. 24

9.4 Refine and regenerate for the infeasible case

  • For infeasible solver outcomes, the prompt requests a revised mathematical model as well as corrected code.It asks the LLM to analyze the cause of infeasibility and potential pitfalls before regenerating both artifacts.Source: §9.4 · pp. 24–25

9.5 System prompt and the LLM responses

  • The system prompt requires separate think, model, and python tags for sequential problem analysis, formulation, and code generation.It follows DeepSeek-R1-Zero [29] by specifying this output format.Source: §9.5 · p. 25
  • The tire-transport example applies the format to minimize trips subject to tire demand, budget, and a limit on plane trips relative to truck trips.The prompt output defines integer trip variables and implements the model in Gurobi code.Source: §9.5 · pp. 26–27

10 Reward function design

The reward uses two stages to assess response structure, code execution, answer accuracy, and—at the second stage—use of advanced modeling techniques.

Reward stages

  • Stage 1 combines format, execution, and accuracy rewards, while Stage 2 adds a bonus for advanced modeling techniques.The bonus is awarded only when the answer is correct and the generated model uses detectable techniques such as binary variables or nonlinear formulations.Source: §10 · p. 28

Format reward

  • The format reward is binary and requires all response tags to appear in the system-prompt order; it also supports extracting and evaluating the model and code.The required tags separate reasoning, model, and executable code.Source: §10 · p. 28

Execution reward

  • The execution reward is 1 when the optimization code runs and 0 otherwise.Source: §10 · p. 28

Accuracy reward

  • The accuracy reward checks the executed answer against ground truth within a 0.01 tolerance, giving 2 for a correct answer and 0 otherwise in Stage 1.Source: §10 · p. 28

11 Details of experiments

Experiments use Qwen2.5-7B-Instruct with a filtered 10,000-instance training set and a two-stage RL setup.

Training setup

  • The two-stage training required about 24 hours per stage on eight 80GB H100 GPUs, totaling 384 GPU hours.Source: §11 · p. 28

Training data

  • Filtering removed instances solved by Qwen-32B-Instruct in at least 8 of 10 attempts, leaving about 70,000 samples from which 10,000 were randomly selected.Source: §11 · p. 28

Training and decoding

  • Training adapted Verl for Qwen2.5-7B-Instruct with a Partial KL strategy and two-stage rewards, while top-P decoding was used in training and inference.Source: §11 · p. 29

12 Further Comparative Analysis

SIRL outperforms the base model and SFT across benchmarks, while error analyses and stochastic sampling results further examine its performance.

Sampling-based performance

  • Under top-p sampling at temperature 0.5 and top-p 0.95, SIRL shows low variance and consistent performance across runs.Table 8 reports mean pass@1 accuracy and standard deviation.Source: §12 · p. 29, Table 8 · p. 29

Performance without external solvers

  • Without solver assistance, textual step-by-step reasoning achieves extremely low performance across most benchmarks, as reported by pass@1 accuracy.Source: §12 · pp. 29–30, Table 9 · p. 30

Comparison with SFT

  • SIRL’s RL method outperforms both the base model and SFT across all benchmarks, with gains of 24.1% over SFT on MAMO Complex and 8.2% on OptMATH.Source: §12 · p. 30, Table 10 · p. 30

Error type analysis

13 In-depth analysis of the Partial KL strategy

The analysis compares surrogate designs and uses case studies to examine how KL choices relate to execution and reasoning on optimization tasks.

Surrogate-function ablation

  • The Without KL design has a significantly lower execution rate than the other two surrogate designs in the all-benchmark ablation.Source: §13 · p. 31, Table 13 · p. 31

Case Study 1

  • The IndustryOR case study shows a Without KL response failing at execution after attempting an unsupported comparison between a decision variable and an integer.Source: §13 · p. 31

Case Study 2

  • For logical constraints, the case study reports that omitting the KL penalty encourages reasoning exploration, while the Partial KL strategy is presented as improving understanding of such questions.Source: §13 · p. 31

Related papers

Loading 2505.11792v3…