PaperlayerView exact revision on arXiv

Source-linked AI summary

Step-Opt: Boosting Optimization Modeling in LLMs through Iterative Data Synthesis and Structured Validation

Yang Wu, Yifan Zhang, Yurong Wu, Yuran Wang, Junkai Zhang, Jian Cheng

arXiv:2506.17637v1cs.CLcs.LG

TL;DR

Natural-language optimization modeling is labor-intensive and requires domain expertise, while existing training-data approaches face reliability and post-processing challenges. Step-Opt-Instruct uses iterative problem generation and stepwise validation to produce fine-tuning data, and Step-Opt outperforms existing approaches across the reported benchmarks, particularly on complex components.

  • Problem

    Translating natural-language descriptions into mathematical models is labor-intensive and requires domain-specific expertise and modeling knowledge.

    Source: §1 · p. 1

  • Method

    Step-Opt-Instruct combines iterative problem generation with real-time stepwise validation to create high-quality fine-tuning data for OR modeling.

    Source: §1 · pp. 1–2

  • Results

    Step-Opt outperforms existing approaches across NL4OPT, MAMO, and IndustryOR, with a 17.01% improvement in micro average on more complex components.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The reported results highlight Step-Opt-Instruct’s effectiveness in combining gradual problem refinement with data validation for complex OR tasks.

    Source: §5 · p. 8

    The method has difficulty covering the wide variety of OR modeling techniques, and its performance has not been fully tested across all OR problem types or other fields.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have revolutionized various domains but encounter substantial challenges in tackling optimization modeling tasks for Operations Research (OR), particularly when dealing with complex problem. In this work, we propose Step-Opt-Instruct, a framework that augments existing datasets and generates high-quality fine-tuning data tailored to optimization modeling. Step-Opt-Instruct employs iterative problem generation to systematically increase problem complexity and stepwise validation to rigorously verify data, preventing error propagation and ensuring the quality of the generated dataset. Leveraging this framework, we fine-tune open-source LLMs, including LLaMA-3-8B and Mistral-7B, to develop Step-Opt--a model that achieves state-of-the-art performance on benchmarks such as NL4OPT, MAMO, and IndustryOR. Extensive experiments demonstrate the superior performance of Step-Opt, especially in addressing complex OR tasks, with a notable 17.01\% improvement in micro average accuracy on difficult problems. These findings highlight the effectiveness of combining structured validation with gradual problem refinement to advance the automation of decision-making processes using LLMs.The code and dataset are available at https://github.com/samwu-learn/Step.

1 Introduction

Optimization modeling is difficult because translating natural-language problems into mathematical models requires substantial effort and domain expertise. Step-Opt-Instruct combines iterative problem generation with stepwise validation to produce training data for OR modeling, and Step-Opt reports stronger benchmark results, especially on complex problems.

1 Introduction

  • Turning natural-language descriptions into mathematical models is labor-intensive and requires domain expertise and modeling knowledge, limiting OR’s broader real-world application.The paper frames OR as a discipline for complex decision-making, with applications including economics, engineering, and computer science.Source: §1 · p. 1
  • Existing LLM approaches use prompting or multi-agent systems, while ORLM fine-tunes on industry cases but still requires substantial manual post-processing and lacks precise control of complexity and diversity.The introduction also identifies delayed error detection and rule-based post-processing’s difficulty with deeper logical and structural issues.Source: §1 · p. 1
  • Step-Opt-Instruct combines iterative problem generation and stepwise validation to create OR fine-tuning data, using feedback and real-time updates to reduce error propagation and manual post-processing.The framework is designed for open-source LLMs; the resulting Step-Opt model is reported to achieve state-of-the-art performance across several benchmarks.Source: §1 · pp. 2–3
  • Complexity-Evolving modifies constraints, parameters, or objectives, while Scope-Evolving transforms domains or combines problems to expand diversity and breadth.The two generator types are distinguished by red and blue backgrounds, respectively.Source: §1 · p. 2, Figure 1 · p. 2
  • Stepwise validation checks description completeness, variables, constraints, and programs, then uses feedback loops to address detected issues.Specially designed prompts guide step-by-step verification of techniques such as the Big-M method.Source: §1 · p. 2
  • 17.01% micro and 12.26% macro average improvements on complex components accompany 6.07% micro and 7.93% macro average improvements across the benchmarks.Step-Opt is reported to outperform existing approaches on NL4OPT, MAMO, and IndustryOR; the benchmarks also received manual corrections to examples with error labels.Source: §1 · p. 3

2 Related Work

Prior work applies prompting, multi-agent systems, and fine-tuning to automate OR modeling, but the paper identifies remaining challenges in complex problems, data filtering, and error propagation. It positions iterative generation and real-time validation as a response to those gaps.

2 Related Work

  • Prompt-engineering and fine-tuning are the two broad categories of LLM-based automated OR modeling; multi-agent methods face difficulties with complex problems, while ORLM uses industry data and rule-based post-processing.The paper says ORLM lacks precise prompting and effective filtering mechanisms.Source: §2 · p. 3
  • Prior OR data-augmentation work synthesizes problems from natural-language descriptions, while ORLM expands industry cases and ReSocratic generates scenarios from solutions.The paper identifies Evol-Instruct as its closest work, then distinguishes its approach by adding stepwise validation for accuracy and error-propagation control.Source: §2 · p. 3

3 Method

Step-Opt-Instruct combines iterative problem generation with stepwise validation to build a dataset of verified optimization problems, models, and programs. Its generators expand problem complexity and scope, while checkers identify and correct errors before data enter the dataset.

3.1 Preliminary

  • The initial dataset consists of problem descriptions paired with mathematical models and programs; descriptions must specify an objective, constraints, and relevant numerical parameters.The paired model implements the stated objective and constraints and produces executable code.Source: §3.1 · p. 3

3.2 Generators

  • Each iteration samples a seed problem and model, applies an evolving method to create a new description, then generates its model and program using the seed and method as references.The solution generator is prompted to keep the new model and program's format and structure consistent with the seed.Source: §3.2 · pp. 4–5
  • Complexity-Evolving modifies constraints, objectives, or parameters while preserving problem structure, with changes limited to one constraint or objective or at most one new entity per iteration.These limits manage complexity and exclude examples considered excessively challenging.Source: §3.2 · p. 4
  • Scope-Evolving transfers a seed problem to another domain or combines it with a second problem to broaden topic coverage and linguistic and contextual diversity.Transferred problems preserve the original problem's logic and constraints; combined problems are kept similar in length to an original.Source: §3.2 · pp. 4–5

3.3 Stepwise Validation Mechanism

  • Stepwise validation checks description completeness before solution generation, then checks variables, constraints, and program quality; failed solutions are revised or discarded at the retry limit.Only data passing all assessments enter the dataset, reducing error propagation across later generations.Source: §3.3 · p. 5

4 Experiment

Across the benchmarks, Step-Opt-LLaMA-3-8B achieves state-of-the-art results, with a particularly strong advantage on complex problems. Ablations examine the evolving methods, training-example components, and comparison with OR-Instruct.

4.1 Experimental Setup

  • Evaluation spans NL4OPT, MAMO EasyLP and ComplexLP, and IndustryOR; manual revisions corrected 16, 78, and 50 instances, respectively, and 23 IndustryOR instances were removed.The study also reports correcting 16 NL4OPT inaccuracies and 78 MAMO inaccuracies.Source: §4.1 · p. 5
  • Step-Opt is fine-tuned on LLaMA-3-8B and Mistral-7B, with problem descriptions as input and mathematical models plus programs as output.The reported inference setup uses greedy decoding in a zero-shot context.Source: §4.1 · p. 6
  • The evaluation comparison rule allows minor numerical discrepancies between generated-program outputs and ground-truth solutions.The rule is introduced as an equivalence criterion for outputs and ground truth.Source: §4.1 · p. 6

4.2 Comparison Analysis

  • 21.33% is the reported advantage of Step-Opt-LLaMA-3-8B on MAMO ComplexLP; Step-Opt-LLaMA-3-8B achieves state-of-the-art results across all benchmarks and exceeds 50% accuracy on complex datasets.Fine-tuned models outperform prompt-engineering methods on average, while differences are less pronounced on the easier datasets.Source: §4.2 · pp. 6–7, Table 1 · p. 6

4.3 Ablation Study

  • Removing domain transformation yields the worst performance across datasets; constraint modification and objective alteration have greater influence on complex than easy datasets.The study reports survival counts of 1,716 for constraint modification, 1,242 for objective alteration, 2,123 for parameter adjustment, 2,077 for domain transformation, and 455 for combination.Source: §4.3 · p. 7, Table 2 · p. 7
  • Including mathematical models improves performance even when datasets are matched at 4.73 million tokens, and the authors describe the models as structured guidance between problem descriptions and code.The ablation removes the mathematical model while retaining the programs used for problem-solving.Source: §4.3 · pp. 7–8, Table 4 · p. 8
  • With 3,000 examples, Step-Opt-Instruct outperforms ORLM except on MAMO EasyLP, improving micro average by 1.61% and macro average by 5.11%.The reported gains are more pronounced on MAMO ComplexLP and IndustryOR.Source: Table 3 · p. 7, §4.3 · p. 8

5 Conclusion

Step-Opt-Instruct combines iterative problem generation and stepwise validation to produce high-quality fine-tuning data, and Step-Opt improves performance across three OR benchmarks, especially on complex tasks. The authors note that the method does not cover the full variety of OR techniques and has not been fully tested across OR problem types or other fields.

5 Conclusion

  • Step-Opt improves performance across NL4OPT, MAMO, and IndustryOR, particularly on complex optimization tasks.The conclusion attributes the gains to combining systematic problem generation with structured validation.Source: §5 · p. 8
  • Step-Opt-Instruct combines iterative problem generation with stepwise validation to remove low-quality data during generation and prevent error propagation.The framework automates data generation without post-processing.Source: §5 · p. 8
  • The method struggles with the variety of OR modeling techniques, limiting its coverage of possible scenarios.Its performance has not been fully tested across all OR problem types, and applicability in other fields remains unvalidated.Source: §5 · p. 9

A.1 Example for training data

COPT is the default solver used in the experiments.

A.1 Example for training data

A.2 Examples for modifications of test sets

The appendix illustrates test-set cases where modeling errors change reported solutions, including failures to enforce subtour elimination or both sides of an absolute-value constraint. It also shows an LP answer adjusted for integer-valued decisions and a problem lacking numerical parameter values.

A.2 Examples for modifications of test sets

  • The macro-counting example's continuous-variable answer is adjusted from 430.7692307692307 to 460 because the number of bowls must be integral.The example asks for a sodium-minimizing meal mix subject to calorie, protein, and egg-share requirements.Source: §A.2 · p. 11
  • The retail allocation example's initial cost is incorrect because it retains only one side of the absolute-value constraint; considering both sides yields a cost of 800.The example is labeled as incorrect handling of an absolute-value constraint.Source: §A.2 · p. 11
  • The Traveling Salesman Problem example changes from an incorrect cost of 50 units to a correct minimum of 127 units when subtour elimination is enforced.The appendix labels the case as incorrect handling of subtour elimination.Source: §A.2 · p. 11
  • The fighter-jet allocation example has no numerical values for its parameters.The problem asks how to allocate yearly production between combat use and pilot training.Source: §A.2 · p. 11

A.3 Prompt Templates for Complexity-Evolving

The Complexity-Evolving prompt templates generate new problems by changing objectives, parameters, or constraints while retaining aspects of the original problem's structure. Each template asks for one new problem and not its solution.

A.3 Prompt Templates for Complexity-Evolving

  • The objective-alteration prompt permits changing the objective or adding one for a multi-objective problem, but rejects coefficient-only changes.It asks that generated problems fit real-world scenarios and requests one problem without a solution.Source: §A.3 · p. 13
  • The template supplies two example placeholders and an original-problem-description placeholder before the answer field.The supplied format is shown for the Complexity-Evolving prompts.Source: §A.3 · p. 13
  • The parameter-adjustment prompt changes parameter values or adds at most one new entity while retaining the problem's logical structure, constraints, and objective.It requests one new problem aligned with a real-world scenario, without solving it.Source: §A.3 · p. 13
  • The constraint-modification prompt modifies or adds at most one constraint while preserving the problem's logical structure.It asks for a single real-world-aligned problem and not a solution.Source: §A.3 · p. 13

A.4 Prompt Templates for Scope-Evolving

The Scope-Evolving prompts generate new OR problems by changing their application domain while retaining logical structure and constraints. They also support combining two input problems into one new, distinct-domain problem.

A.4 Prompt Templates for Scope-Evolving

  • The section supplies separate prompt templates for domain transformation and problem combination.Both are framed as combinatorial optimization modeling tasks.Source: §A.4 · p. 14
  • The domain-transformation prompt changes an input problem’s application domain while preserving its logical structure and constraints.It asks for one new problem without a solution.Source: §A.4 · p. 14
  • The combination prompt merges two input problems into one real-world problem in a different domain with distinct details.The output should be similar in length to one of the original problems and should not include a solution.Source: §A.4 · p. 14

A.5 Prompt Templates for checkers and regeneration

The appendix details regeneration and validation prompts, generation costs, training setup, and results on MAMO ComplexLP. Step-Opt reaches the highest reported accuracy in the comparison, while validation results vary across GPT models.

A.5 Prompt Templates for checkers and regeneration

  • The prompts regenerate erroneous solution outputs and problem descriptions using the detected error and examples as guidance.The regenerated content is asked to retain the original format and structure while making necessary corrections.Source: §A.5 · pp. 14–15
  • Constraint validation checks model constraints against the problem description and code, including specified Big-M cases and model-to-program consistency.The checker is instructed to use common sense without assumptions or conjectures.Source: §A.5 · p. 15

A.6 Performance Comparison on MAMO ComplexLP

  • Step-Opt-LLaMA-3-8B achieves the highest reported accuracy, 61.61%, surpassing GPT-4o and other baselines; Step-Opt-Mistral-7B also achieves 52.61%.Source: §A.6 · p. 16
  • Step-Opt’s task-specific data is presented as supporting more reliable and effective solutions for real-world optimization challenges.Source: §A.6 · p. 16

A.7 Additional Details on Instance Generation

  • 8,400 generations yielded 4,464 samples, with 46.86% of generated samples discarded.The process involved 64K queries and 179M tokens.Source: §A.7 · p. 16

A.8 Hyper-parameters for Training Step-Opt and baselines

  • Experiments used one server with eight 40GB A100 GPUs, reported the best checkpoint results, and capped the maximum token length at 2,500.Source: §A.8 · p. 16

A.9 Study on Stepwise Validation Mechanism

  • On MAMO Complex, stepwise validation reaches 42.18% accuracy with GPT-4, while CoE remains superior for GPT-3.5 and GPT-4o.The reported comparison is against Standard, CoT, and Reflexion methods.Source: §A.9 · p. 16, Table 5 · p. 17
  • During data generation, validation can reuse the original solution and focus on changed components, unlike testing tasks solved from scratch.The authors identify this as an advantage for generating high-quality data, not for test-time solving.Source: §A.9 · p. 16

Related papers

Loading 2506.17637v1…