PaperlayerView exact revision on arXiv

Source-linked AI summary

ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling

Chenyu Huang, Zhengyang Tang, Shixi Hu, Ruoqing Jiang, Xin Zheng, Dongdong Ge, Benyou Wang, Zizhuo Wang

arXiv:2405.17743v5cs.CLcs.AIcs.CEcs.LG

TL;DR

Optimization modeling remains difficult to automate, while existing approaches face gaps in complexity, scope, and data quality. The paper introduces a customizable semi-automated synthesis framework for training open-source ORLMs, which achieve competitive results across NL4OPT, MAMO, and IndustryOR.

  • Problem

    Existing optimization-modeling approaches are hindered by gaps in problem complexity, scope, and types, limiting the full potential of optimization techniques.

  • Method

    The paper introduces a semi-automated, customizable framework that synthesizes high-quality synthetic data tailored to optimization modeling for training open-source large models.

  • Results

    ORLM achieves competitive results on the NL4OPT, MAMO, and IndustryOR benchmarks.

  • Takeaways & Limitations

    The results support a viable path for training open-source models with enhanced optimization-modeling capabilities.

  • Takeaways & Limitations

    The greedy approach may reduce large-model performance on these tasks, although it ensures the stated procedural property.

Abstract

from arXiv · show

Optimization modeling plays a critical role in the application of Operations Research (OR) tools to address real-world problems, yet they pose challenges and require extensive expertise from OR experts. With the advent of large language models (LLMs), new opportunities have emerged to streamline and automate such task. However, current research predominantly relies on closed-source LLMs such as GPT-4, along with extensive prompt engineering techniques. This reliance stems from the scarcity of high-quality training datasets for optimization modeling, resulting in elevated costs, prolonged processing times, and privacy concerns. To address these challenges, our work is the first to propose a viable path for training open-source LLMs that are capable of optimization modeling and developing solver codes, eventually leading to a superior ability for automating optimization modeling and solving. Particularly, we design the {\sc OR-Instruct}, a semi-automated data synthesis framework for optimization modeling that enables customizable enhancements for specific scenarios or model types. This work also introduces IndustryOR, the first industrial benchmark for evaluating LLMs in solving practical OR problems. We train several 7B-scale open-source LLMs using synthesized data (dubbed ORLMs{https://github.com/Cardinal-Operations/ORLM}), which exhibit significantly enhanced optimization modeling capabilities, achieving competitive performance across the NL4OPT, MAMO, and IndustryOR benchmarks. Additionally, our experiments highlight the potential of scaling law and reinforcement learning to further enhance the performance of ORLMs. The workflows and human-machine interaction paradigms of ORLMs in practical industrial applications are also discussed in the paper.

1. Introduction

Optimization modeling remains difficult to automate because it requires substantial expertise, while existing LLM approaches rely heavily on closed-source models and limited test coverage. The paper introduces OR-Instruct, IndustryOR, and open-source ORLMs to address these gaps and reports competitive benchmark performance.

  • Optimization modeling requires high expertise and substantial labor costs, creating a bottleneck for broader use of optimization techniques.
  • Existing LLM approaches depend on proprietary models and online APIs, limiting privacy protection and customization for sensitive industrial applications.
  • Current evaluation relies mainly on relatively homogeneous datasets with low complexity, narrow scope, and few problem types compared with real-world settings.
  • OR-Instruct is a semi-automated framework that synthesizes high-quality optimization-modeling data to train open-source LLMs for modeling and solving optimization problems.
  • The framework supports customization by introducing new seed data and scenarios to adapt synthesized datasets to particular needs.
  • IndustryOR provides an industrial benchmark, while ORLMs achieve competitive performance across NL4OPT, MAMO, and IndustryOR.
  • The results indicate that scaling model and data size and applying reinforcement learning can further improve ORLM performance, although some challenging datasets remain difficult.
  • The work reports open-source training data, an industry-level benchmark, and best-performing fine-tuned LLMs to support broader operations-research applications.

2. Literature Review

Prior work applies AI to optimization problems, including decision support, algorithm design, and automated modeling. ORLM instead focuses on automating optimization-model construction with fine-tuned open-source LLMs and customizable synthetic data.

  • AI for Operations Research: AI methods have been applied to operations research for resource allocation, approximation, forecasting, and decision-making.
  • Positioning of This Work: This work differs by fine-tuning open-source LLMs for optimization modeling and providing a direct-prompting, privacy-preserving alternative to API calls and multi-agent interaction.
  • Automated Optimization Modeling: Automated optimization modeling bridges natural-language problem descriptions and mathematical optimization formulations, with NL4OPT providing a widely used benchmark.
  • LLM-Based Optimization Systems: Existing systems use LLMs for model construction, solver-code assistance, optimization explanation, or infeasibility diagnosis, often relying on GPT-4 or multi-agent interaction.
  • Synthetic Data and Optimization Modeling: Optimization-focused research includes constraint synthesis, natural-language-to-model generation, and data-driven approaches for MILP and inventory problems.
  • Synthetic Data Generation: OR-Instruct synthesizes general-purpose optimization-modeling data from natural-language descriptions while supporting customized augmentation for specific problems or domains.

3. Semi-Automated Framework for Data Generation

The framework defines optimization modeling as mapping natural-language OR problems to mathematical models and solver programs, then constructs training data through iterative expansion, augmentation, and filtering. OR-Instruct targets coverage, adaptability, linguistic diversity, technique variability, and data quality while allowing domain-focused customization.

  • Task Definition: Optimization modeling maps a natural-language problem p to a mathematical model m and solver program c, forming training examples as (p,m,c).
  • Data Requirements: The training dataset should cover diverse scenarios, problem types, and difficulty levels while supporting environmental, linguistic, and technique variability.
  • Framework Motivation: OR-Instruct addresses scarce and private optimization-modeling data through a semi-automated synthetic-data process tailored to these requirements.
  • OR-Instruct Pipeline: The framework iteratively expands scenarios and question types, augments examples by changing objectives, constraints, wording, and modeling techniques, then filters low-quality data.
  • Customization: Concentrating seed data in a specific scenario or problem type substantially improves modeling within that domain, enabling customized enhancement.
  • Data Quality: Generated examples remained easy-heavy: among 50 reviewed cases, 87% were easy, 13% medium, and none hard.
  • Data Quality: Expansion data were judged approximately 70%–75% correct for code and model quality, while each iteration filtered about 39% of generated data.

4. Results

OR-Instruct synthesizes optimization-modeling data for training open-source approximately 7B-parameter models, which are evaluated against established benchmarks and baselines. ORLMs achieve competitive or state-of-the-art results across multiple benchmarks, though performance varies by domain and task complexity.

  • Data synthesis: OR-Instruct generates training data through expansion, objective or constraint alteration, question rephrasing, and additional information incorporation.The framework uses automatic filtration after repeated expansion and augmentation operations.
  • Model training: The authors train approximately 7B-parameter open-source backbones, including Mistral-7B, Deepseek-Math-7B-Base, and LLaMA-3-8B, using OR-Instruct data.Instruction tuning treats natural-language OR problems as inputs and complete mathematical models and programs as target completions.
  • Benchmark results: ORLM-LLaMA-3 and ORLM-Qwen2.5 achieve state-of-the-art performance across four benchmarks relative to GPT-4 and other open-source LLMs.They surpass standard GPT-4 prompting by 42.2% in micro average and 55.4% in macro average.
  • Benchmark results: ORLMs outperform senior undergraduates on simpler problems and approach expert-level performance, but remain inadequate on complex IndustryOR and MAMO-ComplexLP tasks.The paper reports that complex-task performance remains below the average student.
  • Additional analysis: ORLM-LLaMA-3-8B improves mathematics and coding performance while showing a slight decline in general-domain performance, leaving average performance across domains largely unchanged.The result aligns with OR-Instruct’s emphasis on mathematical modeling and solver-assisted coding.

Appendix A.6.

The appendix evaluates ORLM in practical assistance, targeted training, and scenario-specific modeling. Results indicate measurable gains in solution efficiency and accuracy, while targeted data improves modeling of selected structures.

  • Practical evaluation: ORLM improves solution accuracy by approximately 10%–25% and saves 1.8–2.2 hours when solving optimization-modeling problems.Experts save approximately 1.5–2 hours, while students save around 2–2.5 hours.
  • Practical evaluation: The reported experimental effects are substantial, with absolute Hedge’s g > 1 and statistical power > 0.8.The paper presents these statistics as evidence of a large experimental-control difference and high detection probability.
  • Practical evaluation: Engineers rate ORLM positively, with none describing it as useless or minimally helpful and most reporting significant help or high effectiveness.The authors connect these evaluations to human–AI collaboration in industrial OR.
  • Targeted enhancement: Adding 2,000 MILP-specific training points to a 10,000-case general dataset enables comparison of ORLM performance before and after targeted training.The experiment uses 50 fundamental MILP questions and answers as seed data across iterative rounds.
  • Targeted enhancement: Targeted-trained ORLM correctly models logical relationships using indicator variables and the Big M method, unlike GPT-4 and the general ORLM in the illustrated scenario.The example is presented as evidence of customized enhancement through OR-Instruct.
  • Targeted enhancement: OR-Instruct supports customized improvements across modeling types and shows potential for targeted enhancement within specific industrial sectors.The appendix frames this capability as robust generalization from directionally generated training data.

5. Analysis and Discussion

ORLM-LLaMA-3-8B outperforms Standard-GPT-4 across IndustryOR difficulty levels and several question types. Both systems perform poorly on nonlinear and rare types, where the training data contain few examples and the problems are inherently complex.

  • IndustryOR analysis: ORLM-LLaMA-3-8B surpasses Standard-GPT-4 on linear, integer, and mixed-integer programming question types.The reported comparison concerns IndustryOR accuracy by question type.
  • IndustryOR analysis: Both models perform poorly on nonlinear programming and other rare question types.The paper attributes this partly to limited representation of these types in the seed and training datasets and partly to their inherent complexity and scarcity.

Ablation Study on Data Synthesis Strategy

The ablation compares fine-tuning on seed data alone with fine-tuning on OR-Instruct-generated data derived from the same seed. Generated data provide a significant performance boost except for minor variation on simpler linear-programming cases.

  • Data synthesis ablation: OR-Instruct-generated data significantly improve ORLM performance compared with using only the seed data.The comparison is presented in Table 6 using the same underlying seed data.
  • Data synthesis ablation: The main exception is minor performance variation on simpler MAMO_EasyLP cases.The reported improvement pattern otherwise favors fine-tuning with OR-Instruct-generated data.

Ablation Study on OR-Instruct Augmentations

The ablation study tests three OR-Instruct augmentations using controlled datasets and training conditions. Full augmentation performs best, while removing any augmentation reduces performance across benchmarks.

  • Experimental setup: Four datasets of 3,000 instances each isolate the effects of altering objectives and constraints, rephrasing questions, and adding multiple modeling techniques.Each dataset is trained with LLaMA-3-8B under consistent hyperparameters.
  • Results: 68.6% micro average and 55.7% macro average are achieved with all three OR-Instruct augmentations.This setting is labeled Full Augmentations.
  • Results: Removing any augmentation causes a performance drop across all benchmarks in both micro and macro averages.The ablation compares the full setting against versions missing one augmentation.
  • Results: Rephrasing questions contributes slightly more than the other two augmentations to overall performance.The analysis attributes general performance gains to all three augmentation strategies.
  • Implication: ORLM can be tailored to specific domains within optimization modeling through these customizable data augmentations.The finding supports domain-specific enhancement of the training data.

Ablation Study on Question Types

The section examines how training-data composition, model scale, and inference strategy affect ORLM performance. Mixed problem types and larger models improve generalization, while data scaling shows metric-dependent gains and Pass@8 substantially improves reported accuracy.

  • Question-type ablation: Mixed problem types significantly improve ORLM-LLaMA-3-8B performance across datasets, including challenging MAMO linear-programming tests.The analysis attributes this pattern to broader problem coverage and stronger generalization capacity.
  • Scaling laws: Increasing model size steadily enhances ORLM accuracy and follows a power-law trend, indicating a higher performance ceiling for larger base models.The model-size study evaluates Qwen sizes from 0.5B to 14B using macro and micro averages across four test sets.
  • Scaling laws: Increasing training data raises accuracy overall, but macro performance fluctuates while micro performance increases approximately monotonically with a power-law trend.Simple datasets such as NL4OPT and MAMO-easyLP reach bottlenecks and fluctuate under the macro metric.
  • Inference strategy: Pass@8 increases the overall micro average by 17.09% and the macro average by 24.34% relative to the previous version.Under Pass@8, ORLM surpasses senior undergraduate performance and approaches expert performance on the reported averages.
  • Limitations analysis: ORLM demonstrates approximately a 95% COPT pass rate across more than 1,000 test cases, while modeling remains the primary challenge.The reported proficiency concerns COPT use rather than complete optimization-model construction.
  • Limitations analysis: Low Model Completeness accounts for more than half of observed errors, with 8B models producing overly simplified outputs in complex scenarios.The analysis identifies limited expressive and learning capabilities as the underlying issue.

6. Future Direction

The paper discusses ORLM’s potential use in industry and education. Its open-source, customizable design supports domain-specific modeling assistance, rapid initial solutions, and interaction between business and algorithm teams.

  • Applications: ORLM is presented as a practical application platform for optimization modeling and future industrial use.The section builds on experiments analyzing ORLM’s strengths, limitations, and applications.
  • Deployment: The paper reports that ORLM surpasses cutting-edge proprietary models while supporting privacy and security, and that its 7B size enables private deployment on computers with 16GB of VRAM.These claims frame open-source deployment as relevant to accessibility and commercial use.
  • Industry: Companies with extensive training data can customize ORLM to their domains using OR-Instruct.This supports domain-specific adaptation within enterprise settings.
  • Industry: ORLM retains broad pretrained knowledge while improving optimization modeling, helping it handle industry terminology and communication between business experts and algorithm engineers.The paper connects these capabilities to results reported in Tables 2 and 3.
  • Industry: ORLM can generate an initial solution within seconds for engineers and business teams to review.Its Pass@8 capability also supports generating multiple solutions.
  • Industry: ORLM can modify existing models, add business constraints, create preliminary models, and generate initial solution code for engineer validation.The paper states that this can reduce delivery costs and improve accuracy on subsequent work.
  • Knowledge management: RAG could incorporate internal company documents, support knowledge transfer, and turn project reports into training assets.The paper also discusses using ORLM to train junior algorithm engineers.
  • Education: ORLM is proposed for operations-research education, where instructors can use it to provide reference solutions and guidance to beginners.The section also identifies mathematical modeling competitions as a potential application area.

6.2. Future Research Directions

The future-work section identifies opportunities to improve ORLM’s data, scaling, inference, and ranking capabilities. It emphasizes reducing data costs, adding preference-based supervision, and using reinforcement learning or advanced inference methods.

  • Future directions: The authors identify data synthesis, mining, and exploration as priorities for improving ORLM.These directions respond to limitations in current training data and resource use.
  • Inference and ranking: Reinforcement learning could narrow the gap between Pass@1 and Pass@8 by improving the model’s answer-ranking capability.The proposed rationale is to align the model’s policy more closely with an optimal ranking policy.
  • Inference and ranking: Potential future techniques include RLHF, direct preference optimization, KTO, multi-agent collaboration, and internal chain-of-thought structures.These methods are proposed as ways to improve optimization-modeling performance.
  • Dataset construction: The current data synthesis strategy is insufficient for RLHF because preference optimization requires multiple responses ranked for each question.Actual optimal solutions could also broaden the techniques supported by future datasets.
  • Data mining and exploration: For a fixed model size, additional data produces increasingly marginal gains at later stages, motivating optimized minimal datasets and more efficient synthesis.The paper frames this as a way to balance ORLM performance against training costs.

7. Concluding Remarks

The paper presents OR-Instruct and IndustryOR as a path toward training open-source LLMs for optimization modeling. ORLMs trained on synthesized data achieve competitive performance, while scaling laws and reinforcement learning are identified as promising enhancement directions.

  • Contributions: OR-Instruct is a semi-automated framework for generating optimization-modeling training data tailored to specific needs.The framework is paired with the IndustryOR benchmark, described as the first industrial benchmark of its kind.
  • Contributions: IndustryOR provides an industrial benchmark for evaluating LLMs on practical operations-research problems.The paper introduces it alongside the synthesized-data training framework.
  • Results: Open-source LLMs with approximately 7 billion parameters achieve competitive performance across all test datasets after OR-Instruct training.The reported result concerns optimization-modeling capabilities across the evaluated datasets.
  • Limitations: ORLM currently produces overly simplistic outputs for complex problems and has weak optimal-solution ranking and learning capabilities.These limitations define important remaining challenges for the framework.
  • Future directions: Scaling laws and reinforcement learning are suggested as approaches that could further enhance ORLM and potentially reach expert-level performance.The paper presents these as experimental possibilities rather than established outcomes.
  • Implications: The proposed framework is presented as a starting point for translating the strengths of large models into optimization-modeling capabilities.The paper connects this direction to future industrial applications and subsequent research.

Appendix A: Appendix

The appendix describes a heuristic framework for assessing optimization-modeling difficulty and a prompt-based procedure for synthesizing diverse application problems, models, and solver code. It also reports statistical analyses of ORLM’s effects on solution time and accuracy.

  • Difficulty criteria: The study evaluates modeling difficulty using problem size, logical relationships, natural-language ambiguity, and interdisciplinary knowledge requirements.These criteria collectively frame the difficulty of translating natural-language descriptions into mathematical models.
  • Difficulty assessment: GPT-4 assigns each problem to easy, medium, or hard categories using the proposed criteria and illustrative examples.The categorization is explicitly described as heuristic.
  • Dataset statistics: The statistical analysis samples 100 instances for each of three difficulty levels and reports proportions alongside average variable and constraint counts where parameters are determined.For problems with undetermined parameters, only the proportion is calculated and represented.
  • Scenario expansion: The expansion phase uses prompts to generate specific optimization-modeling scenarios across diverse industries and application domains.Listed domains include logistics, manufacturing, energy, healthcare, transportation, finance, agriculture, and defense.
  • Problem synthesis: A second prompt uses few-shot examples to generate a new real-world optimization problem, mathematical model, and COPT solver code from a selected scenario.The selected scenario is randomly drawn from the first prompt’s generated scenario list.
  • ORLM evaluation: At 95% confidence, ORLM significantly improves solution time and accuracy across total, expert, and student populations.Reported accuracy gains are approximately 10%–25%, while total solving time saved is 1.8-2.2 hours.
Loading 2405.17743v5…