ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling

• The paper identifies modeling errors in logical constraints, scarce high-quality training data, and privacy risks from closed-model APIs as obstacles to adoption.A GPT-4 example illustrates errors involving nonlinear terms and logical relationships.Source: §1 · pp. 2–4
• The authors note that existing evaluation relies heavily on NL4OPT, whose simple linear-programming tasks have limited complexity, scope, and variety relative to real-world problems.Source: §1 · p. 4
• OR-Instruct is a semi-automated framework for synthesizing high-quality optimization-modeling data, with customizable seeds and scenarios for specific needs.It uses iterative bootstrapping to progressively enhance specialized modeling capabilities.Source: §1 · pp. 4–5
• IndustryOR is introduced as an industrial optimization-modeling benchmark spanning 13 industries, five question types, and three difficulty levels.The authors also include NL4OPT and MAMO for comprehensive evaluation.Source: §1 · p. 5
• Trained ORLMs show enhanced optimization-modeling capability, with the best-performing model achieving competitive performance on NL4OPT, MAMO, and IndustryOR.The authors train open-source models of approximately 7B parameters as ORLMs.Source: §1 · p. 5

Introduction

• The authors report that collaboration workflows could improve productivity in industry and academia, and that results, an industry-level benchmark, training samples, and fine-tuned models are open source.Source: §1 · pp. 5–6
• The work is presented as the first initiative to train open-source LLMs specifically for real-world industrial optimization modeling, reducing reliance on closed-source models while improving flexibility and privacy.Source: §1 · p. 6
• The highest-performing ORLM achieves competitive results on NL4OPT, MAMO, and IndustryOR, with roughly 7B parameters outperforming many cutting-edge models and agent-based frameworks.The comparison names GPT-4, Llama-3.1, and Qwen 2.5 among those models.Source: §1 · p. 6
• Ablations validate the approach, but ORLM remains relatively weak on some challenging datasets and has weak ranking capabilities for optimal optimization models.The introduction says scaling model and data size continuously improves performance, while reinforcement learning also strengthens it.Source: §1 · p. 6
• OR-Instruct addresses scarce optimization-modeling training data and supports customization for particular industrial needs and model types.Examples include supply-chain management, scheduling, inventory, linear programming, integer programming, and mixed integer programming.Source: §1 · p. 6

• Unlike prior AI–OR work focused on solving OR problems, this paper focuses on facilitating and automating optimization-model construction.Source: §2 · p. 8

Automated optimization modeling

• The cited work includes NL4OPT as a pioneering competition; Chain-of-Experts builds optimization models and significantly outperforms GPT-4 via domain-specific knowledge and cooperative reasoning, while OptiMUS supports solver-code writing and debugging.Source: §2 · p. 8

Synthetic data for optimization modeling

• Prior synthetic-data studies target specialized MILP constraints or NL4OPT, using solution examples, CodeT5 augmentation, or LLMs to identify variables, classify constraints, and generate models (Pawlak and Krawiec, 2017; Sroka and Pawlak, 2018; Prasath and Karande, 2023; Li et al. 2023b).Source: §2 · p. 9
• In contrast, this paper proposes general-purpose synthesis of optimization-modeling data from unstructured natural-language descriptions, enabling problem-specific strategies and customized data augmentation.Source: §2 · p. 9

3. Semi-Automated Framework for Data Generation

• The framework is designed to synthesize data for optimization modeling, whose output includes concrete decision variables, an objective function, constraints, and a solver program.COPT is the paper’s default solver, though the text says any well-established solver may be used in dataset construction.Source: §3 · p. 10, §3.1 · p. 10

An Expected Training Example

• An expected example is a triplet (p,m,c): a natural-language problem, its mathematical model, and a solver-compatible program; challenging cases also need varied modeling techniques.The target mapping is f: p → (m,c); examples of techniques include auxiliary variables for linearization and big M formulations.Source: §3.1 · p. 10, An Expected Training Example · p. 12

3.1. Data Requirements

• Training data should cover varied scenarios, problem types, and difficulty levels; reflect changing objectives and constraints; and include linguistic variation.Diverse descriptions, such as “inventory overflow” and “excess stock,” are intended to improve understanding of varied problem statements.Source: §3.1 · p. 10

3.2. OR-Instruct: Towards Training Effective OR LLMs

• OR-Instruct addresses scarce, difficult-to-collect industrial training data with a semi-automated pipeline that iteratively expands and augments examples, then filters obviously low-quality data.The pipeline is designed to create synthetic data tailored to the stated training requirements.Source: §3.2 · p. 12

Initial Dataset

• The initial dataset comprises 686 anonymized real-world industry cases, and concentrating it in a specific problem type or scenario can strengthen domain-specific modeling.The cases were collected from OR textbooks and prior industrial projects.Source: Initial Dataset · p. 12

Strategy 1: Expansion for Comprehensive Coverage.

• GPT-4-powered bootstrapping expands scenarios and question types from the training dataset; generated samples use three in-context examples, and of 50 generated cases, 87% were easy, 13% medium, and none hard.Source: Strategy 1: Expansion for… · pp. 12–13

Strategy 2: Augmentation for Problem-Solution Diversity.

• Augmentation rephrases questions, alters objectives or constraints, and adds modeling techniques to broaden problem-solution diversity while retaining problem complexity.For objective and constraint changes, GPT-4 proposes five possibilities before modifying the problem, model, and program; it may also generate illogical background scenarios.Source: Strategy 2: Augmentation for… · p. 13
• Contradictory constraints can stem from unreasonable problem descriptions, but accurate mapping of statements to objectives and constraints can still yield desired results on carefully designed datasets.Source: Strategy 2: Augmentation for… · p. 13

Altering Objectives and Constraints for Requirement 2

• Rephrasing problem descriptions while preserving the same optimization model is intended to improve robustness to prompts and linguistic diversity.GPT-4 rewrites descriptions to simplify or complicate them while keeping their core logic aligned with the model and programs.Source: Altering Objectives and Constraints… · p. 14
• GPT-4 chooses among five modeling techniques, including auxiliary variables and the Big M method, to modify objectives or constraints and increase solution variability.The techniques are drawn from engineers’ experience.Source: Altering Objectives and Constraints… · pp. 14–15

Postprocessing and Filtering

• Manual sampling found about 70% accuracy for expansion data and 75% for augmentation data; each iteration filtered approximately 39% of generated examples.The remaining examples were incorporated into the training data pool.Source: Postprocessing and Filtering · p. 15

4.1. Data Generation by Using OR-Instruct

• OR-Instruct data comprises 57% expansion, 17.2% objective-and-constraint alteration, 15.3% question rephrasing, and 10.5% incorporation of multiple modeling techniques.Source: Figure 7 · p. 16, §4.1 · p. 16
• The generated data spans 16 industries, 1,556 scenarios, and 8 question types, with correctness accuracy of 70% for expansion and 75% for augmentation data.The authors consider these accuracy levels acceptable in exchange for automatic filtering and more cost-effective synthesis.Source: §4.1 · p. 16

4.2. Model Finetuning and Inference

• ORLMs are instruction-tuned on natural-language OR problems, with complete solutions—including mathematical models and programs—as targets, using approximately 7-billion-parameter backbones.The backbones include Mistral-7B, Deepseek-Math-7B-Base, and LLaMA-3-8B.Source: §4.2 · p. 17
• Evaluation uses zero-shot greedy decoding and executes generated code to compare predicted optimal values with ground truth, favoring reproducibility despite possible performance loss.Source: §4.2 · p. 17

4.3. Evaluation and Baselines

• IndustryOR contributes 100 real-world problems from eight industries; execution accuracy supports fully automated evaluation by checking executed optimal values against ground truth.The benchmark spans five question types and three difficulty levels.Source: §4.3 · p. 18
• Human-level performance is assessed by asking eight senior undergraduate students and eight domain experts to solve randomly selected questions, with two participants from each group evaluating each test set.Participants submit finalized code without further modifications, and group accuracy is averaged for each test set.Source: §4.3 · pp. 18–19
• ORLM-LLaMA-3 and ORLM-Qwen2.5 achieve state-of-the-art performance across four benchmarks relative to GPT-4 and other open-source LLMs.The supplied passage does not provide the corresponding score values.Source: Table 2 · p. 19, §4.3 · p. 20
• LLM-based methods outperform tag-BART on NL4OPT, while proprietary LLM performance improves from GPT-3.5 to GPT-4 across prompt-engineering methods.Source: §4.3 · p. 19
• ORLMs outperform standard prompting based on GPT-4 by 42.2% in micro average and 55.4% in macro average.Source: §4.3 · p. 20
• Human evaluation finds ORLMs outperform senior undergraduates on NL4OPT/MAMO-EasyLP but trail average students on IndustryOR/MAMO-ComplexLP, despite potential to approach expert-level solutions.Source: §4.3 · p. 20
• ORLM-LLaMA-3-8B improves mathematics and coding performance, slightly declines on general-domain benchmarks, and shows no substantial average change across domains.The comparison evaluates LLaMA-3-8B before fine-tuning against ORLM-LLaMA-3-8B after fine-tuning.Source: Table 3 · p. 20, §4.3 · p. 20
• The human–AI collaboration comparison across accuracy and solution time was statistically significant despite a limited participant sample.Source: Figure 8 · p. 21, §4.3 · p. 21

Appendix A.6.

• At 95% confidence, ORLM significantly improved accuracy by approximately 10%–25% and saved 1.8–2.2 hours overall (experts: 1.5–2; students: 2–2.5).The improvements apply across total, expert, and student samples.Source: Appendix A.6 · p. 22
• Absolute Hedge’s g > 1 indicated a substantial experimental-control difference, while statistical power > 0.8 supported true-effect detection and reduced Type II error risk.Source: Appendix A.6 · p. 22
• Engineer feedback was uniformly nonnegative: none rated ORLM “useless” or “minimal help,” while a majority called it “significant help” or “highly effective.”The evaluations indicated that engineers found ORLM efficient and effective in assisting their work.Source: Table 5 · p. 22, Appendix A.6 · p. 22

4.4. Customization Enhancement

• Starting with 10,000 general modeling cases, the MILP customization process used 50 fundamental question-answer pairs to generate 2,000 targeted training examples through iterative rounds.The examples were added to the training dataset to assess performance before and after their inclusion.Source: §4.4 · p. 23
• GPT-4 and general ORLM misread logical relationships as quantitative, but targeted-trained ORLM produced a correct model using indicator variables and the Big M method.The errors included nonlinear terms that adversely affected solver performance.Source: Figure 9 · p. 23, §4.4 · pp. 23–24
• The MILP example showcases OR-Instruct’s generalization and its potential to enable customized improvements across modeling types and industrial sectors.Source: §4.4 · p. 24

5.1. Detailed Comparison of ORLM vs GPT-4 on IndustryOR

• Weak nonlinear and rare-type performance is partly attributed to too few seed-dataset examples and the inherent complexity and scarcity of those questions.Source: §5.1 · p. 24
• The authors conclude that OR-Instruct enhances coverage across question types and difficulty levels.Source: §5.1 · p. 24
• ORLM-LLaMA-3-8B outperforms Standard-GPT-4 across IndustryOR difficulty levels and in linear, integer, and mixed-integer programming, while both perform poorly on nonlinear and rare types.Figure 10 compares accuracy across difficulty levels and question types.Source: Figure 10 · p. 24, §5.1 · p. 24

Ablation Study on Data Synthesis Strategy

• The ablation compares fine-tuning on seed data alone with fine-tuning on OR-Instruct-generated data based on the same seed data.Source: Ablation Study on Data Synthesis… · p. 25
• OR-Instruct-generated data significantly boost ORLM performance over seed data alone, except for minor variations on MAMO_EasyLP.The comparison uses generated data based on the same seed data.Source: Ablation Study on Data Synthesis… · p. 25, Table 6 · p. 25

Ablation Study on OR-Instruct Augmentations

• The ablation compares four 3,000-instance datasets with different augmentation strategies, training LLaMA-3-8B under consistent hyperparameters.The tested augmentations alter objectives and constraints, rephrase questions, and incorporate multiple modeling techniques.Source: Ablation Study on OR-Instruct… · p. 25
• 68.6% micro average and 55.7% macro average are reported for Full Augmentations; removing any augmentation lowers both averages across benchmarks, with rephrasing slightly more important.The results indicate that all three augmentations contribute to general performance.Source: Table 7 · p. 25, Ablation Study on OR-Instruct… · p. 25
• ORLM can be tailored to specific domains.Source: Ablation Study on OR-Instruct… · p. 26

Ablation Study on Question Types

• Mixed problem-type training significantly improves ORLM-LLaMA-3-8B across datasets, with added types especially helping challenging MAMO linear-programming tests.The authors suggest that broader problem variety strengthens generalization.Source: Table 8 · p. 26, Ablation Study on Question Types · p. 26

5.3. Scaling Laws

• As training data grows, micro-average performance increases monotonically, while macro-average gains fluctuate as performance bottlenecks emerge on simple tests.The reported bottleneck affects NL4OPT and MAMO-easyLP; the authors nevertheless conclude that more data remains beneficial.Source: §5.3 · p. 27
• Larger Qwen2.5 models steadily improve ORLM accuracy, following a power-law trend consistent with Kaplan et al. (2020).The authors interpret this as evidence that the base model’s intrinsic capability sets a higher performance ceiling for optimization modeling.Source: §5.3 · p. 27

5.4. Inference Techniques

• Across four test sets, ORLM's Pass@8 micro average increased by 17.09% and macro average by 24.34% versus the previous version, surpassing senior undergraduates and approaching experts in accuracy.Source: §5.4 · p. 28

5.5. Limitations Analysis

• ORLM still has limitations, but scaling laws and different inference techniques offer opportunities for improvement.Source: §5.5 · pp. 29–30
• ORLM is promising for complex optimization modeling, including certain real-world industrial problems within specific domains.Source: §5.5 · p. 30
• The following section demonstrates ORLM applications in industrial production, education, and other real-world scenarios.Source: §5.5 · p. 30

6.1. Potential Applications

• The authors present ORLM as the first open-source LLM fine-tuned for optimization modeling, with performance beyond cutting-edge proprietary models and privacy, security, and stability for commercial use.Its 7B size also allows private deployment.Source: §6.1 · p. 30
• Companies can customize ORLMs using OR-Instruct data; the models retain pretrained knowledge, grasp industry terminology, bridge business–engineering communication, and generate initial solutions within seconds.The paper points to Tables 2 and 3 for improved optimization modeling capabilities.Source: §6.1 · p. 31
• With strong Pass@8 performance, ORLM can generate multiple candidate solutions for market changes and revise models with new constraints, preliminary formulations, and solver code for engineer validation.The authors say these capabilities can reduce delivery costs and improve accuracy, though the supplied passage does not specify the accuracy measure.Source: §6.1 · pp. 31–32
• With retrieval-augmented generation over internal documents, ORLM could support domain knowledge transfer and updates, turn project reports into training assets, and help train junior algorithm engineers.The paper also proposes ORLM as instructional support: instructors can provide reference solutions and processes for unfamiliar modeling problems to help beginners.Source: §6.1 · p. 32

Incorporating sophisticated techniques

• ORLM can generate correct answers but struggles to rank them effectively; reinforcement-learning techniques could help narrow the gap between pass@1 and pass@8 scores.Source: §6.2 · pp. 32–33

Dataset construction

• Future datasets should include preference-ranked responses and actual optimal solutions, potentially using user feedback or new synthesis methods.The existing synthesis strategy is insufficient for preference-based reinforcement learning such as RLHF.Source: §6.2 · p. 33

Data mining and exploration

• Although scaling laws benefit ORLM, added data yields increasingly marginal gains at fixed model size, motivating minimal datasets and efficient synthesis that balance performance and cost.The paper proposes tailored data exploration to reduce reliance on large-scale data resources.Source: §6.2 · p. 33

• OR-Instruct is a semi-automated synthetic-data framework designed around four critical dataset requirements, and IndustryOR is the first benchmark of its kind in industry.Source: §7 · p. 33
• Approximately 7-billion-parameter open-source LLMs trained on OR-Instruct data significantly enhanced optimization modeling and achieved competitive performance across all test datasets.Source: §7 · p. 33
• ORLM’s limitations include overly simplistic outputs for complex problems, weak ranking of optimal solutions, and poor learning from data; experiments suggest scaling laws or reinforcement learning could bring expert-level performance.Source: §7 · p. 34
• The framework offers a reference for refining large models’ optimization modeling through high-quality data to strengthen comprehension and reinforcement learning to align ranking capabilities.Source: §7 · p. 34

A.1. Criteria for Difficulty Level of Problem

• The study acknowledges that modeling difficulty is hard to quantify and uses four criteria to assess problems as easy, medium, or hard.GPT-4 independently evaluates each problem using the criteria and illustrative examples.Source: §A.1 · p. 40
• The four difficulty criteria are problem size, logical relationships, language ambiguity, and interdisciplinary knowledge requirements.The criteria capture factors that make translating natural-language descriptions into precise mathematical models more difficult.Source: §A.1 · p. 40

A.2. Prompt Template for Expansion

• The expansion workflow first generates diverse OR scenarios, then uses three examples to create a problem for a selected scenario with its model and COPT code.The scenario prompt requests specific applications across industries, while the second prompt uses the first prompt’s output as its scenario list.Source: §A.2 · pp. 41–42

A.3. Prompt Template for Augmentation

• Augmentation templates use placeholders for inputs and prompts that direct the language model to generate improved output in the specified locations.The instructions explain that these placeholders are filled during use, for example with GPT-4.Source: §A.3 · p. 44

A.3.1. Prompt Template for Altering Objective and Constraints

• The alteration template asks the model to propose a feasible real-world change, revise the problem description, and update the model and COPT code accordingly.It calls for a complete revised description, mathematical model, and code in the original format.Source: §A.3.1 · p. 44

A.3.2. Prompt Template for Rephrasing Questions

• The rephrasing template changes the problem description’s wording while preserving its meaning and model validity, leaving the mathematical model and COPT code unchanged.The output replaces only the description and retains the original entry’s format.Source: §A.3.2 · p. 45

A.3.3. Prompt Template for Incorporating Multiple Modeling Techniques

• The multiple-technique prompt links auxiliary variables to complex relationships or nonlinearities, Big M to conditional constraints, and penalty functions to hard constraints.It directs users to identify where techniques fit, modify the mathematical model and corresponding COPT code, and organize the revised output in the original format.Source: §A.3.3 · pp. 45–46

A.4. Alpaca-like Template for ORLMs Training

• The Alpaca-like template asks users to replace {Question} with a natural-language optimization problem and requests a mathematical model plus corresponding coptpy code.The ORLM returns the output in a standardized format.Source: §A.4 · p. 46

A.5.1. Numerical Results for Different Question Types.

• Supplementary tables cover IndustryOR comparisons by difficulty and question type, and performance comparisons across data sizes and model sizes.The supplied passages provide table captions only, not the underlying values.Source: Table 12 · p. 47, Table 13 · p. 47, Table 14 · p. 47

A.6. Hypothesis Testing on the Effectiveness of ORLM

• At a 95% confidence level, ORLM significantly improves results; accuracy improves by approximately 10%–25%, and total time saved solving optimization modeling problems is 1.8–2.2 hours.Source: §A.6 · p. 50

A.7.1. Easy Example

• The easy example is a scooter-and-bike product-mix problem that maximizes profit subject to design-team and engineering-team hours.The stated profits are $200 per scooter and $300 per bike, with monthly capacities of 5,000 design hours and 6,000 engineering hours.Source: §A.7 · p. 50, §A.7.1 · p. 50

A.7.2. Medium Example

• The medium example schedules six employees over ten days, minimizing unmet daily staffing needs and disparities in employee workloads.Daily staffing requirements and employee availability are given.Source: §A.7.1 · p. 50, §A.7.2 · pp. 50–51

A.7.3. Hard Example

• The hard example assigns vehicles to steel-coil transfer and dock tasks, sets execution times, and requires every task to be assigned.Tasks sharing start and end points may be merged on one vehicle, subject to its total-weight and coil-count limits.Source: §A.7.2 · p. 51, §A.7.3 · p. 51
• Vehicles cannot take a new task while en route; the listed task fields are coil ID, weight, starting warehouse, and destination warehouse.The task-format passage ends with a comma, so the field list may be incomplete.Source: §A.7.3 · p. 51

2. Ensure that all tasks are completed as early as possible

• Tasks are to be completed as early as possible, with high-priority tasks prioritized and vehicles beginning from their initial parking spots.Source: §2 · p. 51

4. The vehicle’s transportation speed must be within the maximum and minimum speed limits

• At the same dock, loading of the next ship can begin only after the previous ship’s loading is complete.Source: §4 · p. 51

8. The number of vehicles operating simultaneously in the warehouse area has an upper limit

• No new tasks are assigned to vehicles during the last half hour of the current shift, and the output gives each vehicle’s task sequence and estimated time nodes.Source: §8 · p. 51