OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale

System motivation

• OptiMUS combines LLMs with traditional solvers to automate modeling and solving (mixed-integer) linear programs and accelerate practitioners’ development, tuning, and implementation workflows.The intended user understands the problem domain and can describe a MILP-suitable problem in sufficient detail for modeling.Source: §1 · pp. 2–3

Challenges

• LLM optimization modeling must address long descriptions, large structured data, hallucinated models or APIs, and inefficient formulations that can reduce solver performance.These challenges arise because real-world problems vary in description length, combinatorial complexity, and data scale, while LLM context and verification remain limited.Source: §1 · p. 2

Evaluation scope

• OptiMUS evaluates scalability across description length, problem complexity, and data scale, using NLP4LP, seven real-world case studies, and symbolic external data references.The paper provides empirical evidence for description length and problem complexity, with initial evidence for data scale, but does not claim solver-level scalability for models with thousands of variables or constraints.Source: §1 · pp. 2–4

Central finding

• System architecture, not LLM capability, is the primary bottleneck in LLM-assisted optimization modeling, while modular decomposition with targeted error correction can match stronger or fine-tuned models without retraining.The system targets existing practitioners rather than users who need help selecting a modeling paradigm or concretizing vague problems.Source: §1 · pp. 2–3

Data scale

• 100% success is maintained across input sizes from 3 KB to 935 KB when data is stored externally and referenced symbolically, whereas naive prompting fails beyond GPT-4o’s context window.Separating data from the natural-language description prevents prompt length from growing with data size.Source: §1 · pp. 3–4

Dataset contribution

• NLP4LP contains 361 open-source optimization problems spanning 289 easy LPs, 65 harder LP/MILPs, and 7 real-world MILPs with complex structure and large-scale data.The dataset includes problems less likely to have appeared in existing LLM training data, reducing data-leakage risk relative to prior optimization modeling datasets.Source: §1 · p. 4

System architecture

• A connection graph processes each constraint and objective independently, enabling long natural-language descriptions and large datasets without excessively long prompts.The system’s modular structure supports decomposition of the optimization problem into independently processed components.Source: §1 · p. 4

Framework modules

• Self-reflective error correction evaluates outputs and confidence, using a more powerful LLM or user feedback when OptiMUS is uncertain.The framework also prompts the system to identify important problem structures and solver features, including special ordered sets, to improve modeling and solution efficiency.Source: §1 · p. 4

Ablation studies

• Ablation studies identify which framework components improve performance and which can degrade it.These studies support the reported performance differences between the full system and direct prompting or specialist-model alternatives.Source: §1 · p. 4

Empirical results

• Over 43% on easy instances and over 18% on hard instances, OptiMUS-0.3 beats GPT-4o alone; on seven case studies, it solves 28.6% versus 14.3% and 0%.The case-study comparison is 28.6% for OptiMUS-0.3 (GPT-4o), 14.3% for direct prompting with GPT-4o, and 0% for fine-tuned specialist models.Source: §1 · p. 4

Deployment

• The publicly available OptiMUS web application supports human-in-the-loop modeling by letting users edit outputs and correct low-confidence results.The paper presents this webapp as its first such application.Source: §1 · p. 5

Advances over prior work

• Compared with OptiMUS-0.2 introduced by AhmadiTeshnizi et al. 2024, OptiMUS-0.3 enlarges and open-sources NLP4LP and uses modularity to improve robustness on complicated modeling problems.The modular dataset design supports future improvements to parameter extraction, modeling, and coding, while new solver-functionality modules can speed associated optimization problems.Source: §1 · p. 5

Optimization workflow

• Optimization formulation and solver selection require specialized knowledge because efficient formulations and solver interfaces can materially affect solving performance.Solver manuals are often hundreds of pages, making solver use difficult even after a formulation is available.Source: §2 · p. 5

LLM assistants

• LLM-based tools assist with optimization modeling, coding, and queries, but some target infeasibility repair or narrow application-specific tasks rather than general MILP formulation.These systems include chatbots for Pyomo debugging, supply-chain queries, and meeting scheduling, while direct solution generation is limited to small problems as context grows.Source: §2 · pp. 5–6

Benchmark-driven optimization modeling

• Benchmark-driven systems map natural-language descriptions to LP formulations through intermediate representations or multi-agent modeling workflows.Other approaches decompose modeling into stages and explore candidate models, while MCTS methods output multiple functionally distinct models rather than one formulation and answer.Source: §2 · p. 6

Fine-tuning approaches

• Fine-tuned optimization language models use semi-synthetic or synthetic data generation, providing a complementary route to prompt optimization and agentic approaches.The cited fine-tuning approaches target operations-research modeling or broader optimization classes and can be combined with modular systems.Source: §2 · p. 6

Evaluation resources

• Traditional MILP benchmarks contain diverse standard-form problems, but generally lack natural-language descriptions needed to evaluate optimization modeling from language.Consequently, MIPLIB cannot directly support the paper’s natural-language modeling evaluation.Source: §2 · p. 6

Dataset construction

• NLP4LP contains 361 optimization problems spanning multiple problem types and application domains, with associated modeling resources for each instance.The dataset includes code, extracted parameters, clause-level representations, solutions, and optimal values, enabling evaluation beyond final formulations.Source: §3 · pp. 6–7

Dataset subsets

• NLP4LP comprises easy, hard, and real-world case-study subsets, including LP and MILP problems with short or long descriptions and scalar or multidimensional parameters.The case studies come from real-world operations-research applications published in the INFORMS Journal on Applied Analytics.Source: §3 · p. 7

Dataset construction

• The dataset’s enriched annotations support independent evaluation and improvement of pipeline components such as parameter extraction, clause modeling, and code generation.Researchers can test modules through the OptiMUS webapp without implementing the full software stack.Source: §3 · pp. 7, 9

Evaluation design

• The dataset prioritizes a large test set over a matched training set because high-quality optimization instances are expensive to collect.The easy and hard subsets contain 354 problems, partitioned into 23 development and 331 test problems, with no test-set use during development.Source: §3 · p. 7

Complexity

• The hard and case-study subsets are substantially longer than existing MILP modeling datasets, averaging 912 and 2,578 characters respectively, and case studies challenge current LLM modeling capabilities.Table 2 compares description length and other complexity dimensions across datasets.Source: §3 · p. 8, Table 2 · p. 9

Data leakage

• The study mitigates data-leakage risk by restricting dataset access to authenticated Hugging Face users through Captcha and using entirely original problem text whose sources include no solutions.Source: §3 · pp. 8–9

4. Methodology

• The framework assumes descriptions contain all information needed for modeling, while ambiguous or incomplete user descriptions remain an acknowledged limitation.The implementation currently targets Gurobi and gurobipy but is designed for extension to other solvers and modeling languages.Source: §4 · p. 10
• OptiMUS-0.3 sequentially extracts parameters and clauses, formulates variables and constraints in LaTeX, generates Python code, and runs it to produce an optimal solution.Each component maintains shared model state and uses Error Correction to catch common modeling and implementation errors.Source: §4 · p. 10
• OptiMUS-0.3 adds Structure Detection and customized optimization coding agents to exploit special constraints, solver callbacks, and basic column or constraint sifting.These modules are intended as an initial step toward automating more advanced decomposition algorithms.Source: §4 · p. 11

4.1. State

• The system stores parameters, clauses, variables, background context, and a bipartite connection graph in incrementally completed JSON state.The graph links clauses to participating variables and parameters, allowing coding prompts to include only relevant model elements.Source: §4.1 · pp. 11–13

4.2. Sample LLM Component

• Each LLM component receives a directive containing the task description, relevant problem context, current solution information, and dynamically loaded model elements.Prompt contents vary by stage: formulation uses the current clause and defined parameters or variables, while coding uses its formulation and related elements.Source: §4.2 · pp. 13–14
• LLM components receive fixed sample outputs for in-context learning, with gurobipy API examples for modules that generate Python code.The webapp lets users inspect and correct every LLM-powered component.Source: §4.2 · p. 14

4.3.1. Reflective prompts

• Reflective prompts significantly reduce modeling errors by targeting common LLM mistakes identified on development data.A unit-consistency prompt illustrates how domain-specific reflection can correct constraint modeling errors and improve OptiMUS substantially.Source: §4.3.1 · p. 14

4.3.2. Confidence-based user/LLM feedback

• OptiMUS uses confidence scores from 1 to 5 to request user or stronger-LLM help whenever its score is below 5.The mechanism supports iterative formulation feedback and applies across parameter extraction, variable identification, and constraint extraction; experiments disable it by default for reproducibility.Source: §4.3.2 · pp. 14–15

4.3.3. Debug Code

• The system passes execution errors to an LLM that can modify code and repeats debugging until successful, with a maximum of five iterations.This addresses inconsistencies across code snippets, including variable-name and formatting variations.Source: §4.3.3 · pp. 15, 17

4.4. Structure Detection Agent

• OptiMUS maintains pools of reusable optimization and problem structures, suggesting specialized solvers when natural-language descriptions match structures such as routing or network-flow problems.These structures are identified during formulation and can be conveyed through solver interfaces or surfaced as customized-solver notifications.Source: §4.4 · p. 17
• Structure detection identifies SOS, indicator-variable, and related structures, enabling solver-native formulations that outperform naive implementations in the illustrated cases.Gurobi can generate tighter formulations, including better big-M values, while detected structures may improve branching or automatic reformulation.Source: §4.4 · p. 17

4.5. Advanced Optimization Coding Agent

• The advanced optimization coding agent exploits callbacks and iterative solver calls within sifting schemes to improve computational performance without explicit problem reformulation.The agent runs during the Code Clauses stage and uses template prompts for advanced solver functionality.Source: §4.5 · p. 18

4.5.1. Sifting

• Sifting combines primal simplex and column generation by solving a restricted master problem for linear programs with n≫m.The section illustrates this functionality through a constraint-sifting prompt and a performance plot targeting a 5% gap.Source: §4.5.1 · p. 18, Figure 6 · p. 19
• Variable sifting iteratively updates S ← S ∪ I_S until S = ∅, while dual variables y_S price out dual-infeasible columns.The approach can also generalize to the dual side when m ≫ n by restricting the problem to a subset of rows.Source: §4.5.1 · p. 19
• OptiMUS illustrates constraint sifting on a security-constrained unit commitment problem in Fig. 6.The instance is a suitable candidate because most of its constraints are inactive.Source: §4.5.1 · p. 19

5.1. Overall Performance

• Accuracy requires successful execution, the correct optimal value, and the correct solution when reference solutions exist, but variable-space matching can produce false negatives.For NL4OPT and IndustryOR, correctness uses execution and optimal-value agreement; NLP4LP additionally checks the optimal solution, making the solution criterion a lower bound on true accuracy.Source: §5.1 · p. 20
• The evaluation covers NLP4LP, NL4OPT, and IndustryOR, whose 100 real-world case studies span eight industries, using identical preprocessed inputs across methods for like-for-like comparison.OptiMUS itself can perform parameter extraction, but preprocessing was supplied to all methods to isolate modeling and solving ability.Source: §5.1 · p. 20
• OptiMUS-0.3 outperforms all other direct-prompting and agentic methods across datasets, beating standard GPT-4o by over 43% on easy NLP4LP and over 18% on hard.With o3, OptiMUS comes within 6% of ORLM on Easy and outperforms ORLM on Hard and Case Studies.Source: §5.1 · pp. 20–21
• 350 seconds: OptiMUS-0.3 models and solves any tested NLP4LP instance within this time, with a 108-second median, while coding and debugging dominate runtime.Tang et al. (2024) report approximately 150 minutes for human experts on complex IndustryOR problems, though that comparison does not represent simpler textbook instances.Source: §5.1 · pp. 21–22, Figure 7 · p. 22
• Repeated sampling recovers correct solutions for many hard instances: pass@5 reaches 73% versus 35% pass@1, while 11 of 15 easy instances always pass across five seeds.Source: §5.1 · p. 22
• 0 out of 402: no high-confidence extraction was incorrect at confidence ≥4/5, whereas low-confidence predictions were over-flagged in 4 of 8 cases.Accuracy among low-confidence items increased from 25% at score 1/5 to 67% at 2/5 and 100% at 3/5, confirming that lower scores predict errors.Source: §5.1 · p. 23
• The confidence mechanism records a 0% type-I rate across 402 high-confidence predictions spanning parameters, variables, and constraints.This provides evidence that the mechanism is reliably conservative, avoiding unnecessary review of correct predictions.Source: §5.1 · p. 23

5.2. Ablation Study

• OptiMUS-0.3 is robust to the underlying LLM: its weaker LLaMa-3.1-70B-Instruct configuration outperforms GPT-4o with previous prompting strategies on easy problems.The framework degrades with weaker models, especially on hard tasks, but modular structure allows weaker models to match stronger models on simple optimization modeling tasks.Source: §5.2 · p. 23
• Removing debugging reduces NLP4LP Easy accuracy from 78.2% to 26.7% (−51.5pp), making error correction the dominant component.Iterative solver feedback identifies and repairs syntactically valid but semantically incorrect code, with the largest effect on hard instances.Source: §5.2 · p. 24, Table 5 · p. 25
• Removing Extraction EC reduces accuracy by 17.7 percentage points, from 78.2% to 60.5%, while removing Modeling EC causes a further 12.5-point drop, from 78.2% to 65.7%.Source: §5.2 · p. 24

5.3. Failure Cases

• For harder MILP and multi-dimensional instances, most failures arise during clause extraction and modeling, while easier LP instances more often fail during coding.Hard instances are more difficult because parameter extraction and modeling are substantially more challenging than coding the resulting model.Source: Figure 8 · p. 25, §5.3 · p. 25

5.4. Scalability with Respect to Problem Complexity and Data Scale

• On seven real-world case studies, o3 produces runnable code for five instances but only two correct, whereas GPT-4o produces four runnable and three correct instances.These results demonstrate the challenge that problem complexity and data scale pose for current LLM-based modeling systems.Source: Table 7 · p. 26, §5.4 · p. 26
• OptiMUS stores data externally and exposes only symbolic parameter names and shapes, making prompt length independent of input data size.The experiments vary planning horizon and route count, using five random seeds per instance-scale pair and a 600-second solver budget.Source: §5.4 · p. 27
• Naive prompting declines from 80% at T=5 to 40% at T=30 for Amazon Locker, while exceeding GPT-4o’s 128K-token context window at T≥75.The larger prompts create genuine formulation errors before becoming infeasible beyond data sizes of 245 KB.Source: §5.4 · p. 27
• At small-to-medium scales (2–190 KB), naive prompting achieves 80% for two instances and 100% at other tested scales; at 272–385 KB, it cannot generate code.Source: §5.4 · p. 28
• The experiments identify formulation errors and context overflow as naive prompting’s two failure modes, neither affecting OptiMUS at tested scales.These failures reflect challenges in handling domain-specific data structures, translating multi-indexed formulations, and recognizing needed auxiliary structure.Source: §5.4 · pp. 27–28
• 100% correct-formulation rates persist across all 8 Amazon Locker scales (3 KB–935 KB) and all 9 R-VRP scales (2 KB–385 KB) for OptiMUS.OptiMUS generates code once and applies it across data sizes; Figure 9 reports the comparison with naive prompting.Source: §5.4 · pp. 27–28, Figure 9 · p. 28

• The web app guides users through extracting parameters and clauses, modeling each clause, and translating the resulting model into code.Source: §6 · pp. 28–29
• Users can correct natural-language clauses, LATEX models, and generated code, providing supervision across the entire process.Source: §6 · p. 28
• The interface lets users interact with the system throughout each step, improving modeling convenience while reducing development time and error risk.Source: §6 · p. 28
• The web app presents the coding stage through an interface where users can inspect confidence scores, edit code, or request regeneration.Source: Figure 10 · p. 29, §6 · p. 29
• The publicly available web app includes additional stages and features documented in Appendix F.Source: §6 · p. 29

7. Conclusion

• Modular structure and debugging improve optimization performance, with debugging alone accounting for nearly half of performance gains on hard problems.The conclusion connects these gains to structured, agentic optimization modeling.Source: §7 · p. 29
• OptiMUS-0.3 achieves state-of-the-art performance on NLP4LP and scales to problems with long descriptions and complex real-world data.The conclusion describes the system as a prototype for broader access to classical optimization tools.Source: §7 · pp. 29–30
• System architecture can enable weaker models to match stronger ones and compete with fine-tuned specialist models without retraining costs.The paper identifies architecture as the primary lesson of its LLM-based optimization modeling work.Source: §7 · p. 29
• OptiMUS-0.3 is designed to accelerate modeling and implementation for practitioners who already understand the problem domain.The system is positioned as a productivity tool rather than a replacement for domain expertise.Source: §7 · p. 29
• The system remains limited to small problems and requires substantial work before it is reliable enough to reduce the expertise needed for practical deployment.The paper also identifies vague problem concretization and choosing among modeling languages as open challenges.Source: §7 · pp. 30–31
• Future work must address reliability, trust, ambiguity, fast solvers, larger datasets, and support beyond MILP modeling languages.The paper specifically notes unreliable LLM behavior, unclear natural-language formulations, limited datasets, and the current use of gurobipy.Source: §7 · pp. 30–31

• The system state contains parameters, clauses, variables, and background information that are managed throughout solution generation.Parameters are known quantities, variables are unknown quantities to determine, and clauses represent modeled problem relationships.Source: B. Terms and Definitions · p. 38
• Reflective prompts and confidence-based user feedback are used to detect and correct modeling errors when the LLM is uncertain.Examples include checking units, asking users to verify uncertain constraints, and correcting misidentified parameter shapes.Source: B. Terms and Definitions · p. 38
• OptiMUS extracts parameters, constraints, objectives, and connection graphs before formulating clauses and defining variables or auxiliary constraints.These stages identify known quantities and relationships needed for mathematical representation.Source: B. Terms and Definitions · p. 39
• The implementation pipeline generates code snippets, combines them into a runnable file, and iteratively executes and fixes errors.The resulting code is saved and updated during the problem-solving process.Source: B. Terms and Definitions · p. 39

• Parameter extraction uses prompts asking whether a quantity is known and therefore a parameter or an unknown variable.These prompts target confusion between parameters and variables, incorrect shapes, and missed parameters.Source: C. Reflexive Prompts · p. 40
• Constraint checking asks whether units match, whether a constraint is valid for MILP solvers, and whether it involves variables or only parameters.The system also asks whether constraints are explicit, redundant, trivial, vague, or repeated.Source: C. Reflexive Prompts · pp. 40–41
• OptiMUS applies similar reflective prompts to the objective and uses introspection to correct constraint-modeling errors.The supplied figures document correction of constraint, parameter, and variable identification errors.Source: C. Reflexive Prompts · p. 41, Figure 11 · p. 41, Figure 12 · p. 42

• NLP4LP includes a distribution of specialized problem structures that OptiMUS-0.3 is designed to detect and model.The distribution is summarized in Table 11.Source: D. Additional details on Structure… · p. 42, Table 11 · p. 42
• An indicator constraint links a binary variable to a constraint through the implication z = 1 ⇒ C.The example applies this structure to store-opening and staffing requirements.Source: D. Additional details on Structure… · p. 44
• Special ordered sets constrain how many variables in a set may be nonzero: SOS1 allows at most one, while SOS2 allows at most two.These structures are presented as additional modeling patterns.Source: D. Additional details on Structure… · p. 45
• The airline example combines assignment, rest, flying-hour, layover, and cost requirements in a structured optimization formulation.Each crew member can be assigned to at most one flight at a time, with specified regulatory and utilization limits.Source: D. Additional details on Structure… · p. 45

Objective

• SCUC minimizes total operational cost, including fuel, start-up and shut-down, and applicable emission costs.Source: E. SCUC problem · p. 45

Constraints

• Power balance requires online generation to equal demand plus transmission losses at every interval.Source: E. SCUC problem · p. 46
• Generator outputs must remain within minimum and maximum capacities, while ramp limits restrict how quickly output can change.Source: E. SCUC problem · p. 46
• Start-up, shut-down, and minimum up/down-time constraints govern unit commitment and ensure required operating durations.Source: E. SCUC problem · p. 46
• Reserve requirements provide spinning and non-spinning capacity for demand surges and unexpected generator failures.Source: E. SCUC problem · p. 46

• The application is designed to help experts develop models and make decisions faster while reducing costs.Source: F. Web Application · p. 46

Design principles

• The interface follows the user’s modeling flow rather than requiring users to adapt their workflow to the system.Source: F. Web Application · p. 46

Process

• The workflow extracts and edits parameters, detects and formulates objectives and constraints, and generates code for each clause.Source: F. Web Application · p. 46
• The system infers a data-file structure, accepts uploaded or randomly generated data, and supports user code editing and LLM-assisted debugging.Source: F. Web Application · p. 46

Design principles

• Users can observe and intervene throughout the modeling workflow, improving convenience while minimizing error risks.Source: F. Web Application · p. 46

• Resource allocation is the most common operational area, while manufacturing is the most common industry sector.Source: G. Dataset Figures · p. 47
• Most dataset instances have around four associated labels, with examples ranging from two to eight.Source: G. Dataset Figures · p. 48
• Plagiarism testing uses an online tool, and the dataset includes a histogram of originality scores for all instances.Source: G. Dataset Figures · p. 48

• The reported inference settings reflect API defaults at the time of the experiments, rather than explicitly configured OptiMUS parameters.Source: H. Inference Settings · p. 49
• For o1, temperature and top-p are fixed at 1.0 and cannot be adjusted through the API.Source: H. Inference Settings · p. 49
• For o3, temperature and top-p are unsupported, while reasoning depth is controlled internally by the model.Source: H. Inference Settings · p. 49