Large-Scale Optimization Model Auto-Formulation: Harnessing LLM Flexibility via Structured Workflow

1. Introduction

• Fine-tuned approaches focus mainly on small inputs and outputs, while large-scale problems require separately stored datasets and can see modeling accuracy deteriorate.The paper defines large-scale inputs as instances whose data must be stored separately in datasets.Source: §1 · p. 2
• On Large-Scale-OR, most models’ accuracy decreases as input size grows, with Gemini 3 Pro and GPT-5.2 below 50% when inputs exceed 800 tokens.The reported drop occurs well below their advertised input limits; Figure 1 illustrates the decline as input size increases.Source: Figure 1 · p. 3, §1 · pp. 3–4

1.1. Main Contributions and Key Messages

• LEAN-LLM-OPT uses a classification agent, a workflow-generation agent, and a model-generation agent to structure formulation and data extraction into sequential steps.The generated workflow uses reference problems and customized tools to guide structured reasoning.Source: §1.1 · p. 4
• Large-Scale-OR is a 101-entry testing dataset serving as a benchmark, emphasizing medium-size problem instances (20–99 variables; 26% of the dataset) and large-size instances (at least 100 variables; 50%).Source: §1.1 · pp. 4–5
• In Large-Scale-OR, both models achieve overall accuracy above 76%, substantially outperforming ORLM, Gemini 3 Pro, and GPT-5.2.Source: §1.1 · p. 5
• The authors also report high performance on small-scale benchmarks NL4OPT, IndustryOR, and Mamo (Easy and Complex).The passage names the benchmarks but provides no quantitative results.Source: §1.1 · p. 5
• LEAN-LLM-OPT achieves leading accuracy on Air-NRM-CA for Singapore Airlines’ fare-type capacity allocation problem using simulated datasets.The use case also concerns flight network planning and focuses on the sales-based linear programming formulation.Source: §1.1 · p. 5
• LEAN-LLM-OPT consistently achieves small optimality gaps across 21 Air-NRM-NP instances and outperforms competing methods in almost all instances.The introduction also reports results for a network-planning problem, but the supplied passages do not provide further details.Source: §1.1 · p. 5
• The ablation study identifies both the agentic workflow and data-handling tools as important for high modeling accuracy.The authors present these findings as evidence of the design’s effectiveness and practicality.Source: §1.1 · p. 5

1.2. Related Works

• Unlike prior approaches, LEAN-LLM-OPT combines step-by-step examples and agent collaboration for lengthy descriptions and external data, supports plug-and-play LLMs and datasets, and performs well with Singapore Airlines.The introduction characterizes the framework as more efficient and budget-friendly than approaches requiring extensive training data and GPU rental.Source: §1.2 · pp. 6–7

2.1. Reference Dataset Construction

• Ref-Data contains 96 instances—78 small-scale and 18 large-scale—and focuses on LP and MILP while aiming to generalize beyond them.It includes prevalent problem types such as NRM, RA, and TP, as well as additional types and combinations.Source: §2.1 · pp. 7–8
• Ref-Data instances pair a query with its problem type, relevant data category and details, and an expert-generated optimization model.Some small-scale instances are included to support problem classification.Source: §2.1 · p. 8

2.2. Large-Scale-OR Testing Dataset

• Large-Scale-OR contains 101 test instances pairing natural-language queries and datasets with ground-truth mathematical models and optimal values; half have at least 100 variables.The benchmark spans applications including medical, e-commerce, and supply chain management, and includes common and mixed problem types.Source: §2.2 · pp. 9–10

2.3. Textual Comparisons between Ref-Data and Large-Scale-OR

• Semantic similarity between Ref-Data and the benchmarks lies toward the lower end of the reported scale, indicating limited semantic overlap.The comparison uses each benchmark query’s highest similarity to a Ref-Data instance, averaged across queries.Source: §2.3 · p. 11, Table 2 · p. 11
• Ref-Data instances have much lower average token counts than Large-Scale-OR instances, whose inputs are also much larger than those in other benchmarks.The authors conclude that Ref-Data differs substantially from the benchmarks in both semantic content and token scale.Source: §2.3 · p. 11, Table 3 · p. 11

3. LEAN-LLM-OPT Framework

• Three LLM agents classify the input, construct a formulation workflow from relevant examples, and follow it to generate the optimization model.The input comprises a problem description and associated datasets.Source: §3 · pp. 12–13

3.1. Problem Classification

• The classification agent uses FileQA to retrieve the five most similar instances from Ref-Data, then assigns a predefined problem type or “Others.”The classification agent is optional, but the paper says it can positively affect modeling accuracy.Source: §3.1 · pp. 13–15

3.2. Workflow Generation and Model Generation

• The workflow generation agent selects a structurally similar example and directs model generation to offload mechanical data processing to auxiliary tools.Type-tailored workflows serve classified problems other than “Others” and “Mixture”; those two use type-agnostic workflows.Source: §3.2 · p. 15

3.2.1. Type-Tailored Workflow

• A type-tailored workflow embeds a retrieved example’s query, data category, relevant data, and target model into step-by-step formulation guidance.Its Action step invokes CSVQA, which uses similarity search over line-by-line dataset representations to retrieve relevant data.Source: §3.2.1 · pp. 15–16
• Type-tailored workflows reduce the agent’s workload with prescribed model structures and type-aware data presentation, but depend on classification accuracy and typically do not accommodate complex mixed-type problems.The output uses a type-specific model skeleton, and retrieved information is formatted for placement in the formulation.Source: §3.2.1 · pp. 16–17
• The type-agnostic workflow supports broader scenarios by building an abstract model plan from the query and dataset schema instead of relying on a similar problem type.It directs the agent to inspect the query and dataset snapshots, then identify the target model type and fundamental modeling components such as index sets and decision variables.Source: §3.2.1 · pp. 17–18
• The type-agnostic agent uses Python routines to extract relevant data and parameters, then formulates the objective and constraints while deciding what information to retrieve and how to incorporate it.The workflow uses dataset snapshots to identify relevant rows and columns before extracting model parameters.Source: §3.2.1 · p. 18

3.2.3. Model Generation Agent

• After the workflow is built, the downstream model-generation agent produces the final mathematical model and may also provide Python code.The passages introduce an output example for the running example but do not provide its contents here.Source: §3.2.3 · p. 19
• A dedicated component translates generated optimization models or abstract plans with CSV schemas into executable code for solvers such as Gurobi.Source: §3.2.3 · p. 21
• The classification agent is optional but can tailor workflows to common problem types and free downstream reasoning bandwidth; direct RAG without LLM reasoning can reduce classification accuracy.Source: §3.2.3 · p. 21
• Because LLM performance can degrade beyond roughly 800 input tokens and Ref-Data exceeds 40,000 tokens, placing all of it in the prompt could hinder accurate formulation.Source: §3.2.3 · pp. 21–22

3.3. Novelties in Our Design and Key Messages

• LEAN-LLM-OPT dynamically constructs stepwise workflows that decompose modeling into structured subtasks and offload mechanical data handling to tools, reducing downstream planning and data-handling burdens.Source: §3.3 · p. 22
• Workflows use a reference problem to guide dataset inspection and initial model hypotheses, while LLM flexibility helps handle input variation such as differing column names and focus on challenging modeling components.Source: §3.3 · p. 22
• The structured-workflow approach supports large-scale inputs without task-specific fine-tuning, potentially reducing the labeled-data and resource costs associated with fine-tuning approaches.Source: §3.3 · pp. 22–23
• Ref-Data can support generalization despite differences from benchmark instances, and adding different instances may extend use to other domains and scenarios; the plug-and-play design can also pair with more advanced LLMs.Source: §3.3 · p. 23

Results on Large-Scale-OR

• LEAN-LLM-OPT with GPT-4.1 and gpt-oss-20B achieves the highest overall Large-Scale-OR accuracy, above Gemini 3 Pro’s 50.5% modeling accuracy (EM), while their base models score below 26%.The results report overall accuracies below 26% for the base models; they do not give exact overall values for the LEAN variants in the supplied passages.Source: Table 5 · p. 25, §4 · p. 26
• Optimal-value modeling accuracy closely tracks exact-match formulation accuracy on Large-Scale-OR, supporting it as a proxy even though an incorrect formulation can yield a correct value.The discrepancy between the metrics is described as small for most methods considered.Source: Table 6 · p. 25, §4 · p. 28
• LEAN-LLM-OPT imposes no dataset-structure constraints, unlike OptiMUS’s predefined-format requirement, which frequently caused schema mismatches and errors.LEAN-LLM-OPT instead supplies a problem-relevant workflow and data-handling tools, leaving the model-generation agent to formulate details for the input problem.Source: §4 · p. 26
• Routing every problem through the type-agnostic workflow lowers accuracy versus type-tailored workflows, but still outperforms almost all competing methods.Normally, only problems classified as “Others” and “Mixture” use the type-agnostic workflow.Source: §4 · p. 26
• LEAN-LLM-OPT sustains performance on longer inputs, but its accuracy falls as output variable count rises, while several methods improve as variable count increases.The authors suggest input size may also drive difficulty and that better coefficient placement could further improve LEAN-LLM-OPT.Source: §4 · p. 26, Figure 4 · p. 27, Figure 5 · p. 27

4. Numerical Simulations

• Many Large-Scale-OR instances are commonly studied problem types, yet the reported trend indicates greater challenges for these models.Examples of commonly studied types include NRM, RA, and FLP; the supplied passages do not specify the models or quantify the challenges.Source: §4 · p. 29

5. Case Study: Singapore Airlines Choice-Based Revenue Management

• In a choice-based airline revenue-management case study, LEAN-LLM-OPT examines a generalized SBLP formulation on a more challenging task involving multiple datasets and a more complicated target model, against Gemini 3 Pro and GPT-5.2.Source: §5 · p. 29

5.1. Background Information and Sales-Based Linear Programming

• The Singapore-Hong Kong-Manila study uses generalized SBLP, capturing upsell demand and cross-flight recapture while considering only direct itineraries.The authors state that LEAN-LLM-OPT can accommodate formulations generalizing SBLP or relaxing the study’s no-additional-spillover assumption.Source: §5.1 · pp. 29, 31

5.2. Available Data and Testing Dataset Air-NRM

• Air-NRM inputs include simulated ticket-choice and sales data, aggregate demand, and trained and adjusted attraction values, with GAM parameters estimated using EM.A booking simulator trained on Singapore Airlines’ real sales data models customer choice behavior.Source: §5.2 · pp. 31–32
• Air-NRM contains 15 fare-capacity allocation instances and 21 network-planning instances, the latter jointly modeling fare capacity allocation and flight scheduling.Each instance pairs a query and four input datasets with a generalized SBLP formulation and optimal value.Source: §5.2 · p. 33
• For the Air-NRM-NP example, LEAN-LLM-OPT identifies profitable flights and a fare-type capacity configuration that achieves optimal revenue; instances impose closed-loop flow conservation and span 25 to 61 variables.Source: §5.2 · p. 35

5.3. Performance on Fare Type Capacity Allocation and Ablation Study

• LEAN-LLM-OPT achieves leading modeling accuracy on Air-NRM-CA, while Gemini 3 Pro performs much stronger than GPT-5.2.Source: §5.3 · pp. 35–36, Table 12 · p. 36
• Base GPT-4.1 produces no correct model and code for any Air-NRM-CA instance, as it lacks dataset-retrieval know-how and tools.Its outputs correctly generate some non-negativity constraints, but not the other constraints.Source: §5.3 · p. 36
• The tools-only approach reaches 20% modeling accuracy but struggles to interpret retrieved data and place coefficients correctly, especially as problem size grows.The authors attribute this to the lack of a step-by-step guide on data usage.Source: §5.3 · p. 36
• The workflow-only approach has 0% modeling accuracy because it cannot extract all required data, relying on Python built-ins rather than customized tools.It can interpret retrieved data and generate an abstract model, but not correct and complete parameters.Source: §5.3 · p. 36

5.4. Performance on Network Planning

• LEAN-LLM-OPT (GPT-4.1) achieves the highest modeling accuracy on Air-NRM-NP; Gemini 3 Pro ranks second, and LEAN-LLM-OPT (gpt-oss-20B) is comparable to Gemini 3 Pro.The authors note that gpt-oss-20B has substantially fewer parameters than the other models considered.Source: Table 13 · p. 37, §5.4 · p. 37
• GPT-4.1’s optimality gap stays below 14%, tools-only typically exceeds 55%, gpt-oss-20B stays below 24% in all but two cases, and Gemini 3 Pro below 17% except two.GPT-5.2 often has gaps above 40%, sometimes exceeding 80%.Source: Figure 7 · p. 38, §5.4 · p. 38
• Under the SBLP framework and its variants, LEAN-LLM-OPT’s revenue performance remains relatively stable as network-planning problem size and complexity increase.Source: §5.4 · p. 38

6. Concluding Remarks

• LEAN-LLM-OPT addresses whether LLMs can formulate large-scale optimization models, a challenge left open by work focused on small problems with embedded data and small models.The conclusion identifies long-form reasoning as a reason the large-scale setting remained unclear.Source: §6 · pp. 38–39
• Its agentic workflow reduces LLMs’ planning and data-handling burden, letting them focus on components that are difficult to standardize; its plug-and-play design supports practicality and portability.The conclusion presents workflow construction as the mechanism for structuring the modeling process while retaining flexibility for less-standardized parts.Source: §6 · p. 39
• The workflow approach may generalize beyond optimization: domain expertise can form workflow skeletons while LLMs handle unstructured components, potentially improving effectiveness and efficiency across applications.The conclusion frames this as a potential extension, not a demonstrated result outside optimization modeling.Source: §6 · p. 39
• Older-generation and less powerful open-source models may solve challenging problems when their reasoning is appropriately elicited and guided, motivating further study of systematic steering.The conclusion notes that these models can offer lower cost and/or privacy advantages.Source: §6 · p. 39
• Because semantic workflow instructions operate in probabilistic LLMs and may produce unstable behavior, future work could develop more enforceable and controllable mechanisms to improve performance and reliability.The conclusion raises this as a direction for systematic improvement.Source: §6 · p. 39
• One proposed direction is the assured autonomy framework of Dai et al. (2025), which uses an operations-research lens to ensure formal structure, constraint satisfaction, accountability, and tail-risk control.The paper identifies this framework as a promising way to strengthen workflow control.Source: §6 · p. 39

Appendix

• The appendix provides supplementary details for Ref-Data and Large-Scale-OR.The supplied passage identifies the appendix’s subject but does not include those details.Source: Appendix · p. 43

A.1. Application Domains of the Problem Instances

• Datasets containing demand, price, product name, or similar information are classified as NRM; those providing resource capacity, product value, resource consumption, or similar information are classified as RA.Source: §A.1 · p. 43
• Customer demand, transportation costs, and supply capacity identify Transportation Problems.The instances are categorized in Table 16.Source: §A.1 · p. 43
• Datasets with potential facility locations, fixed opening costs, customer demand quantities, and facility-to-customer transportation costs are classified as FLP; cost matrices mapping resources to tasks by assignment cost are classified as AP.Source: §A.1 · p. 43
• Others and Mixture instances are drawn from real-world applications, textbook exercises, and meaningful problems in the literature.Their application domains are summarized in Tables 19 and 20.Source: §A.1 · p. 45

A.2. Parameters Imputation

• NRM data generation derives product demand from sales data, produces product-level revenue and initial inventory, and rounds inventory to a multiple of 10.The resulting data are intended as inputs to further NRM optimization models.Source: §A.2 · pp. 46–47
• Transportation inputs include a supplier-customer cost matrix based on geographic distance and cost per mile, plus customer demand filtered by region.The matrix uses supplier and customer coordinates; demand is taken from the original dataframe.Source: §A.2 · pp. 47–48
• Facility-location inputs include a geographic facility-cost matrix and simulated fixed setup costs drawn from a $10,000–$50,000 range.The setup costs represent facility opening costs used in the problem.Source: §A.2 · pp. 48–49

A.3. Problem Classification Rules

• The benchmark groups problems into network revenue management, resource allocation, and transportation, defined by revenue maximization, resource distribution, and shipping-cost minimization.These are three of seven groups classified by mathematical formulation.Source: §A.3 · p. 50
• The Facility Location Problem determines optimal facility placement to minimize costs while meeting client demands, whereas the Assignment Problem optimally assigns agents to tasks on a one-to-one basis.Source: §A.3 · p. 50
• The “Others” group covers archetypes such as diet, blending, knapsack, traveling salesperson, and scheduling, while “Mixture” covers hybrid or non-standard problems.The cited archetypes include Winston 2004, Pisinger and Toth 1998, and Pinedo and Hadavi 1992.Source: §A.3 · p. 50

A.4. Sample Problem Instance in Large-Scale-OR

• The instance covers Widget1 through Widget141, with product data specifying labor and Material A and B use per unit plus base profit.The parameters are provided in product resources.csv.Source: §A.4 · p. 50

4. CatalystX Constraints

• Widget3 produces 5 kg of CatalystX per unit; sales are priced at $300/kg and capped at 1,500 kg monthly, while unsold material costs $200/kg to dispose.Source: §A.4 · p. 50

1. Labor Hours Constraint

• Monthly labor hours are limited to 5,000.The resource limits are specified in resource limits.csv.Source: §A.4 · p. 50

2. Material A Constraint

• Material A consumption is capped at 24,000 kg.Source: §A.4 · p. 50

A.4. Sample Problem Instance in Large-Scale-OR

• The optimization chooses production quantities for each widget and CatalystX sold to maximize total profits.Source: §A.4 · p. 50

3. Material B Constraint

• Material B usage is capped at 15,000 kg.Source: §A.4 · p. 50

B. On the Comparison between the Classification Agent and RAG

• The example asks a dealership to choose daily vehicle-type order quantities to maximize benefit subject to per-type daily limits and total inventory capacity.The query describes benefit coefficients in products.csv and capacity constraints in capacity.csv.Source: B. On the Comparison between the… · pp. 52–53
• For the Norway car-inventory query, the classification agent returns the correct type, “Resource Allocation,” while simple RAG returns “Inventory Management,” despite identical retrieved instances.The passage identifies Resource Allocation as the correct problem type.Source: B. On the Comparison between the… · p. 53

C. Demo Problem Used in Workflow Generation

• Products.csv supplies product values and space requirements, while Capacity.csv provides the shelves’ capacity data.The example includes products such as the Sony Alpha Refrigerator, Sony Bravia XR, and Sony PlayStation 5.Source: C. Demo Problem Used in Workflow… · pp. 53–55
• The model chooses units of each Sony product for each shelf to maximize total value while respecting shelf-capacity and brand-specific space limits.The decision variables x_ij represent the number of units of product i placed on shelf j.Source: C. Demo Problem Used in Workflow… · pp. 54–55

AP

• AP data are efficiently stored as a complete cost matrix C, with the cost.csv data shown without simplification or abbreviation.Source: D. Type-Tailored Retrieved Data Format · p. 56

TP

• TP supply and demand data are presented in tables, while costs between supply and demand locations are formatted as a matrix.Source: D. Type-Tailored Retrieved Data Format · p. 56, Table 23 · p. 56, Table 24 · p. 57

FLP

• FLP warehouse operating costs and capacities and store demands are presented in tables, while warehouse-to-store transportation costs are formatted as a matrix.Source: D. Type-Tailored Retrieved Data Format · p. 57, Table 26 · p. 58, Table 27 · p. 58

• The framework translates formulated optimization models into Python code executable by solvers such as Gurobi to obtain solutions.Source: E. Generate Python Program Codes from… · p. 59
• Code-generation details vary by problem type, with examples for NRM and Others.Source: E. Generate Python Program Codes from… · p. 59

NRM

• For NRM, the prompt requests Gurobi code defining the objective, constraints, and decision variables, without added explanations.Source: E. Generate Python Program Codes from… · p. 59

Others

• For Others, the prompt directs the model to follow the user query, CSV schema, and abstract plan to produce complete, executable Gurobi code.Source: E. Generate Python Program Codes from… · p. 61
• The Others code prompt also requires specified imports, error handling, and robust reading of the dataset paths.Source: E. Generate Python Program Codes from… · p. 61

F. Example for Structurally Incorrect Model with a Correct Optimal Solution

• ORLM’s nutrition model selects from a restricted food subset rather than all 240 items, changing the feasible region and optimization problem.Its objective and nutritional constraints retain the labeled algebraic form but are evaluated only over the reduced set.Source: F. Example for Structurally Incorrect… · p. 63
• Structurally inequivalent models can yield the same optimum on particular instances, but this agreement does not imply that the generated formulation is correct.For example, agreement may occur when omitted items are not selected in a cost-minimizing solution.Source: F. Example for Structurally Incorrect… · p. 63
• Gemini 3 Pro’s generated revenue model uses a piecewise-linear Poisson expected-sales objective, omits per-product demand limits, and adds a fixed capacity constraint absent from the label.Its optimum can coincide with the labeled model for some datasets when demand limits are satisfied and the added capacity constraint is non-binding.Source: F. Example for Structurally Incorrect… · p. 64
• LEAN’s generated inventory model omits the labeled demand constraints, enlarging the feasible region, so a matching numerical optimum in the example is incidental.The label also limits each product’s fulfillment by its initial inventory and demand.Source: F. Example for Structurally Incorrect… · p. 65

G. Example Problem Instance in Air-NRM-CA

• The Air-NRM-CA query supplies ticket choices, OD demand, attraction values, and shadow-attraction ratios, then asks for an SBLP maximizing revenue across specified flights.The flights are identified by origin-destination pairs and departure times.Source: G. Example Problem Instance in… · p. 65
• The requested SBLP includes balance, scale, nonnegative, and selection constraints, with Eco-flexi tickets consuming two capacity units and Eco-lite tickets one.The prompt also requests decision variables and an objective function.Source: G. Example Problem Instance in… · p. 65