PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-Scale Optimization Model Auto-Formulation: Harnessing LLM Flexibility via Structured Workflow

Kuo Liang, Yuhang Lu, Jianming Mao, Shuyi Sun, Chunwei Yang, Congcong Zeng, Xiao Jin, Hanzhang Qin, Ruihao Zhu, Chung-Piaw Teo

arXiv:2601.09635v3cs.AIcs.LG

TL;DR

Large-scale optimization modeling requires extracting information from data and formulating intricate problems, while model accuracy can decline as inputs grow. LEAN-LLM-OPT is proposed to address these challenges, but reported results show that leading models can fall below 50% accuracy beyond 800 input tokens.

  • Problem

    Extracting information from data and formulating intricate optimization problems is challenging, and fully leveraging current techniques faces a major bottleneck.

  • Method

    The paper proposes LEAN-LLM-OPT as an approach to address these challenges.

  • Results

    Modeling accuracies decrease as input size grows, with Gemini 3 Pro and GPT-5.2 both dropping below 50% beyond 800 input tokens.

  • Takeaways & Limitations

    LEAN-LLM-OPT provides an affirmative answer to addressing the paper's stated challenges.

  • Takeaways & Limitations

    The work focuses mainly on large-scale inputs, while allowing output scale to vary, and identifies a need for more enforceable and controllable mechanisms.

Abstract

from arXiv · show

Large-scale optimization is a key backbone of modern business decision-making. However, building these models is often labor-intensive and time-consuming. We address this by proposing LEAN-LLM-OPT, a LightwEight AgeNtic workflow construction framework for LLM-assisted large-scale OPTimization auto-formulation. LEAN-LLM-OPT takes as input a problem description together with associated datasets and orchestrates a team of LLM agents to produce an optimization formulation. Specifically, upon receiving a query, two upstream LLM agents dynamically construct a workflow that specifies, step-by-step, how optimization models for similar problems can be formulated. A downstream LLM agent then follows this workflow to generate the final output. The agentic workflow leverages common modeling practices to structure the modeling process into a sequence of sub-tasks, offloading mechanical data-handling operations to auxiliary tools. This reduces the LLM's burden in planning and data handling, allowing us to exploit its flexibility to address unstructured components. Extensive simulations show that LEAN-LLM-OPT, instantiated with GPT-4.1 and the open source gpt-oss-20B, achieves strong performance on large-scale optimization modeling tasks and is competitive with state-of-the-art approaches. In addition, in a Singapore Airlines choice-based revenue management use case, LEAN-LLM-OPT demonstrates practical value by achieving leading performance across a range of scenarios. Along the way, we introduce Large-Scale-OR and Air-NRM, the first comprehensive benchmarks for large-scale optimization auto-formulation. The code and data of this work is available at https://github.com/CoraLiang01/lean-llm-opt.

1. Introduction

Large-scale optimization modeling remains difficult because practical instances combine lengthy, heterogeneous inputs with many variables and separately stored datasets. LEAN-LLM-OPT addresses these challenges through structured agentic workflows, auxiliary data-handling tools, and new large-scale benchmarks, achieving strong modeling results across benchmarks and a Singapore Airlines application.

  • Problem Setting: Existing fine-tuned and commercial models can lose modeling accuracy as large-scale input size grows.Most models decrease noticeably as input size increases; Gemini 3 Pro and GPT-5.2 fall below 50% beyond 800 input tokens.
  • Problem Setting: Large-scale instances require separately stored datasets and often contain many variables, making direct embedding of all input data impractical.The paper defines large-scale inputs by separately stored data and focuses on models with at least 100 variables, while allowing output scale to vary.
  • Challenges: Large-scale optimization combines heterogeneous descriptions and datasets, while training pipelines struggle to generalize across changed rows, columns, or dataset splits.Small input changes can create different problem instances and require new labeled input-output pairs, making generalized fine-tuning unrealistic.
  • Approach: LEAN-LLM-OPT constructs step-by-step agentic workflows that decompose complex modeling and reasoning into structured subtasks.The framework combines agent collaboration with worked examples and supports lengthy descriptions and external data inputs.
  • Benchmarks: The framework introduces Large-Scale-OR, a 101-entry benchmark emphasizing medium- and large-size instances across diverse application domains.Medium-size problems comprise 26% and large-size problems 50% of the dataset; Ref-Data supplies semantically and scale-wise different reference entries.
  • Results: Both model variants exceed 76% overall accuracy on Large-Scale-OR and substantially outperform ORLM, Gemini 3 Pro, and GPT-5.2.The paper also reports leading or consistently small-gap performance in Air-NRM benchmarks and practical performance in a Singapore Airlines application.
  • Results: Ablation results indicate that both the agentic workflow and data-handling tools are important for high modeling accuracy.The framework is designed to offload mechanical data operations while preserving LLM flexibility for unstructured components.

2. Dataset Compilation

The paper constructs Ref-Data for problem classification and formulation learning, then builds Large-Scale-OR as a diverse large-scale testing benchmark. It also compares benchmark novelty and input length to assess generalization across semantic and scale dimensions.

  • Reference dataset construction: Ref-Data contains 96 instances, including 78 small-scale and 18 large-scale problems, and focuses mainly on linear and mixed integer linear programming.The approach is described as general beyond these model classes.
  • Reference dataset construction: The dataset spans network revenue management, resource allocation, transportation, assignment, facility location, minimum-cost flow, and mixed problem types across practical domains.Instances are sourced from textbooks, Kaggle, industry cases, and practical studies, with missing parameters supplemented by standardized rules.
  • Reference dataset construction: Ref-Data instances encode a user query, problem type, data category, relevant data details, and an expert-generated optimization model.Small-scale examples are retained because they support problem classification.
  • Large-Scale-OR testing dataset: Large-Scale-OR contains 101 testing instances, half of which are large-size problems with at least 100 variables.Each instance includes a natural-language query, a ground-truth mathematical model, and an optimal value.
  • Large-Scale-OR testing dataset: Large-Scale-OR covers medical, e-commerce, and supply-chain applications, including common and mixed problem categories such as network revenue management and resource allocation.The benchmark is designed to evaluate applicability across diverse real-world scenarios.
  • Textual comparisons: Semantic-similarity analysis finds Ref-Data and the benchmarks toward the lower end of the similarity scale, while token-count analysis shows Ref-Data is much smaller and Large-Scale-OR much larger than other benchmarks.These comparisons address benchmark novelty and variation in input scale.

3. LEAN-LLM-OPT Framework

LEAN-LLM-OPT constructs optimization models through a coordinated workflow of classification, workflow generation, and model generation agents. The framework combines step-by-step demonstrations, retrieval tools, and structured reasoning to reduce modeling burden and support broader problem coverage.

  • Framework overview: LEAN-LLM-OPT builds large-scale optimization models through collaboration among three LLM agents.The framework receives a problem description and datasets, then coordinates classification, workflow generation, and model generation.
  • Problem classification: The classification agent identifies the input problem type before workflow construction begins.It reasons over the query and assigns either a predefined problem type or “Others.”
  • Workflow generation: The workflow generation agent retrieves similar examples from Ref-Data and builds a step-by-step modeling workflow for the model generation agent.The downstream agent follows this workflow to produce the optimization formulation.
  • Retrieval-assisted reasoning: FileQA uses similarity search to retrieve closely related problem instances, while the workflow demonstrations guide the agent’s reasoning and retrieval actions.The described example retrieves the five most similar instances and returns their descriptions and associated information.
  • Design implications: The framework supports type-agnostic workflows, reduces agent planning and data-handling burden, and can generalize across different problem instances and domains.Its plug-and-play strategy also permits pairing with newer LLMs without task-specific fine-tuning.

4. Numerical Simulations

The simulations evaluate LEAN-LLM-OPT on large-scale and small-scale optimization benchmarks using optimal-value accuracy and exact-match formulation accuracy. LEAN-LLM-OPT achieves the strongest reported results on Large-Scale-OR and across the small-scale benchmarks, while performance varies with input length and output size.

  • Benchmarks: Large-Scale-OR is used as the principal large-scale benchmark, alongside IndustryOR, NL4OPT, MAMO Easy, and MAMO Complex for small-scale evaluation.Large-Scale-OR targets instances whose data are stored separately in datasets; the small-scale benchmarks cover multiple optimization problem types and application domains.
  • Results on Large-Scale-OR: LEAN-LLM-OPT achieves the highest overall Large-Scale-OR accuracies, while its GPT-4.1 and gpt-oss-20B base models each achieve less than 26%.Gemini 3 Pro is the strongest competing method reported, with 50.5% exact-match accuracy; the LEAN variants substantially exceed competing methods.
  • Results on Large-Scale-OR: LEAN-LLM-OPT remains competitive when all problems use the type-agnostic workflow, although both agnostic variants perform worse than their type-tailored counterparts.The agnostic variants still outperform almost all other competing methods, supporting both type-tailored workflow effectiveness and type-agnostic robustness.
  • Input and Output Effects: LEAN-LLM-OPT sustains longer inputs, but its performance drops as the number of output variables increases.The analysis also reports that some other methods improve as output-variable counts increase, suggesting that input size contributes to large-scale modeling difficulty.
  • Small-Scale Benchmarks: LEAN-LLM-OPT (GPT-4.1) performs best across the small-scale benchmarks, while both LEAN variants outperform ORLM and OptiMUS on every benchmark.The results are reported for NL4OPT, IndustryOR, and MAMO Easy, with additional comparisons involving GPT-5.2 and Gemini 3 Pro.

5. Case Study: Singapore Airlines Choice-Based Revenue Management

The Singapore Airlines case evaluates LEAN-LLM-OPT on choice-based revenue management, including fare capacity allocation and network planning, with ablations isolating workflows and data-handling tools. Across these tasks, the framework achieves leading modeling and revenue performance while remaining stable as problem complexity increases.

  • Case study setup: The case study tests LEAN-LLM-OPT on generalized SBLP problems involving fare capacity allocation and network planning for Singapore Airlines.The setting includes multiple datasets and more complicated target models than earlier simulations.
  • Testing dataset: Air-NRM contains 15 fare-type capacity-allocation instances and 21 network-planning instances.The benchmark is introduced for the Singapore Airlines use case.
  • Modeling accuracy: LEAN-LLM-OPT achieves leading modeling accuracy, while its gpt-oss-20B variant is comparable to Gemini 3 Pro.GPT-4.1 continues to achieve the highest modeling accuracy among the compared methods.
  • Revenue performance: GPT-4.1’s optimality gap remains below 14%, compared with over 55% for tools-only across most instances.The gpt-oss-20B variant stays below 24% in all but two cases, while GPT-5.2 often exceeds 40% and sometimes 80%.
  • Case study outcome: Overall, LEAN-LLM-OPT maintains strong revenue performance under SBLP and its variants as problem size and complexity increase.The Singapore Airlines examples also provide profitable flights, fare-type capacity configuration, and optimal revenue.

6. Concluding Remarks

The paper concludes that LEAN-LLM-OPT addresses the difficulty of applying LLMs to large-scale optimization by structuring planning and data handling through an agentic workflow. The findings also motivate broader workflow-based applications and more enforceable mechanisms for controlling probabilistic LLM behavior.

  • Conclusion: LEAN-LLM-OPT provides an affirmative answer to whether LLMs can formulate large-scale optimization models.The paper targets settings where data are too large to embed directly and long-form reasoning may be difficult.
  • Conclusion: Its agentic workflow reduces burdens from planning and data handling while leaving less-standardized components to the LLM.The plug-and-play design is intended to improve practicality and portability.
  • Future directions: Agentic workflows may extend beyond optimization by translating structured expertise into customizable workflow skeletons.The paper suggests this combination may improve effectiveness and efficiency across applications.
  • Future directions: The results suggest older-generation and smaller open-source models can address challenging formulation tasks when reasoning is appropriately elicited and guided.Such models may also offer lower cost or privacy advantages, motivating further study.
  • Limitations and future work: Because the workflow is introduced mainly through semantic instructions, probabilistic LLM behavior can make system behavior unstable.The paper proposes exploring more enforceable and controllable mechanisms, including formal structure and constraint satisfaction.

A.1. Application Domains of the Problem Instances

Large-Scale-OR organizes large-scale optimization instances into categories based on their mathematical formulations and application characteristics. The taxonomy covers canonical models, well-known archetypes, and hybrid or non-standard problems.

  • Canonical categories: NRM instances contain demand, price, or product information, while RA instances contain resource capacities, product values, or resource consumption.These rules identify the relevant application category from dataset contents.
  • Canonical categories: TP instances describe customer demand, transportation costs, or supply capacity, whereas FLP instances describe potential facilities and associated information.The categories distinguish shipping-cost problems from facility-placement problems.
  • Canonical categories: AP uses cost matrices whose rows represent resources and columns represent tasks, with each cell giving an assignment cost.The resulting problem seeks an optimal one-to-one matching that minimizes total assignment cost.

A.2. Parameters Imputation

The supplementary material describes rule-based parameter imputation and data-generation procedures for network revenue management, transportation, and facility-location instances. These procedures transform raw or incomplete data into optimization inputs such as demand, cost matrices, coordinates, and setup costs.

  • Overview: The parameter-imputation procedures generate missing optimization inputs from raw datasets using domain-specific transformations.The appendix presents separate procedures for NRM, transportation, and facility-location data.
  • Network revenue management: NRM generation reads sales data, computes demand and initial inventory, and outputs product names, revenue, demand, and inventory.Inventory is rounded up to a multiple of 10 in the illustrated helper function.
  • Transportation problem: Transportation-cost generation uses supplier and customer locations, geographic coordinates, and a fixed cost per mile to construct a cost matrix.The resulting matrix contains transportation costs between each supplier and customer.
  • Transportation problem: Customer demand for transportation instances is generated by filtering the original dataframe according to the regions represented in the cost matrix.The procedure is presented alongside the transportation-cost construction code.
  • Facility location problem: Facility-location data are converted into geographic coordinates, and facility setup costs are simulated within a predefined cost range.The generated costs support minimizing the total cost of opening and operating facilities.

A.4. Sample Problem Instance in Large-Scale-OR

The section presents a mixture-production instance in Large-Scale-OR, where 141 widgets consume limited resources and Widget3 generates a capped, marketable byproduct. The formulation maximizes production profits plus CatalystX sales while accounting for disposal costs and resource limits.

  • Problem instance: The instance models a precision facility producing Widget1 through Widget141, with labor, Material A, Material B, and unit-profit data for each product.Widget3 additionally generates CatalystX as a byproduct.
  • Byproduct economics: CatalystX is produced at 5 kg per Widget3, sold for $300 per kilogram up to 1500 kg monthly, and otherwise disposed of at $200 per kilogram.
  • Resource limits: Monthly labor is limited to 5,000 hours, Material A to 24,000 kg, and Material B to 15,000 kg.
  • Objective: The optimization determines widget production quantities and CatalystX sales quantities to maximize total profit.
  • Mathematical formulation: The ground-truth model uses a linear objective over production variables and nonnegativity constraints for x_i and s.The displayed objective assigns coefficients to the widget variables, while the resource constraints use product-specific coefficients.

B. On the Comparison between the Classification Agent and RAG

The comparison uses a vehicle-inventory query to show that classification and simple RAG can retrieve the same examples yet assign different problem types. The classification agent identifies the correct type in this case.

  • Query: The query asks a Norwegian car dealership to choose daily vehicle orders that maximize benefit under vehicle-specific and total inventory limits.
  • Query: The decision variables represent daily quantities ordered for each vehicle type, requiring a balance between benefits and inventory constraints.
  • Retrieval: The classification agent and simple RAG framework retrieve identical similar problem instances for this query.
  • Classification: The classification agent returns “Resource Allocation,” whereas the RAG framework returns “Inventory Management Problem.”
  • Comparison result: “Resource Allocation” is the correct problem type, illustrating the classification agent’s ability to identify the query accurately.

C. Demo Problem Used in Workflow Generation

The demo problem used for workflow generation is a shelf-allocation model that combines product values and space requirements with multiple shelf capacities. Its target formulation maximizes placed product value subject to capacity and nonnegativity constraints.

  • Demo setup: The demo assigns products to shelves using decision variables x_ij, representing units of product i placed on shelf j.
  • Retrieved example: The related Ref-Data example describes allocating Sony products across retail shelves using product values, space requirements, and shelf-specific capacity information.
  • Target model: The optimization objective is to maximize total product value while ensuring Sony-product space stays within brand-specific limits.
  • Input data: The supplied products.csv data include product values and space requirements, while capacity.csv provides platform capacities.Examples include Sony Alpha Refrigerator, Sony Bravia XR, Sony HT-A7000 Soundbar, and Sony PlayStation Soundbar.
  • Constraints: The target model lists capacity constraints and non-negativity constraints for the shelf-allocation variables.

D. Type-Tailored Retrieved Data Format

The workflow uses type-tailored data formats: assignment and transportation problems rely on matrices or tables, while facility-location data combine operating costs, capacities, demands, and transportation costs. These formats organize retrieved data for subsequent formulation.

  • Assignment Problem: The assignment-problem data are most efficiently stored as a matrix, with cost.csv providing a complete cost matrix C.
  • Transportation Problem: Transportation-problem data store supply and demand in tables and costs between supply and demand locations in a matrix.
  • Facility Location Problem: The facility-location example stores warehouse operating costs and capacities together with store demand in tables.
  • Facility Location Problem: Transportation costs between warehouses and stores are represented as a matrix A_ij.

E. Generate Python Program Codes from the Optimization Models

The framework translates formulated optimization models into executable Python code, using Gurobi to define and solve the resulting models. Code-generation prompts specify complete implementations that read datasets and include variables, objectives, constraints, imports, and error handling.

  • The framework translates optimization models into executable programming code and uses solvers such as Gurobi to obtain solutions.
  • Representative implementations use few-shot examples and structured inputs such as the user query, schema, abstract model plan, dataset address, and examples.
  • Generated code defines decision variables, objective functions, constraints, and solver instructions for mathematical optimization models.
  • The code-generation prompt requires complete Gurobi Python code with necessary imports, dataset reading, full model implementation, and error handling.

F. Example for Structurally Incorrect Model with a Correct Optimal Solution

The examples show that generated optimization models can differ structurally from ground-truth formulations while still producing the same optimum for particular instances. Such agreement depends on instance-specific conditions and does not establish formulation correctness.

  • Structurally incorrect formulations can produce correct optimal solutions for particular optimization instances.
  • In the nutrition example, ORLM restricts decisions to a subset of food items rather than the full set of 240 labeled decision variables.
  • The reduced ORLM formulation changes the feasible region and optimization problem, yet its optimum may coincide with the full formulation when omitted items are not selected.
  • Coincident optima are incidental or instance-specific and therefore do not imply that the generated formulation is correct.
  • Other examples omit demand constraints or add a global capacity constraint, making generated formulations structurally non-equivalent to their labels.
Loading 2601.09635v3…