Large Language Models for Supply Chain Optimization

1 Introduction

• Supply chain operators often need to explain or revisit optimization decisions, but complex problems and limited optimization expertise can make understanding outcomes difficult.LLMs also face challenges with large-scale combinatorial problems, domain-specific questions, training costs, and mistakes or hallucinations.Source: §1 · pp. 1–2
• OptiGuide translates plain-language queries into optimization code, uses a solver to produce outputs, and has an LLM explain those outputs, including for what-if queries.For what-if questions, the framework modifies solver inputs and reruns the optimization.Source: §1 · p. 2
• OptiGuide uses in-context learning rather than costly model training, co-designs prompts with optimization code, and adds a safeguard to address output mistakes.The code is shaped to map efficiently to human questions while respecting prompt space constraints.Source: §1 · p. 2
• 93% average accuracy using GPT-4 across five scenarios; the paper also reports initial promising results from deployment for server placement in Microsoft Azure’s supply chain.The authors also introduce a benchmark and evaluation methodology for accuracy, within-scenario generalizability, and extrapolation to unseen scenarios.Source: §1 · pp. 2–3
• The authors present the benchmark and methodology as reusable for evaluating future approaches and identify smaller models and interactive optimization as future directions.They also discuss expanding LLM use beyond translation toward refining optimization outcomes.Source: §1 · p. 3

2.1 Decision Making in Supply Chains

• Supply chains span network design, planning, and execution, and optimization approaches must handle many constraints and objectives while producing good solutions in plausible running times.Common approaches include mixed-integer programming techniques and heuristics; supply-chain aspects may be optimized separately or jointly.Source: §2.1 · p. 4

2.2 Explainability

• Operators need to understand optimization outcomes to answer questions and respond to inefficiencies, but complexity and limited optimization expertise make that difficult.Obtaining insights can require planners, managers, and technical experts to coordinate and rerun optimization, making the process time-consuming.Source: §2.2 · pp. 4–5
• Explainability questions cover current solutions, historical trends, and what-if scenarios, while custom dashboards and fixed-query tools tend to cover limited use cases.Examples include cost breakdowns, demand changes, supplier choices, and transportation trade-offs.Source: §2.2 · pp. 4–5

2.3 Large Language Models

• LLMs can be adapted through fine-tuning or in-context learning; the latter uses examples in prompts without updating model parameters, while prompt length and privacy constrain deployment.LLMs can also provide inaccurate or hallucinated outputs, and domain-specific tools may be needed.Source: §2.3 · pp. 5–6

2.4 A Simple Example

• The example minimizes purchasing, roasting, and shipping costs while meeting retail demand across suppliers, roasting facilities, and retail locations.The model formulation represents supplier purchases and shipments of light and dark roast, with supplier capacities and retail demands.Source: §2.4 · pp. 6–7

Explainability.

• The example’s optimal plan splits sourcing across suppliers and facilities, motivating questions about demand changes, supplier constraints, costs, and assignments.For instance, facility 1 uses supplier 3, while facility 2 uses suppliers 1 and 2.Source: Explainability · p. 7

3.1 System Overview

• The LLM formulates code from a user question; a safeguard checks it before an application component processes it, and the LLM turns the resulting logs into a human-readable answer.The query may be automatically refined through multiple iterations until the desired output is achieved.Source: §3.1 · p. 8

3.1.1 Agents

• The coder prepares an in-context query, the safeguard checks and can trigger debugging of generated code, and the interpreter turns component outputs into a user-friendly response.Source: §3.1.1 · p. 8

3.1.2 Application Specific Components

• OptiGuide is modular: adapting it to a different application requires switching its application-specific components, such as a database, solver, documents, or optional helper functions.Helpers written by application engineers can simplify complex data-analysis workflows into a single function call.Source: §3.1.2 · p. 9

3.2 A Running Example

• For the shipping-prohibition query, the LLM adds a constraint, the solver resolves the plan, and OptiGuide returns a visualization and cost comparison without transferring domain data to the LLM.Source: §3.2 · pp. 9–10

4 Evaluation Benchmark

• The benchmark is intended to guide future work on supply-chain optimization with measures of accuracy, generalizability, and extrapolation capability.Accuracy is defined as the average success rate across scenarios, experiments, and question sets.Source: §4 · p. 10, §4.2 · p. 11

4.1 Scenarios and Data

• The benchmark uses five optimization scenarios and generates related question–answer sets with macros, including GPT-rephrased questions; its questions were created in-house and unseen during model training.Scenarios include facility location, multi-commodity network flow, workforce assignment, traveling salesman, and coffee distribution.Source: §4.1 · p. 11

4.2 Evaluation Methodology

• Because pass@k is too forgiving for explainability through code, the evaluation adapts a unit-test approach and judges final answers rather than exact code matches.It evaluates both in-distribution examples from the test question’s set and out-of-distribution examples from other sets, while varying example selection and set size.Source: §4.2 · pp. 11–12

4.3 Performance

• GPT-4 consistently outperforms text-davinci-003 in both evaluation settings; both models score higher in-distribution, and GPT-4 reaches 59% accuracy without examples.More examples improve accuracy across the board and narrow the gap between the models.Source: §4.3 · pp. 12, 14, Table 1 · p. 13
• Nearest-neighbor examples slightly improve in-distribution performance, but random selection performs better out-of-distribution when example sets contain more than one example.The authors suggest similarity-based selection may overfit, whereas random selection provides more diverse examples.Source: §4.3 · p. 14

5.1 The Azure Supply Chain

• Azure’s Intelligent Fulfillment System assigns and ships servers from warehouses to data centers within a broader cloud supply chain.The wider supply chain also includes demand forecasting, strategic foresight, hardware semantic search, fulfillment planning, and document management.Source: §5.1 · p. 14

Main decisions.

• For each capacity demand, the system chooses a hardware supplier, dock-date, and data-center location while minimizing cost under capacity, preference, and compatibility constraints.The problem is a roughly 500 MB mixed integer program solved hourly with Gurobi.Source: Main decisions · p. 14
• Planners rely on engineers for insight because the system’s complexity and global scale obscure the reasoning behind decisions, often prompting repeated what-if discussions.Planners use business context to confirm whether optimization outcomes meet business needs or to override them.Source: Main decisions · pp. 14, 16

5.2 Deploying OptiGuide for Azure Supply Chain

• The deployment uses a multi-user front end, an agent service connected to Azure OpenAI, and virtual machines hosting IFS; VM data and solver solutions are refreshed hourly in rotation.GPT-4 is used as the LLM.Source: §5.2 · p. 16

5.3 Preliminary Feedback and Results

• Users highlighted what-if support as a potential source of greater planner autonomy and reduced engineering on-call burden; the paper illustrates interactions with an anonymized screenshot.Before OptiGuide, one what-if question required more than three coordinating operators and an on-call engineer to inspect the plan output.Source: Figure 9 · p. 15, §5.3 · p. 16
• OptiGuide’s preliminary Azure evaluation reports more than 90% accuracy in-distribution, consistent with the benchmark results.Planners and engineers also gave positive preliminary feedback.Source: §5.3 · p. 16

6.1 Current Limitations

• Users must phrase questions precisely because ambiguous timing language can lead the LLM to misunderstand the request and produce wrong code.“Earlier” could refer to today, the planned date, or the deadline.Source: §6.1 · p. 17
• OptiGuide depends on application-specific components such as database schemas and helper functions, which can require substantial engineering and still have production-setting gaps.Source: §6.1 · p. 17
• Code that runs smoothly can still be completely wrong, including because of string-matching mistakes.Source: §6.1 · p. 17
• The LLM performs well on seen questions but struggles with questions absent from the examples.Source: §6.1 · p. 17
• The benchmark evaluates quantitative questions, excluding visualization queries, and is limited to Python and the Gurobi solver.Source: §6.1 · p. 17

6.2 Future Directions

• Future directions include human feedback, smaller or hybrid models, fine-tuning for unseen questions, and interactive optimization with more comprehensive safeguards.Source: §6.2 · p. 17

• The partial optimization formulation assigns and ships servers from the warehouse to data centers.Source: A Intelligent Fulfillment System · p. 22

• Binary variables indicate whether each demand docks on a particular day or row, or is fulfilled using a particular supplier.Source: §A.1 · p. 22
• A binary variable records whether a demand docks at a particular data center on a particular day.Source: §A.1 · p. 22
• A nonnegative variable represents whether demand docks on a given day using a given supplier.Source: §A.1 · p. 22

• Each demand docks on a single day, and a data-center docking is allowed only when the selected day matches that docking day.Source: §A.2 · p. 22
• Daily capacity restrictions limit the docking volume that sets of data centers can handle.Source: §A.2 · p. 22
• Each demand must use a single supplier, and a row is selected only if a supplier has been found.Source: §A.2 · p. 22
• Supply consumed from a pool must be available at the consumption time, accounting for the supply’s lead time.Source: §A.2 · p. 23
• Undesirable or disallowed demand-supplier combinations can be explicitly blocked.Source: §A.2 · p. 23

A.3 Objective

• The objective minimizes total cost across early or late docking, unfulfilled demand, shipping, and other components.The objective passage describes these components as an aggregate.Source: §A.3 · p. 23

B Engineering Details

• Figure 11 presents an OptiGuide screenshot for Azure IFS with intermediate results for illustration.Source: B Engineering Details · p. 23

B.1 Useful Tricks

• SQL storage may make optimization inputs and outputs easier for LLMs to use and make the system more explainable.Source: §B.1 · p. 23
• Logical simplification can reduce simple prompt-related mistakes, while intermediate outputs help check consistency and support debugging.Source: §B.1 · p. 23

B.2 Failed Attempts

• CoT did not help write complex code; the authors report that application-specific helper functions outperformed it.They hypothesize that one reasoning-chain mistake can make the whole response wrong because self-correction is difficult.Source: §B.2 · p. 24
• Overly complex or specific prompts can be misunderstood or overfit, limiting handling of varied questions.Source: §B.2 · p. 24

C.1 Code

• The code example builds and solves a Gurobi model with an objective, decision variables, and flow, supply, and demand constraints.The supplied fragments show objective setup, model optimization, and constraints for these quantities.Source: §C.1 · pp. 24–26

C.2 Question and Ground Truth Macros

• Demand-change macros increase one cafe’s demand by a chosen percentage or double demand across all cafes.Source: §C.2 · pp. 26–27
• Constraint macros encode exclusive roastery-cafe assignments, relative production or shipping quantities, and a single-supplier restriction.Source: §C.2 · pp. 27–28
• Supplier scenarios change capacity or shipping cost, or prohibit a supplier-to-roastery route through data or constraint edits.Source: §C.2 · pp. 27–28
• Figure 10 compares training-example selection methods in GPT-4 out-of-distribution evaluation; Figure 11 shows Azure IFS intermediate results.Source: Figure 10 · p. 29, Figure 11 · p. 30