Source-linked AI summary
OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen, Gaoyang Dai, Xing He, Aiqian Li, Fengchi Yuan, Aochuan Chen, Chenyi Zi, Yuhan Li, Hongchuan Zeng, Yimo Yan, Nuo Chen, Jie Fu, Zuo-Jun Max Shen, Yong-Hong KUO, Jia Li
TL;DR
Existing OR benchmarks often start from self-contained tasks, leaving limited evaluation of how agents translate original business materials into professionally grounded solutions. OR-LICENSE evaluates agents on real engagement materials, with ORSIM client interaction and hidden assessment against feasibility and client-accepted expert references. Several models exceed the expert quality reference on feasible deliveries, but none matches it in DAQ.
Problem
Many OR benchmarks begin with self-contained descriptions or predefined tasks, while professional OR work requires translating original business materials and acquiring client-held information.
Source: Three gaps to professional OR practice · p. 2Method
OR-LICENSE evaluates agents on de-identified industrial OR engagements using ORSIM for client facts and hidden feasibility and expert-relative quality checks against client-accepted expert solutions.
Source: Three gaps to professional OR practice · pp. 2–3Results
Several models exceed the expert quality reference on feasible deliveries, but none matches the expert in Deployment-Adjusted Quality.
Source: §5 · p. 11Takeaways & Limitations
The findings motivate sustained information-gap resolution, faithful problem formulation, validation, and evidence-linked self-improvement in end-to-end OR agents.
Source: §4.2 · p. 10, §4.3 · p. 10, §5 · p. 11Agents’ information seeking remains concentrated before substantive solver development, and they lack a consistent feedback-responsive process for resolving gaps as solutions evolve.
Source: §4.2 · p. 10
Abstract
from arXiv · showhide
Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.
1 Introduction
Operations research depends on experts translating incomplete business materials into executable decision processes. The paper asks whether LLM agents can perform this work.
- OR deployment requires experts to elicit requirements, formulate problems, select solution strategies, and implement solvers from incomplete business materials.Source: §1 · p. 1
Three gaps to professional OR practice
The paper identifies three gaps in evaluating whether LLMs can perform professional OR work: realistic task complexity, client-information acquisition, and grounding in professional delivery. OR-LICENSE addresses them with real engagement materials, ORSIM, and client-accepted expert references.
- Existing benchmarks may miss operational complexity, the work of acquiring client-held information, and comparison against client-accepted expert deliveries.The paper distinguishes task realism, information acquisition throughout problem solving, and professional-delivery grounding as separate evaluation gaps.Source: Three gaps to professional OR practice · p. 2
- OR-LICENSE preserves realistic requirements, artifacts, ambiguity, and information boundaries from paid industrial engagements, with ORSIM disclosing client facts but not OR expertise.The benchmark compares agents with original expert solutions accepted by clients before benchmark construction.Source: Three gaps to professional OR practice · pp. 2–3
- The dataset contains 31 instances across 10 industries, derived from commissioned projects representing $3.68 million in total project value.Source: Three gaps to professional OR practice · p. 3
- The benchmark evaluates feasibility and expert-relative quality while analyzing client information acquisition, failure patterns, and linked hypothesis–experiment–revision cycles.Source: Three gaps to professional OR practice · p. 3
2 Related Work
Prior OR work includes agents for formulation, solver-code generation, and execution-based revision, while many evaluations start from self-contained descriptions or predefined tasks. OR-LICENSE uses original paid-engagement materials and client interaction within an executable-solution workflow.
- Many OR evaluations start with self-contained descriptions or predefined tasks, while OR-Space and ORAgentBench extend evaluation to executable environments and multi-stage workflows.Source: §2 · p. 3
- OR-LICENSE builds on multi-stage evaluation by using original paid-engagement materials and allowing client queries throughout formulation, solving, and refinement.Source: §2 · p. 3
- Professional-work benchmarks assess economically valuable deliverables and sustained execution across realistic scenarios; OR-LICENSE specializes this approach to OR feasibility and expert-relative quality.Its expert references are original solutions accepted by clients before benchmark construction.Source: §2 · p. 3
- The framework combines expert-curated tasks and reference solutions, ORSIM interaction, and hidden evaluation of feasibility and expert-relative solution quality.Source: Figure 2 · p. 4
3 Benchmark Construction
OR-LICENSE defines real-world OR tasks, curates de-identified engagement materials and references, and gives agents access to client facts through ORSIM. It evaluates feasibility, expert-relative quality, and agent trajectories.
3.1 Task Definition
- Each instance defines a decision structure, business rules, objective, and official test cases, with hidden verifiers checking hard constraints and required relationships among variables.Agents receive an initial brief and input files, may query ORSIM, and submit executable decisions evaluated for feasibility and independently recomputed objective quality.Source: §3.1 · p. 4
- The benchmark represents each scenario as a set of OR instances, with each instance specifying its decision space, objective, verifier, and official cases.Source: §3.1 · p. 4
3.2 Data Curation
- Experts curated 31 instances across 10 scenarios, de-identifying original requirements, data, and deliverable formats while preserving the agent-facing information boundary.Hidden evaluators and client-accepted expert solutions are checked for consistency with original requirements and agent-facing materials.Source: §3.2 · p. 5
3.3 ORSIM
- ORSIM discloses eligible client-held facts throughout the workflow but withholds OR expertise, leaving formulation, algorithm selection, and implementation to the agent.A deterministic controller enforces ownership, visibility, and disclosure rules between fact selection and response generation.Source: §3.3 · pp. 5–6
3.4 Metrics
- DAQ gates expert-relative quality by feasibility, assigning zero to infeasible instances; the reliability–quality profile examines sensitivity to low scores as α increases.The benchmark also reports strict feasibility and conditional quality, and evaluates the profile on a prespecified 21-value α grid.Source: §3.4 · pp. 6–7
3.5 Trajectory Analysis Methodology
- Trajectory analysis annotates client questions by workflow stage, scientific inquiry as HCAEU chains, and confirmed errors by their point of origin.The error categories include understanding, data mapping, representation, constraints, dynamics, uncertainty, objectives, and implementation.Source: §3.5 · p. 7
4 Evaluation
Across eleven frontier models, feasibility and Deployment-Adjusted Quality remain distinct from objective quality on feasible deliveries, while research-chain completion stays below half of opportunities. Agents concentrate client questions early and exhibit prominent formulation errors, alongside persistent implementation failures.
4 Evaluation
- Feasibility and DAQ average five runs within each task and then weight the 31 task means equally across eleven frontier models.Reported spreads are means of within-task sample standard deviations, not standard errors of the benchmark mean.Source: §4 · pp. 7–8, Table 1 · p. 8
4.1 RQ1: Cross-Model Comparison and Performance Against Human Experts
- 86.55% DAQ for Grok 4.6 is the highest primary-comparison score, still 13.45 points below the expert reference; infeasible deliveries account for about 86% of that deficit.Several models exceed the expert quality reference on feasible deliveries, but every model's DAQ remains below it; the gap decomposes into 11.61 points from infeasible deliveries and 1.84 from net objective shortfall on feasible ones.Source: §4.1 · p. 8, Table 1 · p. 8
- 24%–45% of research opportunities have complete HCAEU chains across models, and conditional stage quality and DAQ produce different model rankings.Qwen3.8-Max has the highest conditional stage quality but a 77.97% DAQ and completes chains in only 27% of opportunities; incomplete chains are not proof that issues remain unsolved.Source: §4.1 · pp. 8–9, Figure 4 · p. 9
4.2 RQ2: Client Information Acquisition Across the OR Workflow
- 86.7%–98.3% of questions occur at S1 across models, although later clarification can resolve concrete issues after solution construction begins.Confirmed-resolution proportions are 10.6%–37.0% in S1, versus approximately 56% for DeepSeek-V4.1-Flash in S4 and 71% for Hy4 Preview in S5.Source: Figure 5 · p. 9, §4.2 · p. 10
4.3 RQ3: LLM Failure Patterns in End-to-End OR Projects
- Formulation errors are prominent across models; implementation and delivery is the largest annotated error category for only two of eleven models.Objective/preference and uncertainty/information-structure errors together make up 69.0% of Grok 4.6's error events and 70.5% of Muse Spark 1.3's; implementation failures also persist for GPT-5.6 Sol and Hy3.Source: Figure 5 · p. 9, §4.3 · p. 10
5 Conclusion
OR-LICENSE evaluates agents on original client materials across 31 instances and finds that none of eleven models matches the expert reference in DAQ. Its conclusion motivates better requirement interpretation, formulation, validation, information-gap resolution, and evidence-linked self-improvement.
5 Conclusion
- None of eleven evaluated models matches the expert reference in DAQ, although several exceed it in objective quality on feasible instances.The benchmark covers 31 instances across 10 industrial scenarios using original client materials, ORSIM, hidden feasibility checks, and client-accepted expert references.Source: §5 · p. 11
AI Use Statement
- The authors used generative AI for translation, dataset preparation, analysis, figures, editing, software, and literature discovery, with four human reviewers verifying outputs.The authors take responsibility for the final content and artifacts.Source: AI Use Statement · p. 12
A Metric Definitions and Analysis
The appendix specifies instance-level feasibility and aggregation, objective normalization, and properties of DAQ and the Reliability–Quality Profile. It defines strict delivery criteria and shows how quality and coverage interact in the metrics.
A.1 Instance-Level Feasibility and Aggregation
- An instance is feasible only if the final submission passes the hidden verifier on every official test case; passing a subset yields zero feasibility.Feasibility is aggregated across instances with equal weight, regardless of each instance’s number of test cases.Source: §A.1 · p. 17
A.2 Objective Normalization and Expert-Relative Quality
- Expert-relative quality compares agent and expert objective values after applying the same positive, order-preserving transformation to both.A score of 1 matches the expert objective; scores above or below 1 indicate better or worse objective quality, respectively.Source: §A.2 · p. 17
A.3 DAQ Properties
- DAQ assigns zero quality to infeasible deliveries, and the expert reference has DAQ = 1.When no instance is feasible, DAQ is zero while conditional quality is reported as N/A.Source: §A.3 · pp. 17–18
A.4 Reliability-Quality Profile
- DAQ averages instance scores, allowing quality gains on completed instances to compensate linearly for incomplete coverage.Because expert-relative quality has no explicit upper cap, a few high-quality solutions can offset failures elsewhere; increasing α strengthens sensitivity to low scores.Source: §A.4 · p. 18
- For coverage fraction r and equal feasible-instance quality q, matching the expert reference requires q = r^-α.Larger α therefore requires a greater quality gain to offset the same reduction in complete-delivery coverage.Source: §A.4 · p. 18
B Trajectory Analysis Protocol
The protocol annotates client questions, trajectory errors, and evidence-linked research opportunities, while ORSIM uses controlled evidence selection to disclose client facts. Its default flat selector keeps full rules available during selection but sends only authorized evidence to the responder.
B Trajectory Analysis Protocol
- Client questions are assigned to six workflow stages, from before solver code through after self-checks pass and before final submission.The protocol also records whether agents ask again after observing solver feedback when another decision opportunity remains.Source: B Trajectory Analysis Protocol · p. 19
- Trajectory errors are classified by origin into eight categories, from problem understanding and formulation to implementation and delivery.Category profiles count events, so one trajectory may contribute multiple errors; simulator and infrastructure failures are tracked separately.Source: B Trajectory Analysis Protocol · p. 19
- Research opportunities are consequential uncertainties that agents can potentially reduce through investigation, and experts enumerate them from full recorded trajectories.A complete HCAEU chain links a testable hypothesis and explanation to a diagnostic experiment, evidence interpretation, and evidence-grounded update within the same issue and branch.Source: B Trajectory Analysis Protocol · p. 19
- Stage-quality scores are evaluated conditional on complete HCAEU opportunities, while expert annotations require identifiable supporting evidence.Present stages receive scores from 1 to 3; chain completion is separately established from evidence in a single branch.Source: B Trajectory Analysis Protocol · pp. 19–20
C Details and Reliability of ORSIM
- ORSIM identifies the evidence for each simulated client reply before generation, separating evidence selection from response generation to support controlled disclosure.A deterministic controller validates selected rule identifiers and applies ownership and visibility policies; the responder receives only authorized facts.Source: §C.1 · pp. 20–21
C.1 Simulator Architecture and Selector Variants
- The simulator tests flat, summary-plus-detail, and summary-plus-KG selectors, with different trade-offs in full-text access and intermediate representations.The flat selector is the default: full rules inform selection, while the responder still receives only selected, authorized evidence.Source: §C.1 · p. 21
C.2 Reliability Analysis
The section tests whether ORSim replies are accurate, policy-compliant, complete, and consistent across repeated runs. The flat selector performs best among the selector variants, while direct answers from the full statement show that factual support alone does not ensure reliable, compliant replies.
Evaluation
- Experts assessed 204 standard and adversarial queries across five runs, judging factual-claim support, reference fulfillment, and disclosure compliance; success requires all three.The three selector variants share the same catalog, policy, responder, and decoding settings; the full-statement baseline is evaluated separately.Source: §C.2 · pp. 21–24
- Across four annotation dimensions, raw agreement ranges from 97.58% to 99.31% and Krippendorff’s α from 0.831 to 0.920; the agreement analysis excludes the full-statement baseline.Evidence support covers 5,293 factual claims, while each other dimension covers 3,060 replies.Source: §C.2 · p. 24, Table 2 · p. 25
Results
- Flat-selector faithfulness is 98.71% on standard and 96.55% on adversarial queries, with disclosure violations in 0.34% and 0.24% of replies, respectively.By contrast, summary-plus-detail has 94.98% adversarial faithfulness but 76.24% pass1, which requires claim support, fulfillment, and disclosure compliance.Source: §C.2 · pp. 24–25
- The flat selector leads all six selector metrics; its pass1 is 95.46% on standard and 98.82% on adversarial queries, with pass5 at 94.12% in both.Its pass5 corresponds to 112 of 119 standard and 80 of 85 adversarial queries succeeding in every run.Source: §C.2 · pp. 24–25, Table 3 · p. 26
- On adversarial queries, summary-plus-KG has higher pass1 than summary-plus-detail (81.65% versus 76.24%) but lower pass5 (62.35% versus 69.41%).It has fewer failed replies (78 versus 101), but those failures affect more queries (32 versus 26).Source: §C.2 · p. 25
- The full-statement baseline has high reply-level recall but low precision and frequent disclosure violations: 28.24% of standard and 47.29% of adversarial replies violate policy.Its pass5 is 41.18% on standard and 18.82% on adversarial queries, despite reply-level recall of 95.94% and 94.31%, respectively.Source: §C.2 · pp. 25–26, Table 4 · p. 26
D Implementation Details
This appendix documents the common coding-agent workflow, business-questioning interface, delivery protocols, expert review, and safeguards for grounded simulator responses. It also specifies task-specific judging conventions for bike-sharing and risk-profile variants, including source-code checks, equivalent formulations, and resource-usage reporting.
D.1 Agent Harness Selection
- Claude Code provides a common interaction surface for the full workflow: inspecting briefs and files, editing and running solvers, reviewing outputs, and revising deliveries.Agents contact ORSIM through Bash ask; the harness supplies execution mechanics, not OR modeling expertise or hidden evaluator access.Source: §D.1 · pp. 26–27
D.2 Models / D.3 Settings
- The evaluation covers eleven models; Harbor tasks allow at most 500 rounds and 12 hours, with each solver invocation capped at 3,600 seconds.Table 5 records model-level decoding settings; “Max context” is the effective gateway cap and “thinking strength” is the request-level reasoning control.Source: §D.2 · p. 27, §D.3 · p. 27
D.4 Semantic Validation
- Five LLM-judge reviews per bike-sharing task agreed with both human assessments, while experts manually reviewed all reported results.Each judgment was independently checked by two experts; the agreement supports this configuration within the evaluation setting.Source: §D.4 · pp. 27–28
D.5 Prompts
- Candidate prompts define file-based delivery and stakeholder questions, with rolling tasks imposing irreversible round-by-round submissions and restrictions on visible information.The grocery task has 18 batches; bike-sharing rolling tasks restrict plans to currently visible historical information, while Task 24 progressively reveals future data windows.Source: Business communication · p. 28, Prompt for Candidates: Rolling… · p. 29, Prompt for Candidates: Online Rolling… · p. 29
- Additional prompts permit public-data access and development-window backtesting, while requiring source-code retention for later review on Tasks 26–31.The backtest uses the formal evaluation framework but does not expose evaluation-window data.Source: Prompt for Candidates: Additional… · p. 30, Prompt for Candidates: Development… · p. 30, Prompt for Candidates: Source-Code… · p. 30
- The simulator is instructed to answer from the brief and authorized facts, distinguish supported claims from unknowns, and avoid guessing missing business rules.It should assess the proposition asked about rather than treating the questioner's formulas or assumptions as facts.Source: Prompt for Simulator: Replier · pp. 30–31, Prompt for Simulator: Selector · p. 30
- The simulator's response instructions require evidence-linked conclusions, preserve conditions and scope, and distinguish established causes from hypotheses rather than ranking unsupported explanations.Examples and calculations must use supplied or explicitly derivable premises and must not imply confirmed business rules or overall feasibility.Source: §3 · p. 32, §4 · p. 32, §5 · p. 32
6. Responsibility boundaries and clarification
- The simulator should answer business questions directly and concisely, asking clarification only when a question is genuinely ambiguous and stopping once it is answered.Responses should include only necessary conclusions, conditions, and specific unknowns; they should not shift responsibility, restate the question, or add unsolicited background or follow-up actions.Source: §6 · pp. 32–33, §7 · p. 33
Prompt for Judge: SSA Base
- The SSA judge reviews only modeling choices not established by deterministic verification, using five evidence-backed criteria on the submitted code’s actual solving path.The protocol excludes feasibility replay and comprehensive code review; it requires tracing active modeling and solver calls, marks untraceable criteria uncertain, and does not claim proof of execution order or equivalence.Source: §D.5.6 · p. 33, Prompt for Judge: SSA Base · pp. 33–34
Reference formulas: resolving verbal ambiguity, not prescribing a unique formulation
- The reference formulas require expanding the actual objective, checking coefficients and risk aggregation order, and accepting equivalent formulations rather than literal formula matches.The SSA reference also specifies four eligible historical scenarios with paired demand and returns, shared four-hour relocation, rolling execution, and objective coefficients including 0.35pi terminal shortfall without a terminal discount.Source: Reference formulas: resolving verbal… · pp. 35–36
Task 26: SSA
- For Task 26 SSA, shared relocation cost is counted once while scenario-specific losses are averaged; averaging demand and returns before calculating losses is generally not equivalent.The reference notes that multiplying the entire objective by N is acceptable, whereas underweighting shared relocation cost relative to operating losses by N is incorrect.Source: Task 26: SSA · p. 36
Prompt for Judge: Chance-Constrained Base
- The chance-constrained judge checks four paired historical scenarios, the protected-station event rule and slack penalty, 0.98 hourly discounting, terminal shortfall terms, and rolling optimization.The protected set contains 15 stations; each station-hour allows ⌊0.25N⌋ shortage events, with 5000 charged per excess event through nonnegative slack.Source: Prompt for Judge: Chance-Constrained… · p. 37
Task 28: chance
- Task 28 defines the protected set as the first 15 stations ranked by historical score and station ID, allows ⌊0.25N⌋ events, and charges 5000 per unit of excess slack.A shortage event is counted per station and scenario, not per bike; ordinary shortage charges remain, and equivalent event-count representations are allowed.Source: Task 28: chance · p. 38
Prompt for Judge: Two-Stage Risk Profile
- The two-stage variant shares first-hour decisions across four scenarios, permits scenario-specific recourse for the following three hours, and weights each scenario’s total planning costs by 1/4.Shared first-hour relocation cost is counted once after averaging; subsequent relocation costs and operating losses are included per scenario.Source: Prompt for Judge: Two-Stage Risk Profile · p. 39
- The two-stage risk-profile check focuses on whether the shared scenario plan and penalty for excess shortage events at protected stations are represented.Source: Prompt for Judge: Two-Stage Risk Profile · p. 39
Prompt for Judge: Chance-Constraint Risk Profile
- The chance-constraint profile uses four separate historical demand-and-return scenarios with a shared four-hour plan, and penalizes excess shortage events at 15 protected stations.It allows ⌊0.25N⌋ events per protected station and local hour, charging 5000 for each excess event; the objective combines relocation cost, mean scenario operating losses, and the slack penalty.Source: Prompt for Judge: Chance-Constraint… · p. 39
Prompt for Judge: CVaR Risk Profile
- The CVaR profile combines shared relocation cost, 0.4 times mean scenario operating loss, and 0.6 times CVaR0.8, which equals maximum scenario loss for four equally probable scenarios.The four historical scenarios retain separate demand and return inputs and share a four-hour relocation plan; CVaR includes partial probability mass at the tail boundary.Source: Prompt for Judge: CVaR Risk Profile · p. 39
Prompt for Judge: Distributionally Robust Risk Profile
- The distributionally robust profile minimizes relocation cost plus worst-case expected operating loss over probabilities within L1 distance 0.35 of four nominal probabilities of 1/4.Candidate probabilities must be nonnegative and sum to 1; the prescribed radius corresponds to total variation 0.175.Source: Prompt for Judge: Distributionally… · p. 40
Prompt for Judge: Robust Risk Profile
- The robust profile constructs 16 prescribed adverse cases from historical windows and minimizes shared relocation cost plus the maximum operating loss across those cases.Construction includes the specified historical statistics, shock-selection and rounding rules; duplicate cases are retained, with no probabilities assigned.Source: Prompt for Judge: Robust Risk Profile · p. 40
D.6 Agent Resource Usage
- Resource comparisons report token usage and API cost because rate limits and concurrency vary, averaging only runs with recorded token counts.Only model–task pairs with at least one such run contribute to the denominator.Source: §D.6 · p. 40
D.7 Harness Ablation
Harness effects depend on the model, while the strategy analysis classifies submitted solutions by technique, objective handling, and structural exploitation. The annotation framework distinguishes these dimensions while allowing them to describe the same workflow.
D.7 Harness Ablation
- Codex raises DAQ for GPT-5.6 Sol and DeepSeek-V4.1-Flash but lowers it for GLM-5.3-Flash, with feasibility following the same pattern.The comparison uses the same model endpoints, tasks, execution budgets, and number of runs.Source: §D.7 · pp. 40–41, Table 7 · p. 40
E Solution Strategies and Optimization-Structure Exploitation
- The strategy annotations classify final submitted programs by technique, objective handling, and problem-structure exploitation, excluding abandoned approaches and unused code.Default mechanisms internal to a solver are not counted as additional agent-designed methods.Source: E Solution Strategies and… · p. 41
E.1 Annotation Framework
- Seven technique labels distinguish solver invocation, specialized methods, construction, search, enumeration, and branching, with multiple labels allowed for distinct workflow modules.Solver-internal branch-and-bound, pruning, and heuristics remain within T1 rather than receiving additional T7 labels.Source: §E.1 · p. 41
- Objective handling separates direct preservation of the prescribed business objective (O1) from independently designed surrogate objectives (O2).Faithfully translating a business requirement into a mathematical expression remains O1, even when the problem statement lacks an explicit mathematical objective.Source: §E.1 · pp. 41–42
- Structural exploitation labels cover equivalent reformulation, autonomous decomposition, specialized algorithms, and valid search-space reduction; required multi-period or multi-scenario implementation alone does not qualify.Multiple distinct structural operations may receive multiple labels.Source: §E.1 · p. 42
- Technique, objective, and structure labels can co-occur: a solver-based two-stage objective strategy can receive T1, O1, and S4.Rolling execution alone determines neither technique nor objective handling, and adoption frequency is not a method-quality score.Source: §E.1 · p. 42
E.2 Strategy Patterns and Expert Comparison
Agents and experts differ not only in tool use but also in how they compose objectives and problem structure into solution strategies. Agents often use solvers and reformulations, while expert strategies more frequently organize search through decomposition and tightening.
From tool invocation to method composition
- 78.0% of agent solutions invoke general-purpose solvers, versus 93.5% of expert solutions; greedy or rule-based construction appears in 29.6% versus 19.4%.Agents may combine construction with solver selection or refinement, so tool use extends beyond invoking an optimization library.Source: §E.2 · p. 42, Figure 6 · p. 43
Instance-specific methods and paths to scalability
- 12.3% of agent solutions use explicit enumeration, versus 3.2% of expert solutions, often to generate finite candidate sets for later selection.Such methods are readily implementable when candidate spaces are manageable, but enumeration can grow exponentially; expert approaches more often retain iterative methods that accommodate larger problems.Source: §E.2 · pp. 42–43
Reformulating representations and organizing structured search
- Agents use equivalent compression or reformulation more often than experts (13.2% vs. 6.5%), while experts more often use autonomous decomposition (41.9% vs. 23.2%) and valid tightening (45.2% vs. 27.3%).Agent S1 annotations are concentrated: Tasks 12 and 30 account for 44.4% of positive S1 annotations.Source: §E.2 · pp. 43–44, Figure 7 · p. 44
Tool use and objective design are distinct capabilities
- Agents use autonomous surrogate objectives in 26.4% of tasks, versus 38.7% for experts; 83.3% of expert O2 solutions also invoke a general-purpose solver.Technique and objective handling are distinct: 49.5% of agent solutions using greedy or rule-based construction retain the direct business objective.Source: §E.2 · p. 44
Strategy profiles across models
- Solver adoption ranges from 67.7% to 87.1% across models, while surrogate-objective use ranges from 16.1% to 48.4% and tightening from 9.7% to 41.9%.Joint T–O–S patterns show that frequent solver use does not necessarily coincide with objective design or structured search; connecting these into a complete process remains a distinction from expert solutions.Source: §E.2 · pp. 44–45
F Extended Comparison with Related Benchmarks
OR-LICENSE connects original client materials and fact elicitation with executable optimization deliveries evaluated against client-accepted expert references. This extends benchmark comparisons across task sources, agent workflows, client interaction, and reference provenance.
Optimization modeling and constraint reasoning
- Unlike prepared-task benchmarks, OR-LICENSE starts from de-identified original business requirements and inputs, with partial requirements disclosed initially and additional client-held facts elicited during solution development.This evaluates requirement acquisition, interpretation, and formalization together with solver development.Source: F Extended Comparison with Related… · p. 45
Combinatorial optimization and algorithm engineering
- Compared with agent benchmarks focused on supplied task definitions, OR-LICENSE combines algorithm development with acquiring and interpreting client requirements.This covers both establishing the optimization task and developing a solution that satisfies operational constraints and achieves competitive objective quality.Source: F Extended Comparison with Related… · p. 45
Client interaction and reference solutions
- Agents receive initial briefs and files, while ORSIM makes authorized client facts available throughout formulation, solving, and refinement without supplying OR expertise.Hidden evaluation checks feasibility and objective quality against original expert solutions accepted by clients before benchmark construction.Source: F Extended Comparison with Related… · p. 47
Multi-stage OR agent benchmarks
- OR-LICENSE uses original client materials, client fact elicitation throughout optimization, and reference deliveries accepted before benchmark construction.The comparison distinguishes fact elicitation from scripted event updates or rubric feedback, and requires client-accepted references from original professional engagements.Source: F Extended Comparison with Related… · pp. 46–47, Table 8 · p. 46
G.1 Expert Recruitment and Task Construction
Experts with professional OR experience supplied and curated projects into benchmark instances, preserving original materials after de-identification. Independent review checked the consistency of briefs, references, and hidden evaluators.
G.1 Expert Recruitment and Task Construction
- Contributors had at least three years of professional OR experience and a master’s degree or higher in a relevant technical field.Experts provided source projects and curated them into benchmark instances.Source: §G.1 · p. 47
- The dataset contains 31 instances across 10 industrial scenarios, drawn from de-identified source materials without recasting requirements as complete mathematical formulations.The source engagements represented $3.68 million in project value, averaged 4.7 months, and involved 2.16 experts per project.Source: §G.1 · pp. 47–48
G.2 Task Quality Control
- Experts independently checked reference-code execution, hidden constraint and objective implementations, and whether de-identification preserved original information boundaries and ambiguity.They also verified that the initial brief excluded client-held facts and structural insights reserved for later acquisition or analysis.Source: §G.2 · p. 48, §G.1 · p. 48
G.3 Problem Sizes
- Table 9 summarizes approximate potential decision-variable and constraint scales across tasks, estimated from task conditions and implementation.The supplied passage gives no numerical size estimates.Source: §G.3 · p. 48
G.4 Verification, Evaluation, and Competency Coverage
The benchmark separates hard feasibility verification from independent objective evaluation, then maps task-specific checks and scoring to six non-exclusive competency labels. The task matrix distinguishes direct optimization from problems requiring additional modeling, policy, data-analysis, or uncertainty choices.
G.4 Verification, Evaluation, and Competency Coverage
- Each instance passes a hard verifier that checks the artifact and recomputes constraint-relevant quantities, followed by an independent evaluator that recalculates its objective.Scoring uses the verifier and evaluator versions applied in the reported experiments.Source: §G.4 · pp. 48, 50
- The six competency labels are non-exclusive summaries of task demands and are not used in scoring.They span direct/static optimization, objective or policy design, data analysis and calibration, sequential or multi-scenario modeling, and secondary or tie-aware criteria.Source: §G.4 · pp. 48, 50
- Table 10 summarizes verifier checks, evaluator objectives, and competency labels across the 31 benchmark tasks.The task checks vary by problem, including network structure and flow, material balance, sequencing constraints, warehouse-wave capacity, and strategy or backtest constraints.Source: Table 10 · p. 50, §G.4 · pp. 50–51
- Tasks 9–12, 14–19, and 25 primarily test static or directly scored optimization, including business-contract formulation and choosing a suitable solution method.Tasks 14–16 emphasize network optimization, while Task 19 also requires an approach suited to its combinatorial structure.Source: §G.4 · p. 52
- Tasks 1, 4, 6–8, 13, and 20–24 require partly unspecified modeling or policy choices; tasks 26–31 add scenario construction and risk preferences.Task 13 and tasks 20–22 also require historical-data analysis; tasks 26–31 retain primary feasibility while considering secondary or tie-aware criteria.Source: §G.4 · p. 52