PaperlayerView exact revision on arXiv

Source-linked AI summary

OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents

Chenyu Zhou, Xinyun Lu, Jiangyue Zhao, Jianghao Lin, Dongdong Ge, Yinyu Ye

arXiv:2605.28158v1cs.AI

TL;DR

Existing OR benchmarks largely test one-shot modeling from self-contained prompts, missing persistent multi-artifact workspaces and multi-stage industrial workflows. OR-Space introduces an executable workspace benchmark with Build, Revise, and Explain tasks, and its evaluation shows that workspace-grounded optimization remains challenging while legacy code can substantially change revision performance.

  • Problem

    Existing OR evaluations underrepresent industrial OR workflows by using self-contained prompts and single-shot tasks instead of persistent multi-artifact, lifecycle-oriented settings.

  • Method

    OR-Space evaluates agents in executable workspaces containing documents, data, code, solver states, and evaluators across Build, Revise, and Explain tasks.

  • Results

    Workspace-grounded modeling remains difficult: the strongest Build result is 72.0%, while revision changes performance from 72% to 81% for gemini-3.1-pro and from 59% to 79% for gpt-5.4.

  • Takeaways & Limitations

    OR-Space shifts evaluation from isolated text generation toward solver-grounded engineering workflows for studying reliability, failure modes, and practical readiness.

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly used to assist with operations research (OR) modeling, yet existing OR-oriented benchmarks often reduce evaluation to one-shot translation from a self-contained problem statement into a mathematical formulation or solver program. Such settings abstract away two characteristics of real industrial OR workflows: persistent multi-artifact workspaces and multi-stage task lifecycles. We introduce OR-Space, a full-lifecycle workspace benchmark for evaluating industrial optimization agents across model construction, model revision, and grounded explanation. Each instance is an executable workspace containing business documents, structured data, optional code artifacts, solver outputs, and task-specific evaluators distributed across interdependent files. OR-Space defines three task modes: Build, where agents construct solver-ready optimization models from heterogeneous artifacts; Revise, where agents modify existing models under changing requirements or solver feedback while preserving valid prior logic; and Explain, where agents answer grounded questions about solutions, constraints, and business implications using evidence spread across workspace artifacts. By combining persistent workspaces with lifecycle-oriented tasks, OR-Space evaluates whether agents can perform reliable optimization work beyond end-to-end text generation. We describe the benchmark design, evaluation protocol, and quality-control pipeline, and position OR-Space as a benchmark for studying the reliability, failure modes, and practical readiness of LLM agents in industrial OR workflows.

1. Introduction

OR-Space addresses the gap between isolated, self-contained OR evaluations and the persistent, multi-artifact, lifecycle-oriented work of industrial OR engineers. It introduces a benchmark spanning model construction, revision, explanation, and fine-grained failure analysis.

  • Industrial OR work requires interpreting documents, extracting data, implementing and debugging solver programs, revising models, and communicating decisions within persistent workspaces.These activities extend beyond mathematical modeling ability to reliable operation across heterogeneous artifacts.
  • Existing OR evaluations often compress business context into self-contained prompts and focus on static, single-shot formulation or code-generation tasks.This removes the need to connect information across documents, data, code, and solver outputs and underrepresents iterative model maintenance.
  • OR-Space represents each instance as an executable workspace containing documents, structured data, optional code, solver outputs, and evaluation interfaces distributed across artifacts.Agents must recover the optimization problem from these interconnected workspace components rather than receive a fully serialized statement.
  • The benchmark evaluates three lifecycle modes: Build constructs solver-ready models, Revise updates existing models under change, and Explain answers grounded questions about solutions and business implications.Together, the modes cover construction, maintenance, and communication in industrial OR workflows.
  • OR-Space supports systematic analysis of formulation, data-grounding, revision, execution, and explanation failures across executable model construction, targeted revision, and grounded explanation.The evaluation is designed to identify difficult lifecycle stages and dominant workspace-grounded failure modes.

2. Related Work

Prior work has improved solver-grounded evaluation of LLM optimization modeling and broadened agent evaluation to realistic tool- and workspace-rich environments. However, these strands do not jointly evaluate the domain-specific dependencies among OR models, data, solver behavior, and faithful explanations.

  • Recent OR benchmarks evaluate formulation and solver-program quality across linear, integer, mixed-integer, and nonlinear programming settings.These efforts include solver-grounded objectives rather than relying only on textual similarity.
  • Broader agent benchmarks show that tools, files, execution feedback, deployment constraints, and workplace-style environments affect agent behavior and evaluation outcomes.The benchmark landscape spans software, web, enterprise, mobile, API, and simulated-organization settings.

3. OR-Space Benchmark

OR-Space evaluates industrial optimization agents in persistent, executable workspaces across the lifecycle from model construction through revision and solver-grounded explanation. Its benchmark instances separate business documents, parameters, code, runtime state, and evaluation while testing artifact grounding, consistency, execution, and explanation quality.

  • 3.1. OR Workspace Formalization: OR-Space represents each optimization session as a persistent executable workspace containing documents, parameters, code, runtime state, and evaluation artifacts.The workspace formalization follows W = ⟨D, P, S, E, M⟩, with the evaluator attached by the benchmark harness rather than exposed as an agent-visible artifact.
  • 3.1. OR Workspace Formalization: Documents express business requirements and modeling intent, while separate parameter artifacts provide numerical inputs that may require cleaning, schema alignment, and cross-file integration.Parameter files can contain missing values, inconsistent schemas, mixed encodings, and cross-file dependencies.
  • 3.1. OR Workspace Formalization: Code artifacts and a Docker-based runtime support model construction, legacy-model revision, file I/O, solver execution, execution-state capture, and resource constraints.Build tasks typically start from empty scaffolds, whereas Revise tasks provide executable legacy implementations.
  • 3.2. Lifecycle Task Modes: Build constructs a complete model from requirements, data, and an empty scaffold; Revise updates legacy workflows while preserving unaffected logic across code and formulation variants.Revise variants include code-only, formulation-only, and combined code-plus-formulation changes.
  • 3.2. Lifecycle Task Modes: Explain generates solver-grounded reports about bottlenecks, sensitivities, and allocation decisions using logs, feasibility, runtime statistics, dual variables, and constraint slacks.Build and Revise are solver-scored, while Explain evaluates grounded natural-language reasoning tied to solver states.
  • 3.3. Benchmark Construction Pipeline: The construction pipeline expands 100 IndustryOR problems into 100 instances per task setting, yielding 300 multi-artifact Build, Revise, and Explain instances.Generated tasks are reviewed by operations-research researchers for consistency across descriptions, schemas, solver states, and explanation rubrics.

4. Experiments

OR-Space evaluates 20 models across Build, Revise, and Explain using isolated executable workspaces and solver-backed metrics. Results show that workspace grounding, revision context, solver choice, and explanation are distinct sources of difficulty.

  • Experimental setup: 20 models were evaluated in isolated workspaces across Build, Revise-code, and Explain, with filesystem access restricted to each agent’s own subtree.The default track used Gurobi 12.0.1 as the solver backend and ground-truth oracle.
  • Main results: 72.0% was the strongest Build result, while most frontier models scored between 53–68% when recovering semantics, schemas, and model structure from distributed artifacts.The task used docs/, data/, and an empty src/ rather than a self-contained formulation prompt.
  • Main results: Revise-code helped gemini-3.1-pro rise from 72% to 81% and gpt-5.4 from 59% to 79%, but reduced gemini-3-flash by 23 percentage points and qwen3-32b by 12.The contrast indicates that legacy heuristics can provide reusable structure or introduce implementation noise, depending on the model.
  • Main results: gpt-5.4 achieved the highest Explain score at 86.52, while gemini-2.5-pro and gemini-2.5-flash combined relatively strong Revise performance with much lower Explain scores.gemini-2.5-pro scored 69% on Revise and 32.2 on Explain; gemini-2.5-flash scored 66% and 13.79, respectively.
  • Main results: Build and Revise correlated strongly at rB,R = 0.82, whereas Explain correlated weakly with Build and Revise at rB,E = 0.16 and rR,E = 0.28.These results support evaluating grounded explanation separately from solver-correct model construction.
  • Robustness and context: Formal formulations and executable code provided complementary context, while solver choice changed absolute scores and task profiles despite broadly stable rankings.Gurobi had the highest objective-task average (57.2) and Explain average (64.9); its ranking with COPT had ρ = 0.73.
  • Failure analysis: 19.1% of Revise submissions were WrongValue and 18.2% RuntimeError, while module_not_found rose from 0.4% in Build to 2.8% in Revise.Many failures were executable models encoding the wrong mathematical problem, and legacy code introduced additional engineering errors.
  • Failure analysis: 39.3% of gemini-3.1-pro’s 28 Build failures were data-mapping errors, compared with 21.4% mathematical-modelling mistakes and 28.6% hallucinated constraints.The case study further shows a syntactically valid Optimal script returning 25 instead of the reference value 125 because it optimized the wrong business quantity.

5. Conclusion

OR-Space reframes optimization-agent evaluation as reliable work across persistent, multi-artifact workspaces and the full industrial OR lifecycle. Its benchmark package supports reproducible Build, Revise, and Explain evaluation with explicit artifacts, prompts, oracles, and rubrics.

  • OR-Space addresses existing benchmarks’ reliance on textbook-style prompts and single-shot optimization tasks.
  • OR-Space releases task workspaces, split manifests, oracle objectives, solver artifacts, prompt templates, evaluation rubrics, metadata, and responsible-use documentation.
  • The benchmark evaluates Build, Revise, and Explain within shared workspaces containing documents, data, code, logs, and solver records.
  • Each workspace instance separates business requirements, numeric parameters, executable artifacts, runtime traces, and evaluation records while hiding verified oracle formulations and implementations.
  • The benchmark’s forging pipeline uses staged prompts and quality gates to construct and filter Build and Revise workspace instances.

B.2. Release, Reproducibility, and Responsible Use

The release emphasizes reproducibility and responsible use while defining important scope and evaluation boundaries. OR-Space is synthetic, solver-dependent, and partly reliant on LLM judging and changing provider endpoints.

  • OR-Space is derived from 100 synthetic industrial-style optimization topologies rather than complete enterprise OR deployments.
  • The benchmark emphasizes linear and mixed-integer programming and may underrepresent nonlinear, stochastic, simulation, and human-in-the-loop workflows.
  • Solver-backed scores depend on solver availability, numerical tolerances, and API familiarity, while closed-source results may drift as provider endpoints change.
  • Explain evaluation combines exact-match checks with an LLM judge, leaving possible judge bias and prompt sensitivity despite rubric grounding and hallucination penalties.
  • The release uses compatible research licensing, excludes proprietary solver binaries and API credentials, and contains synthetic workspaces not intended for personal or sensitive information.

C.1. Full Solver-Backend Results

The solver-backend analysis reports exhaustive model-level results across Build, Revise, and Explain, using task-specific metrics to characterize backend sensitivity rather than create a separate leaderboard.

  • Build and the three Revise variants use Pass@1 under the objective oracle Mobj, while Explain uses the rubric mean score Mexp.
  • The appendix results support aggregate comparisons in Table 5 rather than functioning as a separate leaderboard.
  • Table 6 reports model-wise results under four solver backends across 100 instances per task.
  • All reported metric columns are higher-is-better, with bold and underlined values marking within-category best and second-best results.
  • The exhaustive table preserves model-level variation, including solver-interface changes that alter failure modes rather than only objective values.

C.2. Filesystem Interface Delta Details

Filesystem interfaces generally reduce performance relative to flat prompts, but the effect varies by model and task. Explain is comparatively less affected because flattening removes navigation while preserving cross-artifact reconciliation.

  • The filesystem-minus-flat-prompt delta view defines negative values as filesystem penalties under matched task and model conditions.
  • −19 pp is the largest reported filesystem penalty for gpt-4o on Build.
  • −15 pp is the largest reported filesystem penalty for qwen3-max on Revise-code.
  • gemini-3.1-pro is insensitive to the interface on Build but loses 13 pp on Revise-code.
  • Explain remains much closer across interfaces because flattening removes navigation but not reconciliation across documents, data, code, and solver records.

C.3. Solver-Backend Sensitivity Details

Solver-backend sensitivity is highly model-specific, so aggregate evaluation should not rely on a single solver track. OR-Space therefore reports both default Gurobi results and four-backend aggregates.

  • Solver effects vary substantially by model, with some models stable under one backend but failing under another.This spread motivates reading solver-level macro averages cautiously.
  • Backend robustness is not monotone in model scale: gpt-5-mini is competitive under Gurobi, PuLP, and HiGHS but drops sharply under COPT.
  • gemini-3.1-pro and claude-opus-4-6 have small solver ranges, whereas deepseek-v4-pro shows a much larger spread.Stability and peak score are distinct properties in the backend comparison.
  • The evaluation reports both the default Gurobi track and the four-backend aggregate rather than treating one backend as complete evidence.

C.4. Revise Context Model-Level Details

Revise performance depends strongly on which context artifacts are supplied, and the effect varies across models. Formulation context is broadly helpful for strong Gurobi-track models but does not benefit every model equally.

  • The same Revise context choice can help one model and hurt another, as shown by model-level Gurobi deltas.The table compares Code, All, and Model settings using Pass@1 and deltas relative to Revise-code.
  • Adding the formulation to code is broadly helpful for strong Gurobi-track models, but all-context gains range from +7 pp for gemini-3-flash to +20 pp for gemini-2.5-pro.
  • Formulation-only context can rescue models harmed by heuristic code, improving gemini-3-flash by +25 pp relative to code.
  • Formulation-only context can also weaken other models, including qwen3.5-27b.

C.5. Failure Details and Case Studies

The failure audit identifies lifecycle-specific errors spanning execution, workspace grounding, business-quantity interpretation, and objective alignment. These cases show that executable models can still be materially wrong when artifacts are misread.

  • Across all 20 × 100 Revise trials, 19.1% of submissions are WrongValue and 18.2% are RuntimeError.TypeError accounts for 3.1% and module_not_found for 2.8%.
  • module_not_found rises 7× from Build’s 0.4% to Revise’s 2.8%, often because inherited heuristics contain non-solver imports such as scipy or numpy.
  • Table 11 extends the gemini-3.1-pro Build audit with additional representative workspace-specific failure types.
  • In IndustryOR_2, the generated model optimizes combat jets instead of the reference training quantity of 125 trained pilots.The script introduced separate training and capacity variables and still solved to optimality, illustrating a wrong-business-quantity failure.
  • IndustryOR_9 and IndustryOR_27 expose workspace-grounding failures from brittle file-format and schema assumptions.One case triggers a CSV parsing error from commas in natural-language text; the other involves incorrect data access assumptions.
  • IndustryOR_18 returns an optimal objective of 255000 instead of the reference objective 5 by maximizing a plausible CSV profit metric rather than the document’s operational goal.

D. Supplementary Dataset Statistics and Visualizations

Supplementary materials characterize OR-Space instance complexity, reasoning demands, and Explain-task scoring. The evaluation combines dataset visualizations with rubric-based judging grounded in solver outputs and workspace artifacts.

  • The supplementary section provides descriptive dataset statistics and a granular breakdown of agent performance across task phases.
  • Figure 6 visualizes constraint and decision-variable distributions, illustrating optimization-logic scale and instance dimensionality.
  • Figure 7 compares multidimensional complexity metrics across domains and highlights logical increments between Build and Revise.
  • Figure 8 breaks down reasoning hops and multistep logic required for successful Explain-task completion.
  • Explain responses are scored with an LLM-as-judge protocol because answering requires multihop reasoning across business documents, parameters, and solver or diagnostic records.
  • The judge uses ground-truth solver output and a five-dimensional rubric, scoring responses from 0 to 100 with penalties for unsupported constraints, parameters, or solver claims.
  • Scores of 90–100 indicate complete, correctly reasoned, artifact-grounded explanations without hallucination, while lower bands reflect omissions, weak grounding, or incorrect justification.
Loading 2605.28158v1…