PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

Junbo Jacob Lian, Yujun Sun, Huiling Chen, Chaoyu Zhang, Hanzhang Qin, Chung-Piaw Teo

arXiv:2602.15983v3cs.SEcs.AIcs.LGmath.OC

TL;DR

LLM optimization code can execute successfully while encoding semantically wrong formulations, creating a large feasibility–correctness gap. ReLoop combines four-stage structured generation with solver-based behavioral verification and diagnostic recovery. Across benchmarks, the mechanisms complement each other by error structure, with structured generation strongest on compositional problems and behavioral verification strongest on localized defects.

  • Problem

    LLM-generated optimization code can be solver-feasible yet semantically incorrect, with a 90-point feasibility–correctness gap on compositional problems.

    Source: §1 · pp. 1–2

  • Method

    ReLoop combines four-stage structured generation, solver-based behavioral perturbation testing, diagnostic execution recovery, and regression-guarded repair.

    Source: §1 · p. 2

  • Results

    Structured generation drives the largest gain on RetailOpt-190 compositional problems (+8.5pp accuracy on Claude Opus 4.6), while behavioral verification contributes its largest gain on MAMO-ComplexLP (+4.4pp).

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Structured generation and behavioral verification provide complementary safeguards because they target different error structures.

    Source: §1 · p. 2, §6 · p. 10

    ReLoop’s current formulation class targets (MI)LPs with nonnegative physical-semantics parameters, while nonlinear, stochastic, and chance-constrained models are out of scope.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) can translate natural language into optimization code, but silent failures pose a critical risk: code that executes and returns solver-feasible solutions may encode semantically incorrect formulations---a feasibility--correctness gap reaching 90 percentage points on compositional problems. We introduce ReLoop, which addresses this gap through two complementary mechanisms. Structured generation decomposes code production into a four-stage reasoning chain (understand, formalize, synthesize, verify), preventing formulation errors at their source. Behavioral verification detects errors that survive generation by testing whether the formulation responds correctly to solver-based parameter perturbation---an external semantic signal that bypasses LLM self-review and requires no ground truth. The two mechanisms are complementary by error structure: structured generation drives the largest gains on compositional problems (+8.5pp accuracy on RetailOpt-190 with Claude Opus 4.6), while behavioral verification dominates on localized defects +4.4pp on MAMO-ComplexLP, its largest contribution across benchmarks). Combined with diagnostic execution recovery, ReLoop reaches 100% executable code on Claude Opus 4.6 and consistently improves accuracy on chat-tuned foundation models across three benchmarks; we further identify a known limitation of narrowly-tuned SFT models, whose learned output formats are brittle to chain-of-thought prompts---an interaction we document and analyze. We release RetailOpt-190, 190 compositional retail optimization scenarios targeting the multi-constraint interactions where LLMs most frequently fail.

1 Introduction

LLM-generated optimization code can be solver-feasible yet semantically wrong, because solvers and existing checks miss omitted constraints and terms. ReLoop addresses this gap through structured generation, behavioral verification, execution recovery, and diagnosis-guided repair.

1 Introduction

  • Up to 91.1% solver-feasibility versus 0.5% formulation correctness creates a 90-point gap on compositional problems.Existing checks catch syntax or execution issues but cannot reliably detect feasible-yet-wrong formulations.Source: §1 · pp. 1–2
  • ReLoop integrates structured generation, two-layer behavioral verification, execution recovery, and regression-guarded diagnosis-guided repair.The pipeline combines generation-time prevention with post-generation semantic testing.Source: §1 · p. 2
  • ReLoop decomposes optimization-code generation into understand, formalize, synthesize, and verify stages that mirror expert modeling practice.The process includes explicit variable-type reasoning and a grounded completeness check against the original problem.Source: §1 · p. 2
  • Behavioral verification tests constraint presence and objective completeness through solver-based parameter perturbations, without requiring ground truth.Unexpectedly zero response to a relevant capacity or cost perturbation signals a missing model component.Source: §1 · p. 2
  • RetailOpt-190 supplies 190 compositional retail optimization scenarios designed around interacting constraints where LLMs frequently fail.The benchmark targets combinations such as capacity, perishability, substitution, and lead times.Source: §1 · p. 2
  • Structured generation and behavioral verification are complementary: each dominates for different error structures, with gains across foundation models and three benchmarks.Structured reasoning targets formulation errors at generation, while perturbation testing detects surviving localized defects.Source: §1 · p. 2

2 Related Work

Prior optimization-modeling methods emphasize generation or execution feedback, but they generally cannot distinguish feasible-yet-wrong formulations without reliable semantic supervision. ReLoop adds solver-response perturbation as an external behavioral signal and evaluates compositional constraint interactions through RetailOpt-190.

LLM-Based Optimization Modeling

  • ReLoop adds post-generation behavioral verification to optimization modeling, targeting semantically incorrect formulations that remain solver-feasible.The approach complements prompting, fine-tuning, reinforcement learning, and agentic generation methods.Source: §2 · pp. 2–3

Verification and Self-Correction

  • Existing solver feedback, self-critique, environment-guided refinement, and execution reranking each lack a reliable way to identify feasible-yet-wrong models.Their limitations include correlated reasoning gaps, insufficient solver signals, scarce ground-truth programs, or unreliable generated tests.Source: §2 · pp. 2–3, Table 1 · p. 3
  • Semantic-anchor alignment and ReLoop are complementary because the former checks agreement in LLM semantic space while ReLoop supplies solver-behavioral feedback.LLM-reconstructed anchors may share the misreading that caused an error, whereas solver perturbation provides an independent signal.Source: §2 · p. 3

Sensitivity Analysis and Program Verification

  • Solver-response perturbation repurposes sensitivity analysis as optimization-specific metamorphic testing, using expected response patterns to expose missing components.The solver acts as an oracle, avoiding the need for formal specifications or ground-truth labels.Source: §2 · p. 3

Benchmarks

  • RetailOpt-190 addresses a benchmark gap by composing eight scenario families that require multiple constraints to bind jointly.Existing benchmarks primarily test isolated constraint types rather than interacting constraints.Source: §2 · p. 3

3 Method

ReLoop combines structured generation, two-layer behavioral verification, and diagnosis-guided repair to address semantically wrong optimization code that remains executable and feasible. Its pipeline separates formulation, execution, perturbation-based semantic checks, and conservative repair.

3 Method

  • ReLoop combines structured generation, behavioral verification, and diagnosis-guided repair to address silent formulation failures.Structured generation targets errors before verification; behavioral verification tests errors that survive generation, while execution recovery and repair complete the pipeline.Source: §3 · p. 3

3.1 Problem Statement

  • Silent failures are executable, solver-feasible programs whose encoded feasible region or objective does not match the intended formulation.Standard execution and feasibility checks therefore cannot establish semantic correctness.Source: §3.1 · pp. 3–4

3.2 Structured Generation

  • Structured generation uses four stages—understand, formalize, synthesize, and verify—to make assumptions, variable types, implementation, and completeness explicit.The final stage checks generated code against the original problem rather than relying on unconstrained self-critique.Source: §3.2 · pp. 4–5

3.3.1 L1: Execution Verification (Blocking)

  • L1 blocks invalid outputs through syntax, runtime, and solver checks, using IIS constraints or unbounded-ray variables to guide regeneration.Only L1 blocks output; infeasibility and unboundedness receive diagnosis-specific feedback.Source: §3.3.1 · p. 5, §3.3 · p. 5

3.3.2 L2: Behavioral Testing (CPT + OPT)

  • L2 tests constraint and objective presence by perturbing governing parameters and measuring the resulting objective response.Negligible change produces a WARNING, intermediate change produces INFO without repair, and large change or induced infeasibility produces PASS.Source: §3.3.2 · p. 6

3.4 Diagnosis-Guided Repair

  • Diagnosis-guided repair converts FATAL and WARNING diagnostics into targeted regeneration or repair while guarding against unsafe edits, solver regressions, and unnecessary fixes.A safety check blocks data reassignment and dangerous imports, and repairs are rolled back when they crash, regress solver status, or shift the objective by more than 4%.Source: §3.4 · p. 6

4 RetailOpt-190 Benchmark

RetailOpt-190 is a 190-instance benchmark designed to test compositional retail optimization under interacting operational constraints. It combines progressively difficult scenario families with tolerance-based correctness evaluation and serves as the main results benchmark.

4 RetailOpt-190 Benchmark

  • RetailOpt-190 contains 190 multi-period retail inventory instances requiring ordering, timing, and allocation decisions under interacting constraints.Instances typically include 3–5 products, 4–8 periods, 20+ parameters, and 10–30 jointly binding constraints.Source: §4 · p. 7
  • The benchmark spans eight scenario families, from inventory dynamics and substitution to tight feasibility boundaries, discrete decisions, and multi-location coordination.Thirty-eight archetypes receive five ±15% numerical variants, yielding 38 × 5 = 190 instances.Source: §4 · p. 7
  • Correctness requires matching ground-truth feasibility and objective values, with strict ϵ = 10^-4 and practical ϵ = 10^-2 accuracy tiers.MIP family F6 uses ϵ = 10^-2 for both tiers, while cross-benchmark experiments use ϵ = 10^-6.Source: §4 · p. 7

5 Experiments and Results

ReLoop is evaluated across three benchmarks and five model types using execution, strict/practical accuracy, cross-benchmark accuracy, and component ablations. Results show complementary benefits: structured generation helps compositional reasoning, L1 recovers execution failures, and L2 targets localized formulation defects.

5.1 Setup

  • ReLoop adds approximately 3× base LLM-token cost, while solver calls complete in seconds; evaluation uses pass@1 across RetailOpt-190, MAMO-ComplexLP, and IndustryOR.RetailOpt-190 reports execution and strict/practical accuracy; cross-benchmark evaluation uses Acc%(ϵ=10−6).Source: §5.1 · p. 8

5.2 Main Results on RetailOpt-190

  • 100% Exec and 31.1% Acc on Claude show that solver-feasibility remains a poor proxy for formulation correctness, with silent failures persisting after ReLoop.DeepSeek reaches 91.1% feasibility but only 0.5% correctness; Claude reaches 100% execution yet two-thirds of cases remain silent failures.Source: §5.2 · p. 8
  • +8.5pp accuracy makes structured generation Claude’s primary accuracy driver, while ReLoop preserves or improves every foundation-model metric combination relative to Base.For DeepSeek, CoT reduces execution from 91.1% to 53.2% because intermediate notation produces invalid Gurobi syntax; L1 recovers this regression through diagnostic feedback.Source: §5.2 · p. 8, §5.4 · p. 10

5.3 Cross-Benchmark Generalization

  • ReLoop improves foundation models and SIRL across MAMO-ComplexLP and IndustryOR without domain-specific tuning, but OptMATH collapses from 56.2% to 30.0% on MAMO under CoT.The best 32B-model accuracies remain 46.3% on MAMO and 46.0% on IndustryOR.Source: §5.3 · p. 9, Table 7 · p. 9

5.4 Ablation Studies

  • Potential training–evaluation overlap may inflate OptMATH’s unusually high MAMO Base score, while its IndustryOR Base–CoT difference is smaller.The reported IndustryOR scores change from 34.0% to 31.0%.Source: §5.4 · p. 9
  • +44.2pp Exec for DeepSeek shows that L1 is the dominant execution-recovery component, using IIS diagnostics to support targeted reformulation.L1 also improves Claude execution by 5.8 percentage points.Source: §5.4 · p. 10
  • +4.4pp on MAMO makes L2 Claude’s largest single accuracy contributor, because perturbation testing detects localized missing constraints and objective terms.L2 adds +2.0pp for DeepSeek; on RetailOpt-190, strict accuracy is unchanged because residual errors are predominantly structural.Source: §5.4 · p. 10

6 Conclusion

ReLoop combines structured generation and behavioral verification to address semantically incorrect but solver-feasible optimization code. The paper also defines a modular MILP benchmark and identifies formulation choices, scenario interactions, and scope limitations relevant to detecting such failures.

6 Conclusion

  • +8.5pp accuracy on RetailOpt-190’s compositional problems and +4.4pp on MAMO-ComplexLP’s localized defects show that ReLoop’s two mechanisms are complementary by error structure.Structured generation contributes most on compositional problems, while behavioral verification contributes most on localized defects.Source: §6 · p. 10
  • ReLoop targets silent formulation failures through structured generation at the modeling stage and solver-based behavioral verification without ground truth.The reference workflow separates understanding, formalization, synthesis, and verification, while perturbation tests check constraint presence and objective completeness.Source: §6 · p. 10
  • ReLoop is currently limited to selected nonnegative-parameter MILP settings, while nonlinear, stochastic, and chance-constrained models remain out of scope.Additional boundaries include brittle chain-of-thought compatibility for SFT models, possible shared extraction failures, and undetected coefficient-magnitude or present-but-wrong constraint errors.Source: §6 · p. 10

A.2 Decision Variables

  • The reference MILP uses modular variables and constraints so mechanisms activate only when corresponding instance parameters are non-default.Its objective includes purchasing, waste, and other cost terms, while capacity, resource, procurement, and network mechanisms are represented conditionally.Source: §A.2 · p. 13, §A.3 · p. 14, §A.5 · p. 14

A.4 Core Constraints

  • The formulation’s closed conservation system requires every ordered unit to exit through sales, aging, expiration, or lost sales, making omissions semantically detectable.This accounting spans six core inventory-flow constraint families and additional capacity and resource limits.Source: §A.4 · p. 14, Table 10 · p. 15

A.7 Formulation Design Rationale

  • The model prevents common formulation errors through explicit initialization, decrementing shelf-life indices, guarded temporal boundaries, and directional substitution edges.Fresh inventory enters at k = SLp and expires at k = 1; reversing the convention can prevent expiration, while reversed substitution edges were reported as a common modeling mistake.Source: §A.7 · p. 16

B Scenario Family Design

  • RetailOpt-190 emphasizes interacting mechanisms, diagnosability, and feasibility stress across 38 archetypes and 8 families.Families F5–F8 require compositional reasoning, and some instances make structural omissions produce infeasibility while the reference formulation remains feasible.Source: B Scenario Family Design · p. 17, §B.1 · p. 17, Table 11 · p. 17

B.2 Archetype Specifications

This section specifies the archetypes, reproducible variants, shared scenarios, data schema, prompt formats, and documented access considerations for RetailOpt-190. It also defines the worked-example format, deterministic ground-truth solver settings, family-dependent tolerances, and evaluation metrics centered on silent failures.

B.2 Archetype Specifications

  • RetailOpt-190 defines 38 archetypes through active modules, parameter overrides, and expected binding patterns, with all structural modifications listed in Table 12.Unmentioned fields retain base-scenario values, and numeric modifications are multiplicative generator factors.Source: §B.2 · p. 17

B.3 Instance Generation via Controlled Perturbation

  • Each archetype yields five variants, producing 190 instances; v0 is unperturbed, while v1–v4 apply reproducible stochastic perturbations with intensity α = 0.15.Perturbations independently vary demand curves and storage capacities while preserving the structural skeleton, including shelf life, network topology, and cost ratios.Source: §B.3 · pp. 17–18

B.4 Base Scenario Specification

  • All archetypes derive from a moderately complex base scenario with three product tiers, five distribution centers, seasonal demand, perishability, and costs making lost sales dominate waste.The base configuration is generated by get_base_scenario() and uses modular parameter overrides.Source: §B.4 · p. 18

C.1 Prompt Format Comparison

  • RetailOpt-190 provides schema-based and data-embedded prompts with identical semantics but different data-access demands, reporting results in the data-embedded format for benchmark comparability.Moderate scaffolding supplies business structure and selected equations without providing a complete formulation, objective, or full constraint set.Source: §C.1 · pp. 18–19

C.2 JSON Schema

  • All 190 instances share a six-group JSON schema covering dimensions, perishability, demand, capacities, costs, and network topology, with nested fields requiring safe defaulted access.A universal schema lets one solver implementation process every archetype without format-specific logic.Source: §C.2 · p. 19

C.3 Critical Data Access Patterns

  • Two documented access hazards are zero-based demand arrays versus one-based model periods and nested network fields, causing shifted demand profiles or KeyErrors when mishandled.Substitution edges encode upward substitution: premium inventory can serve basic-product demand.Source: §C.3 · p. 19

C.4 Worked Example: retail_f1_52_weeks_v0

  • The worked retail_f1_52_weeks_v0 instance tiles the base scenario to 52 periods, demonstrating schema-based and data-embedded prompts alongside perishability, substitution, and inventory-flow conventions.The data-embedded version is the default reporting format and supports self-contained evaluation without external file access.Source: §C.4 · pp. 19, 21, 22
  • The worked example contrasts schema-based and data-embedded prompt formats, with the latter differing in embedded data and executable-code instructions.The data-embedded prompt parses JSON, solves the optimization problem, and prints status and objective value, while requiring only executable Python code.Source: §C.4 · pp. 22–23

D.1 Ground Truth Solver Settings

  • All 190 ground-truth solutions use the Universal Retail Solver in GurobiPy with Threads=1 and Seed=0 for deterministic, single-threaded reproducibility.Source: §D.1 · p. 23

D.2 Accuracy Tolerances

  • Accuracy requires matching feasibility status and, for feasible instances, a relative objective error below a family-dependent tolerance.Family F6 receives a looser structure-dependent interpretation because MOQ binaries and pack-size integers can leave residual solver gaps within the 60-second limit.Source: §D.2 · p. 23

D.3 Evaluation Metrics

  • Silent Failure Rate is the primary metric, measuring the absolute gap between solver-feasible execution and correctness caused by semantic modeling errors.Execution Rate counts runtime-error-free runs returning feasible solutions, while Accuracy additionally requires status and objective agreement; SF% = Exec% − Acc%.Source: §D.3 · p. 23

E ReLoop Implementation Details

ReLoop combines structured code generation with two-layer behavioral verification, execution recovery, and diagnosis-guided repair to improve reliability of LLM-generated optimization models.

E ReLoop Implementation Details

Chain-of-Thought Generation Prompt

  • Structured generation uses extraction-first or self-contained prompts, with explicit variable types, complete constraints, complete objective terms, and final cross-checking.If parameter extraction fails, the pipeline falls back to embedding data directly in the generated code.Source: §E.2 · p. 29, Chain-of-Thought Generation Prompt · p. 30

E.3 L1: Execution Verification

  • L1 blocks downstream testing until syntax, execution, solver status, and optimality checks pass, regenerating code after fatal failures.Execution occurs in a sandbox with a 60-second timeout, and regeneration can run for up to three attempts.Source: §E.3 · p. 31

E.4 L2: Constraint Presence Testing (CPT)

  • L2 constraint-presence testing perturbs capacity, demand, and other parameters, classifying objective responses as warnings, information, or passes.A change below 5% triggers a warning, changes from 5% to 30% are informational, and larger changes or infeasibility pass.Source: §E.4 · p. 31, L2 CPT Constraint Extraction Prompt · p. 32

E.5 L2: Objective Presence Testing (OPT)

  • L2 objective-presence testing applies the same perturbation logic to expected cost and revenue terms, treating infeasibility as an inapplicable objective test.Cost terms are reduced near zero and revenue terms are increased substantially before measuring objective changes.Source: §E.5 · p. 32, L2 OPT Extraction Prompt · p. 33

E.6 Repair Pipeline

  • The repair pipeline prioritizes actionable warnings, limits repairs to three iterations, and rolls back changes that cause regressions or unsafe code.Safety checks block data reassignment, data mutation, and dangerous imports; failed safety retries preserve the original code.Source: §E.6 · p. 33, Repair Prompt (build_repair_prompt) · pp. 34–35

F.1 Correctness Criteria

  • RetailOpt-190 correctness requires matching feasibility and objective values within tolerance, with stricter and looser tiers for different evaluation settings.The benchmark uses ϵ = 10^-4 and ϵ = 10^-2 tiers, while MIP family F6 uses ϵ = 10^-2 because time limits may prevent proven optimality.Source: Table 19 · p. 28, §F.1 · p. 35

F.2 Metrics

The evaluation reports accuracy and silent-failure metrics under fixed, reproducible configurations, comparing native or base generation with CoT and ReLoop.

F.2 Metrics

  • RetailOpt-190 reports accuracy at ϵ = 10^-4 and ϵ = 10^-2, while cross-benchmark experiments use ϵ = 10^-6 and pass@1 decoding.All metrics are computed over the full dataset.Source: §F.2 · p. 35
  • Silent failure rate is defined as SF% = Exec% − Acc%, quantifying executable solver-feasible outputs that fail the correctness criterion.A silent failure executes and returns a feasible solution, but its objective misses ground truth beyond tolerance.Source: §F.2 · p. 35

G.1 Model Configuration

  • Models are evaluated in Base/Native, CoT, and ReLoop configurations, with ReLoop combining CoT, two-layer verification, and diagnosis-guided repair.The ReLoop configuration uses a repair budget of N = 3.Source: §G.1 · pp. 35–36, Table 22 · p. 36
  • The same temperature-zero LLM generates code and performs L2 verification, leaving cross-model verification for future work.Ground-truth computation uses Gurobi 11.0 with a 60-second time limit, 1% MIP gap, one thread, and fixed seed 0.Source: §G.1 · p. 36, §G.2 · p. 36

G.3 Benchmark Details

Benchmark evaluation uses fixed solver and decoding settings, with data-embedded prompts for comparability and cited baselines where published results exist.

G.3 Benchmark Details

  • The benchmark statistics are summarized in Table 23.Source: Table 23 · p. 36
  • All benchmarks use data-embedded prompts for comparison with prior work, while RetailOpt-190 also supports schema-based inputs for perturbation testing.The schema-based format is used internally by ReLoop’s verification pipeline.Source: §G.3 · p. 36
  • Published MAMO-ComplexLP and IndustryOR baselines are cited for DeepSeek-V3.2, Qwen3-32B, OptMATH-32B, and SIRL-32B, while CoT and ReLoop results are run directly.Additional GPT-4, DeepSeek-R1, and OpenAI-o3 results are cited for context but not directly evaluated with ReLoop.Source: §G.3 · p. 36

G.4 Reproducibility

  • Greedy decoding, a fixed solver seed, and single-run pass@1 evaluation support deterministic results, with identical outputs observed on a repeated 20-instance subset.All LLM calls use temperature = 0.Source: §G.4 · p. 36

H Per-Family Analysis on RetailOpt-190

Per-family RetailOpt-190 results show substantial gains on some interaction patterns but persistent failures on resource-coupled, tight-feasibility, and broader compositional cases.

H Per-Family Analysis on RetailOpt-190

  • 55.0%→95.0% is Claude’s largest absolute per-family gain, occurring on F1 Core Operations under ReLoop.This pattern indicates that ReLoop most effectively addresses basic perishable inventory dynamics in this breakdown.Source: H Per-Family Analysis on RetailOpt-190 · p. 37, Table 24 · p. 37
  • 0% across all models on F3 Resource and F5 Feasibility Stress shows persistent difficulty with multi-resource coupling and deliberately tight feasibility boundaries.The passage attributes these outcomes to limitations that remain despite verification.Source: H Per-Family Analysis on RetailOpt-190 · p. 37
  • DeepSeek improves from 0% to 20.0% on F6 Discrete Logistics, whereas Claude gains nothing on that family.The reported asymmetry links DeepSeek’s gain to L1 crash recovery from integer-variable errors.Source: H Per-Family Analysis on RetailOpt-190 · p. 37
  • Qwen3, OptMATH, and SIRL remain at 0% across all families, indicating that this complexity exceeds their current compositional retail-optimization capability.Source: H Per-Family Analysis on RetailOpt-190 · p. 37

I Injected-Defect Characterization of Behavioral Verification

The injected-defect study shows that L2 behavioral verification reliably detects omission defects that L1 cannot observe, with performance stable across shrink multipliers and independent of taxonomy routing.

Injected-defect construction

  • All 240 omission defects are invisible to L1 because deleting constraints relaxes feasibility and deleting objective terms never causes infeasibility.This structural partition makes omission defects the target domain for L2, while commission errors remain in L1’s domain.Source: I Injected-Defect Characterization of… · p. 37

Detection

  • 99.2% detection was achieved on 240 injected omissions, including all 83 removed constraints and 155 of 157 removed objective terms.The detected set included 74 components inactive under nominal data, which required extreme perturbation to become observable.Source: I Injected-Defect Characterization of… · p. 37

Multiplier robustness and taxonomy independence

  • Detection remained 238/240 at shrink multipliers 10^-4, 10^-3, and 10^-2, with 63–64 intact-code false positives out of 240.Robustness is scoped to shrink-direction perturbations because the benchmark has no inflate-axis test population.Source: I Injected-Defect Characterization of… · p. 37, Table 25 · p. 38
  • A taxonomy-free bidirectional perturbation control also detected 238/240 while reducing intact-code false positives to 54/240.The result indicates detection does not depend on capacity, demand, or cost routing; routing only reduces solver calls.Source: I Injected-Defect Characterization of… · p. 38

J Failure-Attribution and Label-Validity Audit

The audit found that sampled failures were overwhelmingly genuine modeling errors rather than alternative valid readings, while also validating the discrete-variable convention used in MAMO-ComplexLP.

Audit protocol

  • The audit used blind pre-annotation, expert verification, independent re-verification, Gurobi re-solves, and solver-checkable minimal-flip rationales.Raw annotator agreement was 76% with κ = 0.57 before expert adjudication.Source: J Failure-Attribution and… · p. 38

RetailOpt-190

  • 39 of 40 sampled RetailOpt-190 failures were true modeling errors, with one numerical/tolerance case and no defensible alternative readings or metric-penalized equivalent formulations.The reference generator reproduced all 190 published ground truths to 10^-6.Source: J Failure-Attribution and… · p. 38
  • Across 43 audited modeling errors from both benchmarks, 42 were structural, comprising 22 wrong inter-mechanism couplings and 20 missing structural linkages.Only one audited error was a data-extraction slip.Source: J Failure-Attribution and… · p. 38

MAMO-ComplexLP

  • MAMO’s integer-valued diet ground truths were validated as intended discrete-item optima, and all 11 L2-corrected cases restored integrality without changing constraint sets.Independent LP+IP re-solves confirmed the convention; 197 of 203 ground truths were integer-valued and all 6 fractional cases used continuous units.Source: J Failure-Attribution and… · p. 38

K Benchmark Composition Audit

The composition audit finds that MAMO-ComplexLP and IndustryOR are broad optimization benchmarks rather than revenue-management datasets, while the checklist documents the paper’s reproducibility, limitations, and release plans.

K Benchmark Composition Audit

  • Neither MAMO-ComplexLP nor IndustryOR contains revenue-management instances; both are dominated by production, blending, transportation, scheduling, and assignment problems.Revenue- or profit-maximization objectives occur in 11/203 MAMO instances and 33/100 IndustryOR instances, but these do not define the revenue-management class.Source: K Benchmark Composition Audit · p. 38

NeurIPS Paper Checklist

  • The paper’s four stated contributions are structured reasoning, behavioral verification, diagnosis-guided repair, and the RetailOpt-190 benchmark, each supported by experiments in Sections 5.2–5.4.RetailOpt-190 contains 38 archetypes across 8 mechanism families and is planned for release with documentation and reference solutions.Source: NeurIPS Paper Checklist · pp. 39, 41
  • The principal limitations are SFT incompatibility with chain-of-thought formats, linear L2 overhead, possible shared-LLM failure correlation, and unrepresented failure modes.The reported SFT interaction included 84 crashes and 65 regressions on OptMATH/MAMO.Source: NeurIPS Paper Checklist · p. 39
  • The paper does not present formal theorems requiring proofs; its perturbation assumption is stated with applicability conditions and validated empirically.The assumption is used as a design principle rather than as a formally proved result.Source: NeurIPS Paper Checklist · p. 39
  • Reproduction is supported by full algorithmic specifications, prompt templates, benchmark-generation procedures, evaluation tolerances, solver settings, and documented model configurations.The ReLoop codebase and RetailOpt-190 benchmark are planned for permissive release upon acceptance.Source: NeurIPS Paper Checklist · pp. 39–40
  • Evaluations use deterministic greedy decoding, so reported results are single-run pass@1 measurements without stochastic error bars.Repeated executions on a random subset of 20 instances produced identical outputs, although API behavior may drift over time.Source: NeurIPS Paper Checklist · p. 40
  • The study uses commercial APIs for foundation models, local BF16 inference for 32B models, and single-threaded Gurobi solver calls that complete in seconds for benchmark instances.ReLoop adds approximately 3× the base token cost for foundation-model evaluations.Source: NeurIPS Paper Checklist · p. 40
  • The work reports no human-subject or private-data involvement, and its released artifacts are synthetic optimization scenarios and a verification framework.The checklist identifies no foreseeable harmful applications and no high-risk artifacts requiring special release safeguards.Source: NeurIPS Paper Checklist · pp. 40–41

Related papers

Loading 2602.15983v3…