Fix What's Wrong, Keep What's Right: Hallucination Repair for LLM-Based Optimization Modeling

1 Introduction

• Optimization modeling hallucinations are errors in generated artifacts that make them inconsistent with the original problem.Examples include unsupported additions, omitted requirements, formulation errors, model–code inconsistencies, data-binding faults, and result-extraction errors.Source: §1 · p. 1
• Optimization-modeling repair must detect omitted requirements and model–code errors that statement-level support checks or successful code execution cannot rule out.The task therefore requires evidence from the original problem and coordinated updates to the model and code.Source: §1 · pp. 1–2
• Repair must correct errors without damaging valid content, with Rescue, Regression, and Error Coverage tracking correction, harm, and retained repair opportunities.The running example changes the incorrect inequality in both model and code while leaving variable domains, objective coefficients, and other constraints unchanged.Source: §1 · p. 2, Figure 1 · p. 2
• OPT-Kraken extracts requirements and instance data, localizes errors with directional fidelity checks, selects repair scope, and uses evidence-bound contracts and commit gates.It is a training-free post-processor for existing problem–model–code artifacts.Source: §1 · pp. 2–3
• Accuracy improves by 6.3–20.1 percentage points across the four generation pipelines.The framework is also evaluated across six benchmarks, four generation pipelines, four fine-tuned generators, and three repair backbones.Source: §1 · p. 3

2 Related Work

• General hallucination methods check claim support, consistency, or tool feedback, but those checks alone do not establish requirement coverage or fidelity across models and solver code.The cited examples include FActScore, SelfCheckGPT, RARR, and CRITIC.Source: §2 · p. 3
• Optimization-modeling generation and training methods improve model and code generation, but do not by themselves establish that artifacts remain faithful to the original problem.The passage describes expert collaboration, feedback, counterfactual reasoning, tree search, and training approaches.Source: §2 · p. 3
• Verification and correction work largely addresses generation-time errors or refinement, while scope-aware post-hoc repair that preserves correct content remains underexplored.OPT-Kraken selects granularity by affected model and code locations, bounds local edits, and audits candidates before acceptance.Source: §2 · p. 3

3.1 Problem Formulation and Available Information

• Given a problem and generated model and code, OPT-Kraken checks fidelity to the problem and correct use of instance data, while treating equivalent formulations as valid.Its stated goal is to return a model faithful to the problem and code faithful to the model.Source: §3.1 · p. 4

3.2 Dual Evidence and Directional Fidelity Checks

• Requirements capture problem semantics, while separate instance data bind values and structure; directional checks examine complementary fidelity obligations across the problem, model, and code.Requirements cite source text and link related conditions without adding requirements; separating requirements from data can expose binding errors such as assigning correct profits to the wrong products.Source: §3.2 · pp. 4–5
• Findings are merged by root cause, and detection distinguishes CLEAN, ERROR, and UNRESOLVED; a non-CLEAN finding guides review but alone does not authorize edits.Different targets remain separate, and conflicting effects withhold local writes.Source: §3.2 · p. 5

3.3 Evidence-Guided Repair Granularity Selection

• Repair scope is the smallest supported route whose closed write set covers the evidenced dependencies: S-only, M-local, or Structural.Structural repair requires evidence that local repair is insufficient; uncertainty alone yields Unresolved, and local failure does not authorize reconstruction.Source: §3.3 · pp. 5–6

3.4 Evidence-Constrained Local Repair

• For M-local repair, a RepairCard's category rules and a RepairTask's case evidence are compiled into contracts that restrict edits and specify acceptance criteria.The contract identifies writable locations and read-only dependencies, while the dependency closure determines which locations must change together.Source: §3.4 · p. 6
• Local patches are checked against their permitted scope, protected content, and intended effects before code is synchronized; the example changes only the inequality direction.The example yields 5B ≥2A while preserving coefficients, profits, capacity, and domains.Source: §3.4 · p. 6

3.5 Artifact-Level Auditing and Safe Commit

• A candidate replaces retained artifacts only after execution and artifact-level auditing pass; otherwise the current artifacts remain, with a separately audited final-direct repair allowed for remaining ERROR or supported UNRESOLVED cases.Auditing checks problem fidelity, model-code agreement, data binding, result extraction, edit scope, and preservation.Source: §3.5 · p. 7

4.1 Setup and Evaluation Metrics

• Evaluation paired fixed problem, model, and code artifacts across six benchmarks, using outputs from three zero-shot frameworks and four fine-tuned generators.Accuracy checks objective values or required solve status; Rescue, Regression, and Error Coverage measure correction, damage to initially correct outputs, and screening of initially incorrect outputs.Source: §4.1 · p. 7

4.2 Post-Processing Generation Frameworks and Fine-Tuned Models

• Zero-shot accuracy rose from 44.8% to 64.9% with 36.9% Rescue; fine-tuned generators gained 4.1, 7.7, 8.6, and 8.7 percentage points.Regression stayed at or below 1.0%. Blind review found source-level artifact fidelity rose from 2.7% to 51.8% among 330 adjudicable pairs, with 50.8% Artifact Rescue.Source: §4.2 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 8

4.3 Comparison with Generic Repair Baselines

• Against repair baselines, OPT-Kraken achieved 64.9% final accuracy, 36.9% Rescue, and 0.6% Regression while screening 94.1% of initially incorrect outputs.The two enhanced generic baselines improved Rescue and Error Coverage but introduced more Regression.Source: §4.3 · p. 9, Table 4 · p. 9

4.4 Repair-Path Contributions Across Upstream Error Structures

• Local repair supplied 69.9%, 81.7%, and 48.5% of successful Rescues for DeepSeek, Qwen, and GPT, while structural and direct paths also varied by upstream.These are shares conditional on successful Rescue, not routing accuracy or a local-only counterfactual.Source: Figure 4 · p. 9, §4.4 · p. 10

4.5 Mechanism-Wise Ablation Study

• Removing detection mechanisms reduced final accuracy, Rescue, and Error Coverage by 14.7, 24.5, and 7.0 points; granularity and contracted-repair ablations also lowered accuracy and Rescue.The corresponding drops were 9.2 and 16.3 points under the granularity ablation, and 7.1 and 12.5 points under the contracted-repair ablation.Source: §4.5 · p. 10, Table 5 · p. 10
• Figure 5 profiles 688 train-free and 950 fine-tuned pre-repair findings; implementation and reporting account for 24.3% of train-free findings.For fine-tuned outputs, balance and bounds account for 31.3%, and relational algebra for 14.9%.Source: §4.5 · p. 10, Figure 5 · p. 11

5 Conclusion

• OPT-Kraken combines dual-evidence diagnosis, evidence-guided repair-granularity selection, contract-constrained repair, artifact-level auditing, and safe commit.It operates post hoc on frozen problem–model–code artifacts.Source: §5 · p. 11
• The framework makes error correction and content preservation joint requirements of optimization-modeling repair.Its post-hoc transformation targets frozen problem–model–code artifacts.Source: §5 · p. 11
• Across six benchmarks, OPT-Kraken improves accuracy and Rescue while maintaining broad Error Coverage and low Regression.The paper frames this as moving beyond a sole focus on initial success toward evidence-grounded screening and risk-controlled artifact repair.Source: §5 · p. 11

Reproducibility Statement

• The repository releases the solver-independent post-processor and runnable interface, while appendices document the catalog, prompts, evaluation, error profiles, and an end-to-end repair record.Sections 3–4 specify the interface, fidelity checks, repair hierarchy, contracts, evaluation protocol, and metrics.Source: Reproducibility Statement · p. 12

A.1 Operational RepairCard Catalog

• The operational RepairCard catalog contains 24 reusable categories spanning formulation, data binding, model-to-code translation, and result extraction.Each card specifies applicability, required evidence, authorized operators, protected content, and effect checks.Source: §A.1 · p. 15, Table 6 · p. 16

A.2 A Formal-Run Objective-Repair Artifact

• For an objective error, dual-evidence checking localizes inconsistent coefficients and selects O02 when the profit metric and objective direction are already correct.The example’s source-grounded coefficients are 0.65 and 0.45, rather than the generated 1.15 and 0.65.Source: §A.2 · p. 16

A.3 RepairTask and Compiled Contract

• The compiled contract permits replacing the objective and its two affected terms while freezing every variable and constraint.The RepairTask binds source spans, observed and intended coefficients, and a stable target reference.Source: §A.3 · p. 17

A.4 Code-Form Ratio-Repair Trace

• The O15 code-form trace shows an atomic ratio repair that changes only a constraint’s relation, protecting its coefficients, right-hand side, and other model elements.The passage identifies this as an illustrative mechanism example, not an additional benchmark result.Source: §A.4 · p. 19, Figure 6 · p. 19

A.5 RepairCard Field Quick Reference

• The O01–O24 quick reference is a conceptual projection, not the executable specification; the released implementation governs runtime behavior.The implementation includes additional evidence schemas, operator restrictions, protected fields, acceptance predicates, uncertainty handling, and code-level branching.Source: §A.5 · p. 20

B.3 CoT and Multi-Stage Prompt-Based Repair

• CoT-based Repair diagnoses and returns a complete model–code pair in one response, while Multi-stage Repair separates diagnosis, repair, independent review, and possible revision.Both preserve faithful content and return complete artifacts rather than partial patches.Source: §B.3 · p. 23
• Both enhanced baselines use the same 801 frozen tuples at temperature zero without references or execution feedback; Multi-stage allows up to three review rounds and nine logical calls.CoT makes one call, and the last complete candidate is executed once; Error Coverage uses each method’s native screening signal, not a calibrated cross-method detection comparison.Source: §B.3 · p. 24

B.4 Operational Realization of Detection, Routing, and Commit

• Minimum sufficient scope governs routing, but LLM-assisted scope and dependency proposals do not authorize writes; ambiguous evidence or unresolved dependencies can yield UNRESOLVED.Deterministic checks enforce source grounding, valid references, and permitted edits, but do not certify globally minimal edits or complete dependency discovery.Source: §B.4 · pp. 25–26
• For local repairs, the compiler turns a source-bound RepairTask into finite operations with preconditions, writable references, intended effects, and freeze rules for other model elements.Automatic predicates check the precondition, permitted diff, exact effect, frozen content, schemas, grounded data, and executable code.Source: §B.4 · p. 26
• A separate LLM audit checks problem fidelity and model–code agreement, while deterministic validation checks its schema, quotes, references, and decision consistency.The online audit does not use solver objective values or reference answers.Source: §B.4 · p. 26
• Failed audits retain the last artifact pair, and repair attempts are bounded by per-root and per-instance limits; a failed path restores the retained pair and terminates without commit.A final direct-repair candidate is allowed only for a remaining ERROR or sufficiently evidenced UNRESOLVED instance, followed by an independent audit.Source: §B.4 · pp. 26–27

C Detailed Experimental Results

• 94.1% non-CLEAN Error Coverage was reported for DeepSeek V4 Flash Zero-shot among 442 initially incorrect outputs.245 (55.4%) were assigned ERROR, 171 (38.7%) UNRESOLVED, and 26 (5.9%) CLEAN.Source: C Detailed Experimental Results · p. 27
• For DeepSeek V4 Flash Zero-shot, non-CLEAN Error Coverage was 94.1% among 442 outputs scored incorrect before repair, with coverage defined as the fraction assigned a non-CLEAN screening outcome.Source: C Detailed Experimental Results · p. 27
• Rescue measures initially incorrect outputs corrected, Error Coverage measures those assigned non-CLEAN screening outcomes, and Regression measures initially correct outputs made incorrect.Source: C Detailed Experimental Results · p. 27
• Regression is not a detector false-positive rate, and neither Regression nor Error Coverage measures detection precision, specificity, or calibrated binary hallucination detection.Source: C Detailed Experimental Results · p. 27

• 4/359 (1.1%) numerically correct initial outputs contained hallucinations whose effects were masked in those instances.Source: §C.1 · p. 27, Table 7 · p. 28
• Outcome correctness and source fidelity are related but non-equivalent; paired fidelity analysis complements, rather than replaces, outcome-level evaluation.Source: §C.1 · pp. 27, 30
• Reviewers assessed 339 initially outcome-incorrect artifacts in blinded, shuffled pairs, assigning faithful, unfaithful, or indeterminate source-fidelity labels.Source: §C.1 · p. 28
• 49.1 percentage points: source-level fidelity rose from 2.7% (9/330) to 51.8% (171/330), with 163/321 initially unfaithful artifacts restored (50.8%).Source: §C.1 · p. 29, Table 9 · p. 29
• Eight of nine initially faithful adjudicable artifacts remained faithful; one became unfaithful, while nine indeterminate pairs stayed excluded from primary binary rates.Source: §C.1 · p. 30
• Treating all nine indeterminate pairs as either unfaithful or faithful still yields post-repair fidelity of 50.4% or 53.1%, respectively.Source: §C.1 · p. 30

• ULE requires more evidence before repair eligibility is determined; ULC conservatively retains the current artifacts.Source: §C.2 · p. 30
• 44.8% to 64.9% accuracy, with 163/442 initially incorrect artifacts rescued, 2/359 initially correct artifacts regressed, and 94.1% Error Coverage.Source: §C.2 · p. 30, Table 10 · p. 30
• 339/416 initially incorrect artifacts assigned a non-CLEAN screening outcome entered ERROR or ULE follow-up, while 77/416 were conservatively routed to ULC.Source: §C.2 · p. 30
• Of 53 initially correct artifacts routed to ERROR or ULE, 51 remained correct after the full pipeline and two regressed.Source: §C.2 · p. 31

• Each panel plots benchmark-wise Rescue rates, while its title gives the method's overall Rescue rate.Source: §C.3 · p. 31, Figure 8 · p. 31
• The top row shows four training-free pipelines and the bottom row four fine-tuned generators; the shared 66.7% outer ring is the largest observed benchmark-level rate.Source: §C.3 · p. 31, Figure 8 · p. 31

C.4 Training-Free Generation Pipelines

• DeepSeek V4 Flash served as OPT-Kraken's repair backbone for the training-free pipeline results, with a full six-benchmark breakdown reported.Source: §C.4 · p. 31, Table 11 · p. 31

C.5 Qwen3.8-27B Zero-Shot Generation

• The Qwen3.8-27B zero-shot results provide a full six-benchmark breakdown and retain its local-service configuration without an explicit thinking override.Source: §C.5 · p. 32, Table 12 · p. 32

C.6 Model Calls, Token Usage, and Runtime

• 15.70 seconds per instance end to end (median 9.48 seconds): OPT-Kraken's timing includes detection, repair, execution, artifact audit, and retained-output selection.Source: §C.6 · p. 33, Table 13 · p. 33
• OPT-Kraken averages 3.65 logical calls versus 3.71 for multi-stage repair, with its calls covering evidence extraction, consistency checks, scope selection, candidate generation, and auditing.Source: §C.6 · p. 33

C.7 Mechanism-Wise and Fixed-Entry Ablations

• Ablations B and C retain Full’s 94.1% Error Coverage because coverage is finalized before hierarchical repair.Their zero Coverage deltas are expected because B and C act downstream; Table 14 reports signed changes relative to Full.Source: §C.7 · p. 33, Table 14 · p. 34
• 27.5% versus 7.5% Rescue when contracts replace generic local rewriting under the same fixed-entry conditions.The variants share frozen detection, routing, cases, correction budget, execution feedback, audit, and other repair paths.Source: §C.7 · p. 34, Table 15 · p. 34
• 36.9% versus 24.4% end-to-end Rescue for Full and Ablation C, a 12.5-point gap after other repair paths partly recover local failures.The controlled fixed-entry gap is 20.0 points; the end-to-end rates cover all 801 artifacts.Source: §C.7 · p. 34
• OPT-Kraken rescues 11/40 erroneous artifacts versus SAC-Opt’s 1/40, with zero Regression; paired outcomes favor OPT-Kraken (two-sided exact McNemar p = 0.00635).SAC-Opt initiates no repair for 123/140 inputs; among 17 supported targets, it rescues 1/8 errors and regresses 2/9 correct controls.Source: §C.7 · p. 34, Table 16 · p. 34

D Hallucination Taxonomy and Error Profiles

• Experts iteratively developed the hallucination taxonomy and RepairCards using naturally generated artifacts and controlled cases, then froze them before evaluation on six disjoint benchmarks.Controlled cases informed taxonomy and card development only; formal correctness labels came from objective or solve-status criteria, not expert judgments.Source: D Hallucination Taxonomy and Error… · p. 35

D.1 DeepSeek V4 Flash across Eight Upstreams

• Tables 17–18 profile O01–O24 findings separately for four training-free and four fine-tuned upstreams, totaling 688 and 950 findings, respectively.Source: §D.1 · p. 35, Table 17 · p. 35, Table 18 · p. 36

D.2 Qwen and GPT Zero-Shot Outputs

• The Zero-shot profiles cover 319 classified Qwen findings and a GPT review of 200 retained findings across 159 artifacts; these within-column profiles are not a matched-cohort comparison.The Qwen findings come from 302 artifacts; the GPT review covers unrecovered outputs.Source: §D.2 · p. 36, Table 19 · p. 36

D.3 Outcome-Conditioned O01–O24 Composition

• Tables 20–22 show within-outcome percentages of retained classified findings by operational type, not overall Rescue, Regression, or Error Coverage rates.GPT-5.6 Sol has zero Regression and no samples, so its Regression distribution is undefined and shown with dashes.Source: §D.3 · pp. 37–38, Table 20 · p. 37, Table 21 · p. 38

E End-to-End Repair Output

• The end-to-end trace records the detector output, committed model, synchronized solver code, and final audit.Source: E End-to-End Repair Output · p. 38

E.1 Hallucination Record and Candidate Patch

• The detector grounds its diagnosis in price, raw-material cost, and variable-cost statements, authorizing one model-level target.Source: §E.1 · p. 38

E.2 Final Mathematical Model and Solver Code

• The committed model changes only objective coefficients; the synchronized solver code implements the approved patch through the required solve_model() interface.Resource constraints, bounds, and variable domains remain unchanged; the wrapper returns the required four-field record.Source: §E.2 · pp. 39–40

E.3 Execution, Artifact Audit, and Commit

• The repaired program reaches objective 77,500, passes the artifact audit, and is retained as the output.Source: §E.3 · p. 40