PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models

Rui Zhu, Minglong Cao, Chenyu Zhou, Jianghao Lin, Dongdong Ge

arXiv:2609.09957v1math.OCcs.AI

TL;DR

OR language-model training still relies on costly verified supervision, coarse credit assignment, and privileged-context self-distillation that can cause style mismatch. SOLID uses executable on-policy rollouts, solver-outcome consensus, and masked self-distillation to provide evaluator-free sequence- and token-level supervision. Across challenging benchmarks, it improves solution accuracy for both general-purpose and OR-tuned models over outcome-only group-relative training.

  • Problem

    OR post-training lacks scalable supervision without verified answers, fine-grained credit assignment without costly evaluators, and deployment-compatible self-distillation.

  • Method

    SOLID executes multiple candidate programs, selects a majority-group solver artifact as a pseudo-reference, and combines group-relative advantages with masked token-level self-distillation.

  • Results

    SOLID consistently outperforms self-distillation and label-free reinforcement-learning methods across challenging benchmarks for both base and post-trained models.

  • Takeaways & Limitations

    Solver artifacts from a model’s own rollouts can support scalable OR self-improvement without verified answers or external evaluators.

  • Takeaways & Limitations

    Uniform OPSD supervision applies across the response and can suffer from style mismatch, motivating SOLID’s LP-structure masking.

Abstract

from arXiv · show

Modern large language models (LLMs) can translate natural-language descriptions into operations research (OR) formulations. Post-training techniques including reinforcement learning and on-policy self-distillation have further improved this capability. However, three limitations remain in training LLMs for OR formulations. First, training commonly relies on synthetic formulations validated by human experts or stronger models, constraining scalable supervision. Second, credit assignment is either coarse or costly: outcome rewards score an entire trajectory without locating the responsible modeling decision, whereas process-level supervision requires an additional evaluator. Third, privileged self-distillation can induce style mismatch by using solver context unavailable at deployment. We find that a model can improve from solver-artifact feedback generated by its own rollouts, making self-distillation a practical, evaluator-free source of dense supervision. Therefore, we propose SOLID: Solver-Informed On-Policy LearnIng through Self-Distillation, a novel framework for self-improving OR language models without verified answers or external evaluators. SOLID executes candidate programs from multiple rollouts, clusters their objectives, and selects a majority-group artifact as a pseudo-reference. The model then performs updates using group-relative advantages and dense self-supervision signals. Across multiple OR benchmarks, SOLID improves solution accuracy for both general-purpose and OR-tuned models over outcome-only group-relative training. These results show that solver artifacts can support scalable self-improvement without trusted answers.

1. Introduction

SOLID addresses costly supervision, coarse credit assignment, and privileged-context style mismatch in OR language-model training. It uses solver artifacts from on-policy rollout groups for sequence- and token-level self-supervision, improving accuracy across challenging benchmarks without verified answers.

  • Motivation: The framework targets three OR-training bottlenecks: costly supervision, coarse within-trajectory credit assignment, and privileged-context style mismatch.These arise because validation needs trusted answers or expertise, outcome rewards do not localize modeling choices, and privileged context is unavailable at deployment.
  • Motivation: On-policy self-distillation offers dense credit without a stronger teacher, but privileged-context re-scoring can create style mismatch in OR.The framework combines this economical supervision with masking to mitigate harmful updates from mismatched reasoning.
  • Method: SOLID combines group-relative solver-outcome learning with token-level self-distillation from a majority-group solver artifact.A sample mask restricts self-distillation to structurally mismatched variable, objective, constraint, and code sections.
  • Method: SOLID formulates post-training as a closed self-bootstrapping loop over executable on-policy rollouts and solver-outcome consensus, requiring no external evaluator.This enables learning from OR problems when descriptions and solver access exist without verified answers.
  • Results: SOLID consistently outperforms self-distillation and label-free reinforcement-learning methods across challenging benchmarks.The reported effectiveness extends to both base and post-trained models.

2. Related Work

Related work spans language-model optimization modeling, solver-informed supervision, and self-supervised post-training. Prior methods use solver artifacts, majority agreement, or privileged-context self-distillation, often with annotations, evaluators, or external teachers.

  • Language Models for Optimization Modeling: Optimization-modeling systems progressed from structured LP-component generation to end-to-end formulation, code generation, solving, and agentic tool interaction.This literature includes constrained generation, human validation, and external interaction protocols.
  • Solver-Informed Supervision: Solver-informed methods use executable programs and optimization artifacts to filter data or construct learning signals for OR post-training.Examples include verifiable rewards from code and LP artifacts, process reward models, and semantic-execution rewards with external teachers.
  • Self-Supervised Post-Training: Self-supervised post-training derives signals from on-policy rollouts through sequence-level majority agreement and token-level privileged-context self-distillation.TTRL uses group-relative rewards without ground truth, while OPSD provides dense guidance without a stronger teacher.

3. Preliminaries

The preliminaries define OR responses as structured progressions ending in executable solver code and review sequence-level, step-level, and token-level supervision. They motivate SOLID by showing that uniform token-level self-distillation and sequence-level rewards do not localize responsible modeling choices.

  • OR Response Structure: OR responses progress from problem interpretation through variables, objective, constraints, mathematical model, and executable solver code.The rollout policy samples multiple structured responses, which can then be executed and compared using solver outcomes and artifacts.
  • Step-level Credit Assignment: Step-level methods redistribute supervision across intermediate decisions, matching OR’s natural structure of variables, objectives, constraints, models, and code.StepOPSD uses action-centered segments, while StepORLM makes intermediate OR structure explicit.
  • Reinforcement Learning without Labels: Sequence-level reinforcement learning normalizes group rewards and applies a PPO-style clipped update without a learned value model.This is the GRPO-based implementation used for label-free OR reinforcement learning.
  • Step-level Credit Assignment: Every token shares the same sequence-level advantage, so the update cannot locate the responsible modeling choice.This limitation follows from assigning one trajectory-level signal across variable definition, objective, constraints, and code decisions.
  • On-policy Self-distillation: Whole-response OPSD supplies dense token-level supervision but applies it uniformly and omits the group-relative objective.SOLID instead uses a majority-LP pseudo-reference and a binary LP-structured mask to localize supervision and mitigate style mismatch.

4. Methodology

SOLID combines solver-evaluated on-policy rollouts, majority-group pseudo-references, and masked self-distillation to train a deployable OR policy. Its updates pair group-relative sequence rewards with token-level corrections restricted to solver-identified structural mismatches.

  • 4.1. SOLID: The LP-structured mask localizes self-distillation to mismatched variable, constraint, objective, and code sections identified by candidate–reference artifact comparison.The mapping covers response sections K = {3, 4, 5, 9}.
  • 4.1. SOLID: The rollout template separates modeling stages so solver evidence can be mapped to corresponding response sections during contextual re-scoring.The reference field is empty during rollout and contains the majority-LP pseudo-reference only during re-scoring.
  • 4.1. SOLID: SOLID evaluates multiple on-policy programs, clusters executable objectives, and uses the rollout closest to the largest cluster median as a majority-LP pseudo-reference.If no rollout executes, no reference is constructed.
  • 4.1. SOLID: Group-relative rewards and masked KL self-distillation jointly update the policy through LGRPO + βLKL.Majority-cluster responses receive GRPO only, whereas non-majority responses can receive self-distillation in LP-mismatched mapped sections.
  • 4.1. SOLID: The rollout policy re-scores responses with privileged pseudo-reference context, while the updated student conditions only on the original problem.This same-policy comparison supplies token-level credit without verified answers or another evaluator.

5. Anchored Correction under Style Mismatch

SOLID separates contextual correction from style mismatch by combining an LP-structured mask with group-relative anchoring. The mask limits where contextual updates act, while the group advantage controls whether they can alter the sampled response’s update direction.

  • 5. Anchored Correction under Style Mismatch: Uniform self-distillation can conflate task correction with changes in wording, ordering, or derivation detail caused by privileged context.SOLID assigns localization to the structural mask and correction strength to the group advantage.
  • 5. Anchored Correction under Style Mismatch: Non-majority rollouts are vote-negative but are not necessarily incorrect, so negative-sample reinforcement can suppress a locally useful token.The anchored sign-reversal result addresses this risk only within activated sections.
  • 5. Anchored Correction under Style Mismatch: The LP-structured mask removes the KL channel for majority, matched, and unmapped tokens, regardless of surface-form shifts.Only activated LP-mismatched sections can receive contextual correction.
  • 5. Anchored Correction under Style Mismatch: For negative group advantages, weak contextual preferences cannot reverse a negative update, whereas strong LP-informed preferences can preserve a locally useful token.The group advantage therefore anchors the direction of contextual correction.

6. Experiments

SOLID is evaluated against outcome-only and whole-response self-distillation baselines on three OR benchmarks, with matched-budget results for both general-purpose and OR-tuned models. It improves cross-dataset performance while localized masking avoids the decline observed with pure OPSD.

  • Main Results: 6.02 points in majority accuracy and 4.09 points in average pass@k separate SOLID from TTRL across the 266 OptMATH and InOR problems for Qwen3-4B-Instruct.On OptMATH alone, majority accuracy rises by 9.64 points and pass@k improves by an average of 6.31 points.
  • Main Results: SOLID improves or matches majority accuracy on all three datasets for the OR-tuned model and outperforms TTRL on eight of twelve metrics overall.On MAMO-Complex, pass@k improves by an average of 3.75 points; OptMATH and InOR remain broadly comparable to TTRL.
  • Reasoning Capacity: The privileged-context diagnostic indicates that prompted students can exploit sound solver information, but correction quality is sensitive to the reference source.This motivates combining group-relative learning with localized self-distillation rather than relying on privileged context alone.
  • Masking Ablation: LP-structured masking provides the strongest cross-dataset balance, leading most OptMATH and MAMO-Complex metrics while remaining competitive on InOR.The matched-budget ablation compares no KL, whole-response KL, random section masks, and SOLID’s LP-structured localization.
  • Training Dynamics: The complete SOLID objective maintains an advantage over TTRL, whereas pure whole-response OPSD rises briefly and then declines under both majority-LP and diagnostic correct-LP context.The sharper majority-LP decline and the correct-LP diagnostic indicate that whole-response self-distillation can fail even when reference error is removed.
  • Further Discussion: Future extensions target more reliable LP extraction, stronger pseudo-reference aggregation, and adaptive confidence-weighted masking.These changes are proposed for cases where solver artifacts are incomplete, rollout consensus is incorrect, or current gains are less consistent.

7. Conclusion

SOLID bootstraps OR language models from unlabeled problems using on-policy rollouts, solver outcomes, and localized self-distillation. Improvements across general-purpose and OR-tuned models support solver-informed self-improvement while the OPSD diagnostic exposes a self-distillation failure mode.

  • Conclusion: SOLID combines solver-outcome sequence learning with same-policy token-level self-distillation localized to mismatched reasoning sections.The LP-structured mask targets variable, objective, constraint, and code sections rather than applying guidance uniformly.
  • Conclusion: Improvements across general-purpose and OR-tuned models show that solver-artifact feedback remains useful before and after domain-specific tuning.The framework is presented as operating on unlabeled OR problems with on-policy rollouts and solver outcomes.
  • Conclusion: The OPSD diagnostic exposes a self-distillation failure mode in OR and motivates LP-structure masking.Pure whole-response self-distillation can decline even with diagnostic correct-LP context, linking the failure to contextual style mismatch without proving it is the unique cause.

Appendix B. Benchmark Details

The appendix defines the evaluation benchmarks and solver-execution criteria used to assess generated OR programs. It also specifies the required response structure and gurobipy implementation conventions.

  • Output Format: The requested answer format contains ordered sections for mathematical modeling, nonlinear relationships, the final model, and Python code using gurobipy.The code requirements include importing gurobipy and GRB, creating gp.Model("model"), and using explicit Gurobi variable types.
  • Evaluation Sets: The evaluation uses OptMATH, corrected MAMO-Complex, and corrected IndustryOR benchmark releases.OptMATH contains 166 instances, while the corrected IndustryOR release retains all 100 problems.
  • Evaluation Sets: The benchmark appendix records Table 5 as the summary of the exact evaluation sets used in the experiments.The supplied table material identifies the benchmark-size summary but does not provide cell values.
  • Evaluation Procedure: Generated programs count as correct only when execution succeeds and the extracted objective satisfies the benchmark criterion.Missing code, extraction failures, timeouts, exceptions, nonnumeric objectives, and nonoptimal outcomes are scored as incorrect.

Appendix C. Additional Reproducibility Details

The reproducibility setup specifies software versions, rollout and execution infrastructure, extraction fallbacks, and deterministic selection of a solver artifact for voting.

  • Software Environment: The implementation uses asynchronous vLLM rollouts with veRL, Gurobi for Qwen3-4B-Instruct, and COPT for StepORLM.The recorded environment requires Python 3.10 or newer, PyTorch 2.6 to below 2.10, vLLM 0.8.5 to below 0.16, gurobipy 12.0 or newer, and coptpy 7.2 or newer.
  • Code Extraction: A deterministic extraction cascade searches tagged code, Python Markdown fences, raw solver imports, and finally designated step blocks.Responses without extractable solver code are marked as failed.
  • Execution Infrastructure: Generated programs run in isolated worker processes with a 30-second timeout, but the execution layer is not a hardened security sandbox.No explicit per-program CPU or memory quota is set; cluster scheduling governs those resources.
  • Artifact Selection: Only completed executions with finite numeric objectives enter the vote, with objective clusters ranked by size, deviation, and rollout order.Within the winning cluster, the usable LP closest to the median is selected, with rollout order breaking ties.

Appendix D. Proofs for Anchored Correction

The appendix derives the anchored correction mechanism under fixed-teacher and fixed-mask assumptions. It characterizes how contextual token preferences combine with policy-gradient updates and where the approximation is valid.

  • Masked Contextual Direction: Under fixed teacher probabilities, masks, advantages, and averaging weights, the contextual channel changes token scores according to the teacher-to-policy probability relationship.It increases a sampled token’s score when q_i,t exceeds p_i,t, decreases it when q_i,t is smaller, and vanishes when the gate is zero.
  • KL Interpretation: The reverse-KL interpretation is unbiased under exact student sampling but becomes a nonnegative surrogate under replay without importance correction.The two interpretations coincide at the on-policy snapshot.
  • Combined Gradient: The combined gradient depends on compatible token-averaging measures or an effective mask that absorbs normalization differences.The unscaled gate is valid when both losses share a normalizer or when the scale is absorbed into β.
  • Approximation: The small-shift Taylor expansion is only a local approximation and should not be extrapolated to large |Δ_i,t|.The derivation uses the gate bound 0 ≤ b_i,t ≤ 1.

D.3. Proof of Proposition 1

The proposition establishes when contextual correction can reverse a negative update. Structural masking restricts this reversal to activated, error-related reasoning stages rather than style changes or consensus-equivalent responses.

  • D.3. Proof of Proposition 1: For A_i<0 and β>0, a fully activated contextual gate reverses the coefficient only when the likelihood shift exceeds the derived threshold.With b_i,t=0, the contextual channel cannot change the negative coefficient; for 0 < Δ_i,t ≤ τ_i, the effective coefficient remains nonpositive.
  • Boundary Cases: Majority-cluster rollouts and structurally matched or unmapped sections are masked out of the contextual update.The gate blocks the update regardless of teacher probability or surface-form shift.
  • Anchored Correction: A negative group signal anchors the update, so only a sufficiently strong LP-informed preference can preserve or reverse a locally useful token direction.The LP-structured mask confines this behavior to error-related reasoning stages and prevents style-mismatch-driven reversals.
  • Scope: The derivations do not guarantee a correct majority LP, semantic correction, or improved expected solver accuracy.They require fixed-teacher and fixed-mask assumptions and apply to the unclipped branch; clamping or clipping can alter the gradient.
Loading 2609.09957v1…