PaperlayerView exact revision on arXiv

Source-linked AI summary

OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling

Hongliang Lu, Zhonglin Xie, Yaoyu Wu, Can Ren, Yuxuan Chen, Zaiwen Wen

arXiv:2502.11102v2cs.AIcs.LG

TL;DR

High-quality optimization-modeling data remains scarce, limiting robust translation from natural-language descriptions to optimization models. OptMATH introduces a scalable bidirectional synthesis and verification framework, and models trained on its data achieve superior performance across multiple modeling benchmarks.

  • Problem

    High-quality optimization-modeling datasets are scarce, making robust modeling of practical optimization problems from natural-language descriptions difficult.

  • Method

    OptMATH generates controllable problem data from curated formulations, backtranslates it into natural language, and uses forward modeling with rejection sampling to verify correspondence.

  • Results

    Models trained on OptMATH-Train achieve superior performance across multiple modeling benchmarks, with OptMATH-Qwen2.5-32B surpassing larger proprietary models across all benchmarks.

  • Takeaways & Limitations

    The framework provides verified training triplets and a challenging long-context benchmark for optimization modeling.

  • Takeaways & Limitations

    Exact equivalence between two mathematical formulations remains an open research question, despite 99.6% accuracy in manual analysis of sampled instances.

Abstract

from arXiv · show

Despite the rapid development of large language models (LLMs), a fundamental challenge persists: the lack of high-quality optimization modeling datasets hampers LLMs' robust modeling of practical optimization problems from natural language descriptions (NL). This data scarcity also contributes to the generalization difficulties experienced by learning-based methods. To address these challenges, we propose a scalable framework for synthesizing a high-quality dataset, named OptMATH. Starting from curated seed data with mathematical formulations (MF), this framework automatically generates problem data (PD) with controllable complexity. Then, a back-translation step is employed to obtain NL. To verify the correspondence between the NL and the PD, a forward modeling step followed by rejection sampling is used. The accepted pairs constitute the training part of OptMATH. Then a collection of rejected pairs is identified and further filtered. This collection serves as a new benchmark for optimization modeling, containing difficult instances whose lengths are much longer than these of NL4OPT and MAMO. Through extensive experiments, we demonstrate that models of various sizes (0.5B-32B parameters) trained on OptMATH achieve superior results on multiple modeling benchmarks, thereby validating the effectiveness and scalability of our approach. Our dataset is publicly available at https://github.com/AuroraLHL/OptMATH.

1 Introduction

Optimization modeling translates natural-language problem descriptions into solver-ready formulations, but ambiguity, multiple valid models, and limited high-quality data make the task difficult. OptMATH addresses this scarcity with a validated bidirectional synthesis framework and datasets for training and long-context evaluation.

  • Motivation: Natural-language optimization modeling is difficult because descriptions are ambiguous, formulations require specialized expertise, and multiple valid models may exist.Manual formulation typically requires iterative refinement and is time-consuming and inaccessible to many practitioners.
  • Related limitations: Existing optimization-modeling methods rely on prompting or fine-tuning, but synthetic datasets are often small, inconsistently generated, and insufficiently complex.The cited approaches include NL4Opt, OptiMUS, ORLM, and LLMOPT.
  • Contributions: OptMATH proposes triplet-aligned (NL, MF, PD) synthesis with a closed-loop workflow, optimal-value matching, and rigorous validation.The framework is designed to address data scarcity while ensuring semantic equivalence across natural language, mathematical formulation, and problem data.
  • Contributions: 53 seed generators cover 10+ real-world applications, while manual analysis reports 99.6% equivalence accuracy across generated triplets.The framework spans applications including logistics, energy, and finance.
  • Datasets: OptMATH-Train contains verified triplets, whereas OptMATH-Bench comprises rejected hard instances with extended contexts and complex constraints across LP, MILP, IP, NLP, and SOCP.The benchmark’s natural-language contexts are reported as 2.9× longer than MAMO EasyLP.
  • Results: Models trained on OptMATH-Train achieve state-of-the-art performance on established modeling benchmarks including NL4OPT and MAMO.The paper presents these experiments as evidence of the framework’s effectiveness and scalability.

2 Backgrounds & Overview

The paper distinguishes natural-language descriptions, mathematical formulations, and solver-ready problem data as complementary representations of optimization problems. Its framework generates and verifies these representations through backtranslation, forward modeling, and solver-based rejection sampling.

  • Representations: Problem data includes LP/MPS files, concrete mathematical expressions, and solver-ready code that enables obtaining an optimal value with a solver.Mathematical formulation instead represents the problem before concrete numerical values are specified.
  • Problem setting: Optimization modeling must convert informal natural-language descriptions into precise formulations and extract solver-ready data, tasks that traditionally require deep expertise.The paper identifies these conversions as practical challenges despite efficient modern solvers.
  • Autoformulation: Autoformulation maps natural language to generated mathematical formulations and problem data, with a quality metric evaluated against verified triplets.The modeling prompt template produces MF′ and PD′, which are assessed using the verified (NL, MF, PD) data.
  • Pipeline: The reverse-generation phase uses LP/MPS files and expert-curated seed generators, then backtranslates their formulations and data into natural-language descriptions.The seed sources include challenging benchmarks and more than 50 expert-curated generators.
  • Pipeline: Forward modeling translates generated descriptions back into solver-code problem data, retaining only instances whose original and generated LP files have matching optimal objective values.This rejection-sampling criterion is used to filter the training data for quality.
  • Training: Accepted instances are augmented and used to fine-tune AutoFormulator, a specialized model for automated mathematical optimization modeling.The cited passage describes the enriched training pairs as the input to foundation-model fine-tuning.

3 Feedback-Driven PD Generation

OptMATH generates problem data from configurable seed generators and measures modeling complexity using variable, constraint, formulation, and expression features. Parameter-selection feedback keeps generated instances within target complexity, feasibility, and runtime requirements.

  • Seed generation: More than 50 seed problem classes each have an instance generator that produces problem-data distributions with controllable scale and complexity.The generators use problem-specific configurations to vary the resulting instances.
  • Complexity measurement: The complexity score combines counts of binary, integer, and continuous variables with linear, indicator, quadratic, and general nonlinear constraints.The score also includes Big-M frequency and average expression length.
  • Complexity measurement: The scoring function includes tunable weights that determine how each variable, constraint, Big-M, and expression component contributes to overall complexity.The weights reflect the contribution of each component to the score.
  • Controlled generation: Parameter selection begins with target bounds and uses prompt-based configuration to generate instances satisfying complexity, feasibility, and solving-time requirements.The workflow is formalized in Algorithm 1.
  • Controlled generation: Runtime thresholds and an LLM feedback pipeline keep generated problem data expressive, feasible, and computationally tractable.The paper states that the feedback process regulates both complexity and feasibility.

4 The Data Synthesis Framework

The framework synthesizes optimization data bidirectionally: it generates natural-language descriptions from formulations and problem data, then forward-models them for solver-based validation. Rejection sampling retains instances whose generated and original solutions agree, while manual analysis reports 99.6% equivalence accuracy on a 1% sample.

  • Framework overview: The framework uses an LLM for reverse generation and a fine-tuned AutoFormulator for forward modeling within a bidirectional synthesis workflow.The workflow generates natural-language descriptions from mathematical formulations and problem data, then transforms those descriptions back into formulations and solver code.
  • Backtranslation pipeline: Backtranslation proceeds through initial generation, self-criticism, and self-refinement to improve mathematical accuracy, constraint completeness, and description clarity.The self-criticism stage checks equivalence, completeness, clarity, and parameter consistency before refinement.
  • Validation: Generated descriptions are forward-modeled into mathematical formulations and solver code, with optimal values compared against those from the original instances.Only samples passing the solution-based comparison are accepted into the dataset.
  • Validation: The dataset retains samples when rejection sampling finds matching solutions, providing a practical mechanism for filtering semantically consistent optimization instances.The authors note that matching optimal values does not guarantee perfect equivalence between optimization problems.
  • Validation: 99.6% equivalence accuracy was found in manual analysis of a randomly sampled 1% of the dataset.The authors identify exact equivalence determination as an open research question despite the reported practical accuracy.

5 Fine-Tuning

Fine-tuning trains AutoFormulator with OptMATH triplets to generate mathematical formulations and solver code from natural-language descriptions. Data augmentation expands the training distribution, and independently sampled augmented descriptions undergo rejection sampling.

  • Data augmentation: Data augmentation applies rewriting, semantic substitution, constraint expansion, and numerical augmentation rules to create more non-standard problems.The stated goal is to improve dataset diversity and model generalization performance.
  • Data augmentation: Approximately 10 qualified augmented datasets are produced per problem, and augmentation is applied to 50 thousand instances.Each augmented description is sampled twice independently and filtered using the rejection-sampling procedure.
  • Fine-tuning: The model is fine-tuned with LoRA to generate mathematical formulations and solver code from natural-language problem descriptions.The target concatenates the formulation and problem data, enabling unified sequence-to-sequence generation.
  • Fine-tuning: Training uses the standard sequence-to-sequence loss over target tokens conditioned on the problem description.The objective predicts each target token from preceding tokens and the input description.

6 Experiments

Experiments characterize OptMATH as a large, diverse, and complex dataset, then evaluate models trained on it across multiple benchmarks and scales. The results show strong benchmark performance, broad problem coverage, and diminishing gains as model or data size increases.

  • 6.1 Statistics of the OptMATH Dataset: Over 600,000 quality-filtered LP files span 53 problem types across five hardness levels.Their lengths range from 1,000 to 25,000 characters, with balanced proportions concentrated at medium difficulty.
  • 6.1 Statistics of the OptMATH Dataset: OptMATH contains more complex problem descriptions than existing benchmarks, with longer descriptions posing greater comprehension and reasoning challenges for LLMs.The comparison uses average question lengths shown in Figure 3.
  • 6.1 Statistics of the OptMATH Dataset: OptMATH covers LP, MILP, IP, NLP, SOCP, and other optimization problems across diverse application scenarios.The benchmark selects representative mathematical optimization problems from these categories.
  • 6.1 Statistics of the OptMATH Dataset: t-SNE projections form distinct clusters by source, while OptMATH surrounds other benchmarks and captures diverse problem families.The authors connect this distribution with improvements across benchmarks after training on OptMATH-Train.
  • 6.2 Autoformulation: OptMATH-Qwen2.5-32B achieves superior performance across all evaluated benchmarks, while OptMATH-Qwen2.5-7B surpasses the similarly sized ORLM-LLaMA-3-8B.The 32B model also surpasses GPT-3.5-Turbo, GPT4, and Deepseek-V3 despite their larger parameter counts.
  • 6.2 Autoformulation: All Qwen2.5 models improve substantially after OptMATH-Train fine-tuning, but relative gains diminish as model size increases and data grows.The data-scaling study also finds diminishing gains, with smaller models more sensitive to additional data.

7 Conclusion

The paper concludes by presenting OptMATH as a bidirectional data synthesis framework for optimization modeling. Evaluation on NL4OPT, MAMO, and OptMATH-Bench shows that AutoFormulator generates accurate, well-formed models compared with baseline approaches.

  • 7 Conclusion: OptMATH combines reverse data generation with autoformulation to synthesize optimization-modeling data bidirectionally.Reverse generation uses iterative LLM refinement, while autoformulation translates natural language into mathematical representations.
  • 7 Conclusion: AutoFormulator demonstrates superior performance in generating accurate and well-formed optimization models on NL4OPT, MAMO, and OptMATH-Bench.The comparison is against baseline approaches.

Impact Statements

OptMATH provides a large-scale training set and a challenging benchmark for optimization modeling. The dataset is intended to help non-experts translate real-world problems into mathematical formulations and improve LLM-based modeling.

  • Impact Statements: OptMATH comprises the large-scale OptMATH-Train dataset and the challenging OptMATH-Bench benchmark.Its structured data is presented as supporting optimization modeling and broader integration.

A.1 An Introduction of Different Benchmarks

The evaluation uses NL4OPT, MAMO, and OptMATH-Bench to assess optimization modeling across differing problem types and difficulty. OptMATH-Bench broadens coverage beyond linear programming with challenging problems spanning multiple mathematical optimization families.

  • Evaluation preparation: The dataset analysis examines benchmark scenario distributions, while NL4OPT ground truth was generated with an LLM and then expert-validated and corrected.
  • Benchmark composition: NL4OPT evaluation retained 245 high-quality test instances after filtering low-quality examples.
  • Benchmark composition: MAMO contains 652 Easy LP and 211 Complex LP instances covering LP and MILP scenarios, but excludes nonlinear programming.
  • OptMATH-Bench: OptMATH-Bench was constructed because NL4OPT and MAMO insufficiently represent broader optimization modeling capabilities and are limited to linear programming.
  • OptMATH-Bench: OptMATH-Bench spans LP, MILP, IP, NLP, SOCP, and other common optimization problems across broad application scenarios, with inherently challenging instances.
  • OptMATH data: OptMATH-Train contains over 150k reverse-generated samples and 50k augmented instances, with sequence lengths centered around 5,000 characters and generated LP files organized across five difficulty levels.

C.2 Ablation Study on the Impact of Self-Refine Iterations

The ablation study evaluates whether self-critique and self-refinement improve the backtranslation pipeline and examines acceptance-rate behavior under different iteration limits. The pipeline combines generation, critique, refinement, and rejection sampling to filter generated descriptions.

  • Ablation design: The study compares a Generate-only process with a complete Generate, Self-criticize, and Self-refine pipeline.
  • Self-refine iterations: The acceptance-rate experiment randomly evaluated 500 instances while varying the maximum iteration limit of the self-refine loop.
  • Backtranslation: Backtranslation transforms mathematical formulations and LP data into human-readable natural-language optimization descriptions.
  • Iterative training: The training framework uses Qwen2.5 base models from 0.5B to 32B parameters and describes AutoFormulator as an iterative data-quality feedback cycle.

D.4 Detailed Ablation Studies on Model Size and Data Size

The ablations examine how model scale and training-data volume affect optimization modeling across benchmarks. Larger models improve strongly on simpler tasks but remain limited on complex benchmarks, while performance typically stabilizes after using about 40% of training data.

  • Model size: Model-size experiments compare baseline and OptMATH-Train-finetuned Qwen2.5 models from 0.5B to 32B parameters across multiple benchmarks.
  • Model size: NL4OPT accuracy rises from 12.7% at 0.5B to 96.7% at 32B, while MAMO EasyLP increases from 31.9% to 90.5%.
  • Model size: At 32B, accuracy reaches 52.6% on MAMO ComplexLP and 30.6% on OptMATH-Bench, indicating more modest gains on harder benchmarks.
  • Fine-tuning effects: OptMATH-Train fine-tuning provides consistent improvements on simpler benchmarks and substantial gains across model sizes on MAMO ComplexLP and OptMATH-Bench.
  • Data scaling: Smaller models show greater initial training variance, whereas 7B and 14B models have more stable learning curves from the outset.
  • Data scaling: Models generally reach relative performance stability after using approximately 40% of the training data, with the remaining data mainly supporting fine-tuning adjustments.

E Prompt Templates

The appendix presents prompt templates and implementation examples for reverse generation, critique and refinement, baseline modeling, AutoFormulation, parameter selection, and augmentation. These templates require complete natural-language descriptions, mathematical formulations, solver code, validation, and structured parameter outputs.

  • E Prompt Templates: The prompt-template appendix organizes instructions into reverse data generation, baseline prompting, AutoFormulation, configuration selection, metadata, generator, and augmentation subsections.
  • E.1 Reverse Data Generate Prompt: Reverse-generation prompts transform mathematical expressions and LP data into complete, accurate natural-language descriptions without mathematical notation or meta-commentary.
  • E.2 Baseline Prompt: The baseline prompt asks for a mathematical model and corresponding gurobipy code, including explicit attention to variable integrality and valid constraint operators.
  • E.3 AutoFormulation Instructions: AutoFormulation instructions structure solutions around variables, objectives, constraints, mathematical formulation, Gurobi implementation, and results interpretation.
  • E.4 Prompts for Configuration Selecting: Configuration-selection prompts analyze optimal-solution rates, requirement satisfaction, solve times, model structure, complexity scores, and parameter distributions before returning JSON suggestions.
  • E.5 An Example of Metadata / E.6 An Example of Generator: The metadata and generator examples document references and LaTeX expressions, define generator parameters, and implement a bin-packing objective with capacity constraints.
  • E.7 Augmentation Prompt: The augmentation prompt rephrases problem descriptions while maintaining the same mathematical structure.
Loading 2502.11102v2…