PaperlayerView exact revision on arXiv

Source-linked AI summary

NLData2Opt: Benchmarking Parameter Derivation and Prediction for Modeling Optimization Problems from Natural Language and Data

Lijia Liu, Kyohei Atarashi, Jiyi Li, Koh Takeuchi, Shunji Umetani, Hisashi Kashima

arXiv:2610.22889v1cs.CL

TL;DR

Existing benchmarks often assume complete, relevant inputs, unlike practical settings where data may be noisy, incomplete, or separated from problem descriptions. NLData2Opt tests parameter extraction, derivation, and prediction using separate problem descriptions and data. Evaluations find that current LLMs and baselines achieve limited success, while potential pretraining-data contamination cannot be fully ruled out.

  • Problem

    Existing benchmarks assume complete, relevant inputs, while practical modeling may require selecting relevant information, deriving parameters, and predicting unavailable values from data.

    Source: §1 · pp. 1–2

  • Method

    NLData2Opt separates problem descriptions from data and evaluates parameter extraction, derivation, and prediction in AMPL formulation tasks.

    Source: §1 · p. 2, §3.2 · pp. 3–4

  • Results

    Current LLMs and baseline methods achieve only limited success on NLData2Opt.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The findings underscore the need for more robust approaches to optimization problem formulation.

    Source: §6 · p. 8

    Because NLData2Opt derives from NLP4LP and Mamo, potential pretraining-data contamination cannot be fully ruled out.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Formulating optimization problems from raw inputs of problem descriptions and data is time-consuming and costly. While Large Language Models (LLMs) offer a promising way to automate this process, existing benchmarks overestimate their capabilities by relying on idealized inputs where all data is relevant and complete. This diverges from reality, where data-centric reasoning is required to extract, compute, and predict parameters from noisy sources. To address this, we introduce NLData2Opt, a benchmark to evaluate LLMs on the multi-stage reasoning inherent in practical optimization workflows. NLData2Opt decomposes problems into problem descriptions and data, and requires models to (1) extract relevant parameters from data containing misleading attributes, (2) compute missing parameters derived by transforming existing data attributes, and (3) predict unavailable future values from historical observations. Evaluation results show substantial performance decline relative to existing benchmarks, indicating that benchmark progress does not directly translate to formulation accuracy in practical optimization workflows. NLData2Opt is available at https://github.com/sumarivyn/NLData2Opt.

1 Introduction

Existing optimization-modeling benchmarks largely assume complete, relevant inputs and often place data within problem descriptions. NLData2Opt separates descriptions from data and tests parameter extraction, derivation, and prediction; current LLMs achieve limited success on these tasks.

1 Introduction

  • Existing benchmarks often embed data in problem descriptions and assume all required parameters are ready to use.Real-world modeling separates problem construction from data collection, and noisy, incomplete data may require analysts to identify relevant information, derive values, or forecast unavailable ones.Source: §1 · pp. 1–2
  • NLData2Opt separates natural-language problem descriptions from CSV-style data tables and requires models to extract, compute, and predict parameters.The tasks include misleading attributes, parameters derived from existing data, and future values estimated from historical observations.Source: §1 · p. 2
  • Current LLMs perform competitively on simplified tasks but achieve limited success on NLData2Opt’s multi-stage tasks.The benchmark evaluates extraction of relevant parameters, computation of derived parameters, and prediction of unavailable future values.Source: §1 · p. 2
  • The benchmark transforms NLP4LP and Mamo tasks while preserving the feasible region and optimal objective value.Its evaluations of prompting, agent-based, and fine-tuned baselines reveal gaps between structural correctness and functional executability.Source: §1 · p. 2

2 Related Work

Prior work explores LLMs for optimization formulation through prompting, multi-agent systems, training, and output-format techniques. Other approaches target formulation quality and more complex problem domains.

2 Related Work

  • Recent work explores automating formulation to reduce the time and expertise required for optimization modeling.Applications described include task scheduling, vehicle routing, and healthcare planning.Source: §2 · p. 3
  • Training-free methods use step-by-step instructions and prior knowledge in prompts, while learning-based methods apply synthetic-data SFT, reinforcement learning, or contrastive warm-up.Multi-agent systems and chain-of-thought variants are also explored.Source: §2 · p. 3
  • Formulation-quality techniques include entity tagging, five-element formulations, and inference-time format control.LLMOPT reports an 11.08% average improvement in solving accuracy over SOTA methods; NED-Tree and LEAN-LLM-OPT address complex nonlinear and large-scale industrial formulations.Source: §2 · p. 3

3 NLData2Opt

NLData2Opt evaluates optimization modeling under realistic data conditions, using separated problem descriptions and data, parameter transformations, and a validity criterion that preserves the original problem’s solution properties.

3.1 Mixed Integer Linear Program

  • The benchmark focuses on Linear Programs and Mixed Integer Linear Programs.It targets optimization formulations with broad industrial applications.Source: §3.1 · p. 3

3.2 Task Definition in NLData2Opt

  • An LLM receives a problem description and data tables, then produces executable AMPL model and data files.Relevant parameters are separated from irrelevant attributes.Source: §3.2 · p. 3
  • Four cases expand into nine prompt settings: NL, NL+Data, Derivation, and Prediction, which computes implicit parameters or forecasts masked values from at least seven historical observations.In the NL case, numerical parameters are embedded in the problem text; in NL+Data, the description and dataset are provided separately.Source: §3.2 · pp. 3–4

3.3 Validity Criterion

  • Base, Prediction, and Derivation transformations must preserve the original feasible region and optimal solution.The criterion is intended to keep the ground-truth solution invariant under transformations.Source: §3.3 · p. 4

4 Automatic Construction of NLData2Opt

NLData2Opt is constructed to reflect workflows in which problem descriptions and data are separate, and cases require prediction or derivation. The paper generates AMPL gold solutions, evaluates them with complementary metrics, and validates them through execution and semantic checks.

4.1 Separation of Problem Description and Data

  • Each task separates a numerical-value-free problem description from CSV-style data; prediction cases add historical observations and targets, while derivation cases first introduce intermediate quantities.The derivation process revises the natural-language task before separating it into a description and corresponding data tables.Source: §4.1 · p. 4

4.2 Generating Gold Solutions in AMPL

  • AMPL keeps model structure distinct from data values, while a multi-expert framework classifies tasks, constructs formulations, and iteratively refines the resulting code.When a standard template exists, it is passed to the AMPL Formulator; otherwise, a Model Constructor and Data Extractor build the formulation.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3 Evaluation Metrics

  • Evaluation combines entity-level F1 for structural correctness, execution-based comparison with the ground-truth objective, and GPT-4o-mini holistic scoring of constraints and parameter values.The holistic score uses a 0–10 scale.Source: §4.3 · p. 6

4.4 Quality Control

  • Every generated formulation is executed and its objective compared with the ground-truth solution, while LLM reviewers check soundness and whether transformations preserve the feasible region and optimum.The reviewers verify that transformation rules are satisfied for validated problem-description and data pairs.Source: §4.4 · p. 6

5 Experiments

Experiments evaluate NLData2Opt using LLMs and structured baselines, finding limited success across its task cases. The analyses identify challenges in separating model and data, handling irrelevant attributes, deriving and predicting parameters, and judging formulations against human ratings.

5.1 Experimental Setup

  • NLData2Opt is constructed from NLP4LP and Mamo tasks and evaluated with prompting, structured, and fine-tuned baselines using AMPL execution.Generated formulations were normalized to AMPL format, run with CBC, and non-executable outputs received zero EBC.Source: §5.1 · pp. 6–7

5.2 Results and Analysis

  • Current LLMs and baselines achieve limited success on NLData2Opt; the strongest models reach an EBC score of 0.77 across the nine task cases.Figure 5 ranks models from worst to best and encodes EBC by heatmap saturation.Source: §5.2 · p. 7, Figure 5 · p. 7
  • Moving from NL to NL+Data generally lowers EBC while EF1 remains relatively stable, although proprietary models score higher on Mamo Complex with NL+Data.The authors suggest that separate tables may provide more clarity for Mamo Complex's more complicated tasks; qualitative analysis also finds parameterized models more scalable and generalizable despite producing the same solution.Source: §5.2 · p. 7
  • Semantically similar irrelevant attributes reduce performance across three metrics, while derivation and prediction cases expose difficulties integrating intermediate values into valid constraints.For derivation, undefined-entity errors rise from 36.3% to 47.6%; oracle-prediction baselines show forecasting errors alone do not explain the performance gap.Source: §5.2 · p. 8

5.3 Human Alignment

  • LLM-ES aligns positively but moderately with human judgments on 108 Claude Opus 4.6 formulations, so the authors recommend interpreting it cautiously alongside execution-based metrics.Reported correlations are Spearman ρ = 0.39 and Pearson r = 0.30.Source: §5.3 · p. 8

6 Conclusion

NLData2Opt evaluates optimization modeling under realistic data conditions, including separated problem descriptions and data, misleading attributes, parameter derivation, and future-value prediction. Evaluations find that current LLMs and baselines achieve only limited success.

6 Conclusion

  • NLData2Opt evaluates optimization modeling under realistic data conditions by separating data from problem descriptions and including misleading attributes.It also requires deriving missing parameters and predicting future values, tasks common in applied optimization but largely absent from prior benchmarks.Source: §6 · p. 8
  • The benchmark tests parameter derivation and future-value prediction in addition to extracting relevant information from data.These requirements reflect practical workflows that prior benchmarks largely omit.Source: §6 · p. 8
  • Current LLMs and baseline methods achieve only limited success on NLData2Opt.Source: §6 · p. 8

Limitations

The benchmark's construction and evaluation have limitations involving automated validation, possible pretraining contamination, the studied optimization setting, and the relationship between formulation and conceptual reasoning.

Limitations

  • LLM-based benchmark transformations may introduce hallucinations, temporal inconsistencies, logical distortions, or synthetic parameters unlike real-world distributions, and automated checks may miss some errors.The study uses execution-based checks and LLM validation against consistency criteria, but these cannot guarantee detection of all semantic or logical errors.Source: Limitations · p. 9
  • Because NLData2Opt derives from existing datasets, some source instances may have appeared in evaluated models' pretraining data, so contamination cannot be ruled out.The tasks are substantially transformed, but this does not eliminate the possibility.Source: Limitations · p. 9
  • Conclusions about industrial realism and general optimization are limited to the LP/MILP tasks, data perturbations, and AMPL formulation setting studied.They should not be read as directly representing the full complexity of industrial optimization problems.Source: Limitations · p. 9
  • The workflow may emphasize formalization ability over deeper conceptual reasoning about how predicted or derived parameters affect optimization outcomes.A model may construct an AMPL formulation using a predicted parameter without fully understanding its effect on the feasible region or optimal solution.Source: Limitations · p. 9

Ethical considerations

This section describes NLData2Opt’s data and construction practices, including its licensing and exclusion of personal or sensitive data. It also outlines pipelines and prompt-template requirements for creating prediction and derivation cases, verifying transformations, and separating problem descriptions from numeric parameter data.

Ethical considerations

  • NLData2Opt adapts NLP4LP and Mamo consistently with their CC BY NC 4.0 license, excludes personal or sensitive data, and is planned for non-commercial research release under that license.The authors state that they respect the original creators’ ownership and intellectual property; AI assistants supported prompt-template development and paper clarity, while the authors retain responsibility for the content, experimental design, and conclusions.Source: Ethical considerations · p. 9

Appendix

  • GPT-4o-mini and GPT-4o generated benchmark cases using temperature 0.1, with GPT-4o handling derivative and more challenging tasks.GPT-4o-mini primarily constructed NL+Data and prediction cases for NLP4LP; the models were identified as gpt-4o-mini-2024-07-18 and gpt-4o-2024-11-20.Source: Appendix · p. 11

I Templates for AMPL formulation construction

  • A CoE pipeline constructs executable AMPL gold standards through expert roles for formulation, data extraction, execution-error diagnosis, review, and code revision.The initial formulator identifies the model and objective while treating numerical inputs as generic parameters; subsequent experts revise formulations iteratively after execution.Source: I Templates for AMPL formulation… · pp. 11–12

II Templates for model–data separation

  • A separator converts each optimization problem into a numerical-value-free natural-language statement and a CSV dataset containing explicitly provided values, which a verifier checks for completeness.The verifier checks that all numerical values required to solve the problem are present in the data file.Source: II Templates for model–data separation · p. 13

III Templates for prediction case construction

  • Prediction cases are constructed by transforming NL+Data cases into tasks requiring models to predict future parameter values from historical observations.The supplied passage ends mid-sentence after “his-”; the full description is not available here.Source: III Templates for prediction case… · p. 14
  • Prediction cases blank verified target values in historical data, support single or multiple targets and increasing or stable trends, and specify targets and time periods in natural-language descriptions.The pipeline adds seasonality, abrupt changes, intermittent missing data, and outliers; separate verifiers check single- and multiple-variable cases.Source: III Templates for prediction case… · p. 14

IV Templates for derivation case construction

  • Derivation cases decompose exactly one numerical quantity into meaningful intermediate components while preserving the problem’s mathematical equivalence.A two-stage context-generation process retries cases that fail verification; the verifier checks preservation of the feasible region and optimal solution. Rewriting only in algebraic form or merely rephrasing the problem does not satisfy the decomposition requirement.Source: IV Templates for derivation case… · pp. 16–17
  • For derivation cases, the NL-Data Separator includes only intermediate quantities when both aggregated parameters and intermediate quantities are available, requiring derivation during modeling.It separates the optimization model from its associated data.Source: IV Templates for derivation case… · p. 16
  • The verifier accepts a rewrite only if it preserves the feasible region and optimal solution and splits at least one quantity into meaningful components.Source: IV Templates for derivation case… · p. 17
  • The separator prompt asks for a natural-language problem statement that omits numbers, units, placeholders, and constraints already encoded in the tables.Source: IV Templates for derivation case… · p. 17
  • The separator prompt requires numeric CSV-style tables covering all referenced variables and containing only values explicitly stated in the problem.When aggregated and intermediate parameters are both available, the prompt says to include only the intermediate parameters.Source: IV Templates for derivation case… · p. 17

V Templates for LLM evaluation

The evaluation templates extract generated AMPL formulations into compatible files and assess whether the model and data match the ground truth. A separate holistic score rates overall correctness.

V Templates for LLM evaluation

  • Generated outputs are reformatted to extract AMPL formulations and ensure compatibility with the CBC solver.The template handles conversational headers and missing .mod/.dat markers; it returns an empty model marker when no tangible formulation is present.Source: V Templates for LLM evaluation · p. 18
  • Holistic evaluation adds an LLM-based 0–10 rating normalized to a score between 0 and 10, alongside EF1 and EBC.The rating is intended to be consistent with the other two metrics.Source: V Templates for LLM evaluation · p. 18
  • The overall-correctness template evaluates whether candidate constraints are equivalent to ground truth and parameter values exactly match.It compares both .mod files and .dat files.Source: V Templates for LLM evaluation · p. 18

VI Controlled Prediction Baselines

The study separates forecasting from formulation errors using simple-trend and oracle-prediction baselines. Results indicate that forecasting errors alone do not explain the performance gap.

VI Controlled Prediction Baselines

  • The baseline comparison isolates forecasting difficulty from formulation difficulty across three representative models.Simple forecasting uses linear trend extrapolation, while oracle prediction removes forecasting error by supplying ground-truth future values.Source: VI Controlled Prediction Baselines · p. 18
  • Replacing LLM predictions with simple trend extrapolation generally reduces EBC, while oracle predictions improve it only slightly.The simple baseline extrapolates a linear trend from historical data; the oracle baseline substitutes ground-truth future values, leaving the formulation process unchanged.Source: VI Controlled Prediction Baselines · p. 18
  • Oracle future values yield only small improvements, indicating persistent formulation errors even when forecasting difficulty is removed.For Llama 3.3 70B Instruct on One-Variable Prediction, EBC rises from 0.1824 to 0.1847 under oracle prediction; lower-capacity models still struggle to incorporate predicted values and formulate correctly.Source: VI Controlled Prediction Baselines · p. 18

VII Further Analysis

Further analyses examine model identification, prediction accuracy, and AMPL execution errors. They find executable formulations can still contain semantic errors, while structural consistency remains challenging.

VII Further Analysis

  • For the derivation case, a correct model requires exact agreement with ground truth on constraint relation types and objective function type, regardless of parameter values.The criterion checks ≤, ≥, or = constraints and maximize or minimize objectives.Source: VII Further Analysis · pp. 18–19
  • Over 30% of methods achieve exact prediction correctness (EC), and over 60% achieve relative correctness (RC).These metrics evaluate future-value prediction before optimization.Source: VII Further Analysis · p. 19
  • Figure 6 distributes AMPL execution-error types across nine task cases, while Table 6 groups failures into tool-specific syntax and structural errors.Syntax errors include formatting violations and dimension mismatches.Source: VII Further Analysis · p. 19
  • Structural consistency between model and data is a significant challenge, with undefined entities and undeclared data assignments among prevalent errors.Undefined entities are used without declaration or initialization; undeclared assignments place data values on parameters or sets absent from the model.Source: VII Further Analysis · p. 19
  • Exact objective match rates are consistently below compilation success rates, showing that executable formulations can still encode semantic errors and yield incorrect solutions.The discrepancy is reported in Table 6.Source: VII Further Analysis · p. 19

VIII Comprehensive Evaluation Results

Evaluation uses a random 15% benchmark subset and checks selected models on the full benchmark; full-benchmark results preserve model rankings and task-difficulty patterns.

VIII Comprehensive Evaluation Results

  • Tables 7–15 use a random 15% subset of the benchmark, sampled with seed 42, and report entity F1 score, execution-based correctness, and LLM-based evaluation score.LLM-based evaluation score is averaged over four runs with prompt variations, with mean and standard deviation reported.Source: VIII Comprehensive Evaluation Results · p. 19
  • On the full benchmark, Claude Opus 4.6 outperforms GPT-4o Mini and Llama 3.3 70B Instruct, while derivation, prediction, and irrelevant-attribute cases remain more challenging than NL and NL+Data.The full-benchmark results show trends consistent with the subset evaluation.Source: VIII Comprehensive Evaluation Results · p. 20, Table 16 · p. 26, Table 17 · p. 26
  • The evaluation also reports failure-type breakdowns across all task cases and AMPL execution-error distributions across nine cases, including compilation and logical errors.Examples include undefined entities, undeclared data assignments, and syntax errors.Source: Table 6 · p. 21, Figure 6 · p. 22
Loading 2610.22889v1…