Source-linked AI summary
Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization
Sihan Ge, Yichen Lin, Chenyu Zhou, Jianghao Lin, Tao Yao, Dongdong Ge
TL;DR
LLM-based optimization assistance often lacks evidence that a request is complete enough to model, because missing business facts can change the formulation. The paper introduces OR-Clarify and InterOPT to benchmark and guide clarification before modeling, finding strongest gains in choice-based clarification and competitive performance in free-form interaction. These results position OR assistance as a selective decision about when to clarify and when to stop.
Problem
Real-world OR requests can omit formulation-critical facts, while existing evaluations largely assume complete specifications and overlook whether agents know when clarification is needed before modeling.
Method
OR-Clarify benchmarks bounded clarification with hidden slots and simulated users, while InterOPT uses Dynamic Gap Search and Gap-Guided Action Search to decide what to ask or when to stop.
Results
InterOPT is most effective in the Choice setting, improving exact recovery over training-free baselines, while remaining competitive but not uniformly dominant in open/free-form clarification.
Takeaways & Limitations
OR agents should be evaluated before modeling on requirement recovery, readiness, interaction cost, and silent assumptions, not only on the final model.
Takeaways & Limitations
InterOPT can require more interaction, and stopping remains imperfect while silent assumptions are not eliminated.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to formulate optimization models from natural-language problem descriptions, yet realistic operations research (OR) requests are often incomplete: missing objectives, constraints, or business rules can change the resulting mathematical program. Existing evaluations largely assume a complete specification and therefore overlook whether an agent knows when clarification is needed before modeling. We introduce OR-Clarify, a benchmark for pre-formulation clarification. Each task presents a partial public problem description, withholds structured hidden slots, and evaluates agents through bounded interaction with a simulated user. The benchmark supports both openended and choice-based clarification, and measures slot recovery, stopping behavior, silent assumptions, and interaction cost. We further propose Interactive Optimization (InterOPT), a two-stage framework that identifies unresolved formulation-critical gaps and uses them to guide whether to ask the next question or to stop. In our choice-based experiments, InterOPT substantially outperforms all baselines in exact slot recovery; in the open-ended setting, it remains competitive with strong prior methods. Together, OR-Clarify and InterOPT reframe OR assistance as a selective completeness decision: clarify when needed, stop when ready, and quantify what remains missing.
1. Introduction
Real-world OR requests often omit formulation-critical business facts, making premature modeling a central failure mode. OR-Clarify benchmarks this pre-formulation clarification task, while InterOPT tracks unresolved gaps and guides questioning or stopping.
- Motivation: Missing objectives, constraints, or business rules can fundamentally change an optimization model’s mathematical structure.Examples include unspecified objectives and unclear hard-versus-soft time windows.
- Motivation: Strong LLM agents may declare readiness while core facts remain unresolved or silently assume unsupported defaults.A silently assumed closed vehicle-routing tour changes the constraint structure.
- InterOPT: InterOPT separates Dynamic Gap Search from Gap-Guided Action Search to track unresolved gaps and decide what to ask or when to stop.The framework targets specification recovery while avoiding unnecessary interaction.
- OR-Clarify: OR-Clarify converts fully specified optimization tasks into controlled clarification instances with hidden formulation-critical facts, simulated-user responses, and slot-level evaluation.It supports both free-form and choice-based interaction protocols.
- Evaluation: The benchmark evaluates requirement recovery, readiness decisions, silent assumptions, and interaction cost before model formulation.The empirical study compares InterOPT with strong LLM-based baselines in free-form and choice settings.
2. Related Work
Prior work studies optimization modeling, interactive problem solving, clarification, elicitation, and abstention, but OR-Clarify targets formulation-critical requirement recovery before an optimization model is built. Its readiness decision combines uncertainty detection, assumption localization, and interaction control.
- Optimization modeling: Optimization benchmarks and systems mainly evaluate translating complete natural-language specifications into models, solver-ready code, or structured modeling pipelines.Examples include NL4Opt, OptiMUS, LLMOPT, ORLM, and ORPilot.
- Clarification and elicitation: Clarification and preference-elicitation research studies underspecified intent, informative questioning, user-selectable interpretations, and multi-turn decision improvement.These lines of work establish both free-form and option-based clarification settings.
- Interactive benchmarks: Interactive benchmarks increasingly evaluate whether dialogue reaches a correct formal, environmental, or database state.Examples include τ-bench and CLARITY for tool-use and schema-level ambiguity.
- OR-Clarify: OR-Clarify differs by making formulation-critical business requirements private targets and evaluating their recovery, silent assumptions, and readiness before modeling.This distinguishes it from benchmarks focused on search, database state, or schema ambiguity.
- Abstention: Unlike generic abstention, this task asks whether information is sufficient to build the correct model and requires identifying, obtaining, and resolving formulation-changing facts.Readiness therefore combines uncertainty detection, assumption localization, and interaction control.
3. Problem Formulation
Pre-formulation clarification asks whether a business request is sufficiently specified for a meaningful optimization formulation and recovers unresolved formulation-critical information. The interaction continues through targeted questions until the agent judges the specification sufficient or reaches the turn limit.
- Task definition: The task jointly assesses whether a public business request is model-ready and recovers missing information before formulation begins.A successful policy avoids both unnecessary interaction and premature readiness.
- Public state: The public problem statement is induced by the initial business brief and the preceding public interaction transcript.The transcript contains agent clarification actions and user responses.
- Completeness: A problem is formulation-complete when all plausible business completions consistent with the public statement induce the same formulation structure.The formulation structure includes objectives, constraints, decision variables, and other formulation-level structures.
- Completeness: A problem is formulation-incomplete when plausible completions can induce more than one formulation structure.The incompleteness concerns unresolved business conditions whose alternative resolutions change the optimization formulation.
- Interaction: At each turn, the agent asks a clarification question or judges the current information sufficient, terminating at READY_TO_MODEL or the maximum turn limit.An asked question updates the public transcript and induces the next public problem statement.
4. OR-Clarify Benchmark and Evaluation Framework
OR-Clarify converts incomplete OR tasks into controlled clarification cases with hidden formulation-critical slots, bounded interaction, and transcript-based evaluation. It measures whether agents recover required facts, stop appropriately, avoid silent assumptions, and limit interaction burden.
- Benchmark representation: OR-Clarify represents each case with a public brief, source-grounded facts, and hidden slots containing requirements absent from the brief.Each hidden slot links to supporting evidence, bounded user answers, acceptable questions, recovery rules, and a severity label.
- Benchmark construction: Eligible facts about objectives, constraints, and assumptions are masked after screening, while visible facts remain in a self-contained public brief.Masking excludes facts whose mathematical meaning is already determined by visible information and uses a fixed, case-specific pseudorandom procedure for roughly half of eligible facts.
- Benchmark composition: OR-Clarify contains 100 clarification cases and 178 hidden slots: 75 P0, 83 P1, and 20 P2.Cases contain 1–5 slots, with a mean of 1.78, and human auditing verifies slot boundaries, severity labels, answer support, and rubric consistency.
- Controlled interaction: The benchmark enforces an information boundary in which agents see only the public brief and transcript, while simulated users answer only the current question from private facts.A protocol detector separately checks interaction format, and a post-hoc judge scores the public transcript against frozen hidden-slot annotations.
- Interaction settings: Open/free-form and Choice settings evaluate natural-language questions versus questions with candidate options.Choice variants can use a corrective option when none of the listed choices is supported by the private facts.
- Evaluation metrics: Core Exact succeeds only when all P0/P1 slots are exactly recovered, while additional diagnostics measure stopping behavior, silent assumptions, and interaction burden.Reported burden includes Avg Turns and Avg Q; Avg Q counts atomic clarification questions and can exceed Avg Turns.
5. Method
InterOPT separates diagnosing formulation-critical gaps from deciding whether to ask or stop. Its two-stage loop maintains public-evidence gap memory, anchors questions to open gaps, and preserves model-controlled stopping.
- Framework: InterOPT separates formulation-gap diagnosis from interaction control through Dynamic Gap Search and Gap-Guided Action Search.The first stage identifies and tracks unresolved gaps; the second uses them to guide clarification actions.
- Stage 1: Dynamic Gap Search: Dynamic Gap Search maintains a persistent ledger of unresolved business conditions whose values could change the optimization formulation.The ledger is derived from the public brief and dialogue history rather than benchmark-private facts.
- Stopping: The stopping decision remains with the model: it may ask or emit READY_TO_MODEL, even when the ledger is nonempty or empty.The ledger provides memory and question–gap alignment but does not guarantee complete discovery or safe stopping.
- Stage 1: Dynamic Gap Search: The gap memory searches for missing objectives, decision scope, constraints, time boundaries, entity relationships, and hard-versus-soft policies.This stage records gaps but neither ranks them nor asks the user a question.
- Stage 2: Gap-Guided Action Search: Gap-Guided Action Search generates three candidate clarification states and questions, binding each candidate to an open gap when one exists.A selector compares the candidates against the public history before choosing the next action.
- Stage 2: Gap-Guided Action Search: The selector prioritizes possible changes to objectives, constraints and rules, decision scope, and silent-assumption risk over mathematical detail.Only after the selected question enters the public transcript does its gap move from Open to Asked.
6. Experiments and Results
The experiments compare controlled clarification protocols under a fixed evaluation harness across 100 OR-Clarify cases. They measure recovery, stopping behavior, silent assumptions, and interaction burden in both Choice and open/free-form settings.
- Evaluation Setup: All main experiments use 100 OR-Clarify cases with five independent runs per case and aggregate repeated runs at the case level.Headline denominators retain runs that fail the post-hoc protocol audit.
- Evaluation Setup: The evaluation reports All-Slot Exact, Core Exact, Silent/run, Avg Turns, and Avg Q to assess recovery, assumptions, and interaction cost.Avg Q counts atomic clarification questions and can exceed Avg Turns when one turn contains multiple questions.
- Choice-Based Setting: The Choice comparison includes MC, MC-D, ReadyGate, and InterOPT under five runs per case.MC uses three generated options, MC-D adds a free-form correction option, and ReadyGate adds an independent stopping reviewer.
- Choice-Based Setting: InterOPT combines persistent gap memory, gap-anchored candidate questions, and agent-controlled ask-or-stop decisions in the Choice protocol.The tested agent controls planning decisions, while the simulated user, protocol detector, and post-hoc judge remain fixed evaluation components.
- Open/Free-Form Setting: The open/free-form comparison uses a shared FreeQA interface with natural-language questions or READY_TO_MODEL and a maximum of 20 turns.Adapters receive the same public brief, dialogue history, tested model, simulated user, and post-hoc judge.
6.3. Performance across Tested Models
Baseline performance varies across tested models and interaction protocols, while OR-Clarify remains difficult: no model exceeds 60% Core Exact and all retain substantial silent assumptions.
- Model Comparison: Opus-4.8 is strongest in both Open/FreeQA and Choice/MC-D settings, but no tested model exceeds 60% Core Exact.The comparison covers off-the-shelf LLMs across both protocols.
- Model Comparison: All tested models retain substantial silent assumptions, confirming that OR-Clarify remains challenging.The result is reported across the baseline model comparison rather than for a single model only.
6.4. Method Effectiveness and Interaction Cost
InterOPT performs best in the Choice setting but remains competitive rather than dominant in Open, revealing a recovery–interaction-cost trade-off.
- In Open, ORPilot has the strongest recovery, GATE remains competitive, and InterOPT is close but not best.
- In Choice, InterOPT achieves the highest recovery among evaluated Choice methods while using substantially more turns and atomic questions.
- Figure 4 compares exact restoration against cumulative atomic-question budgets separately for Open and Choice, using All-Slot Exact and Core Exact.Curves average K=5 runs with 95% case-level bootstrap confidence intervals, and comparisons stay within protocols.
- Structured gap-guided clarification improves recovery most under constrained answer spaces, but stopping and question efficiency remain limiting factors.
- Under Choice, InterOPT continues gaining exact recovery after MC-D and ReadyGate largely plateau, so endpoint gains should be read alongside budget-matched curves.
6.5. Contributions of the Two Stages
The ablations assign complementary roles to InterOPT’s stages: gap discovery primarily supports coverage, while guided action selection increasingly matters in Choice.
- The ablations disable gap search or collapse guided action selection while keeping the same cases, repetitions, temperatures, simulator, and judge as full InterOPT.
- Removing Stage 1 yields 0.460 Core Exact in Open, while removing Stage 2 yields 0.506 but nearly doubles questions versus full InterOPT, 9.5 vs. 4.8.
- Under Open, Stage 1 contributes more to recovery, whereas Stage 2 mainly reduces interaction length.
- Under Choice, removing Stage 1 causes the larger Core Exact reduction, 11.1 vs. 6.0 percentage points.
6.6. Protocol-Specific Diagnostics
Diagnostics show that InterOPT reduces silent assumptions, while protocol-specific weaknesses differ: Choice exposes non-causal event differences, and Open is dominated by premature stopping.
- Choice diagnostics record 132 no-match events among 1,129 audited events, but these rates are not causal estimates because agents follow different question trajectories.
- InterOPT leaves 0.366 silent assumptions per run, compared with 0.692 for MC-D and 0.614 for ReadyGate.Across 890 judged slots per K=5 method, InterOPT receives 617 yes, 20 partial, and 253 no labels, versus MC-D’s 461, 16, and 413.
- InterOPT declares readiness in 98.8% of Open runs, yet 203 of 500 runs, or 40.6%, are flagged as premature, leaving 0.560 silent assumptions.
- Open-setting difficulty lies primarily in recognizing when questioning is sufficient, not in asking questions or receiving unusually informative multi-slot answers.Multi-slot disclosures occur only 0.116 times per run.
7. Conclusion
The paper introduces OR-Clarify and InterOPT to evaluate and guide pre-formulation clarification, with strongest gains in Choice but persistent interaction and stopping limitations.
- OR-Clarify benchmarks recovery of formulation-critical hidden requirements before modeling while auditing interaction cost, stopping behavior, and silent assumptions.
- InterOPT maintains formulation gaps to guide questioning and stopping through a two-stage framework.
- In Choice, InterOPT improves exact recovery over training-free baselines, while in Open it remains competitive without uniformly dominating strong baselines.
- Ablations indicate that both gap tracking and guided action selection contribute to recovery.
- InterOPT can require more interaction, stopping remains imperfect, and silent assumptions are not eliminated.