SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science

Nithin Somasekharan; Youssef Hassan; Shiyao Lin; Gihan Panapitiya; Patrick Emami; Anurag Acharya; Sameera Horawalavithana; Shaowu Pan

Source-linked AI summary

SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science

Nithin Somasekharan, Youssef Hassan, Shiyao Lin, Gihan Panapitiya, Patrick Emami, Anurag Acharya, Sameera Horawalavithana, Shaowu Pan

arXiv:2605.18630v1May 18, 2026cs.AIphysics.comp-ph

TL;DR

Existing scientific benchmarks usually assume well-posed tasks, leaving unclear whether models can clarify incomplete or contradictory computational-science requests before computation. SCICONVBENCH evaluates this through multi-turn dialogue across four domains, finding that no model dominates both regimes and that disambiguation remains harder than inconsistency resolution.

Problem
Existing benchmarks rarely test whether models can identify and resolve missing or conflicting scientific requirements before producing a well-specified computational task.
Method
SCICONVBENCH uses a structured scientific-task ontology and rubric-based evaluation to assess multi-turn clarification across four computational-science domains.
Results
No single model dominates both clarification regimes: GPT-5.2 leads disambiguation at 52.7% CGRR, while GEMINI 2.5 PRO leads inconsistency resolution at 82.7%.
Takeaways & Limitations
The benchmark shows that scientific clarification comprises distinct dialogue skills and that models often finalize specifications without grounding repairs or assumptions in the conversation.
Takeaways & Limitations
SCICONVBENCH covers only four domains and English-language, text-only prompts at undergraduate-to-early-graduate difficulty, so its absolute results should not be extrapolated broadly.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly deployed as scientific AI as- sistants, and a growing body of benchmarks evaluates their capabilities across knowledge retrieval, reasoning, code generation, and tool use. These evaluations, however, typically assume the scientific problem is already well-posed, whereas practical scientific assistance often begins with an ill-posed user request that must be refined through dialogue before any computation, analysis, or experiment can be carried out reliably. We introduce SCICONVBENCH, a benchmark for multi- turn clarification in scientific task formulation across four computational science problem domains: fluid mechanics, solid mechanics, materials science, and par- tial differential equations (PDEs). SCICONVBENCH targets two complementary capabilities: eliciting missing information (disambiguation) and detecting and correcting erroneous requests containing internally contradictory information (in- consistency resolution). Our benchmark pairs a structured task ontology with a rubric-based evaluation framework, enabling systematic measurement of LLM per- formance across three dimensions: clarification behavior, conversational grounding, and final-specification fidelity. Current frontier models perform relatively well on inconsistency resolution, but even the best model resolves only 52.7% of the disambiguation cases in fluid mechanics. We further find that frontier LLMs fre- quently make silent assumptions and perform implicit specification repairs that are not grounded in the conversation with users. SCICONVBENCH establishes a foundation for evaluating the upstream conversational reasoning that a reliable computational science assistant requires. The code and data can be found at https://github.com/csml-rpi/SciConvBench.

1 Introduction

SCICONVBENCH evaluates whether LLMs can clarify incomplete or contradictory computational-science requests before computation, coding, or tool use. It introduces a multi-turn benchmark and evaluation framework designed to expose failures that standard end-state metrics miss.

Motivation: Most scientific LLM benchmarks assume complete task formulations, overlooking the need to transform incomplete or internally inconsistent requests into well-specified scientific tasks.Such formulation errors can undermine reliable computation, code generation, and tool invocation.
Motivation: Existing clarification benchmarks do not capture the domain-specific demands of computational science, where information distributed across dialogue remains difficult for current models.The paper identifies both conceptual and quantitative gaps relative to general-domain clarification settings.
Benchmark design: SCICONVBENCH covers fluid mechanics, solid mechanics, materials science, and PDEs, with instances requiring either disambiguation of missing information or resolution of conflicting information.Models interact with users over multiple turns before producing a final clarification-based specification.
Contributions: The benchmark shifts evaluation upstream by asking whether models can define scientific tasks correctly before assessing their ability to solve, code, or execute them.Its framework formalizes conversational scientific task formulation around unresolved ambiguity and inconsistency.
Contributions: Its evaluation separates intent-faithful final resolution from conversation-grounded resolution, revealing silent assumptions and repairs that standard end-state metrics can miss.The paper also benchmarks models across domains and ontology categories while testing robustness across judges, prompts, and user simulators.

2 Related work

Prior work establishes clarification and multi-turn interaction as measurable capabilities, while scientific benchmarks largely evaluate already-specified tasks. Existing clarification benchmarks mainly address ambiguity in meanings, search subtopics, factoid readings, or user preferences rather than scientific regimes.

Clarification and ambiguity: Clarification benchmarks evaluate question selection, ranking, generation, and mining across ambiguous-query and conversational retrieval tasks.Examples include QULAC, CLARIQ, CLARQ, AMBIGQA, and CAMBIGNQ.
Clarification and ambiguity: Their ambiguities concern polysemous meanings, search subtopics, valid factoid readings, or user preferences rather than scientific regimes.These benchmarks establish clarification as measurable, but their ambiguity types differ from scientific task formulation.
Multi-turn, agentic, and simulator-based evaluation: Multi-turn evaluation has expanded from general dialogue quality toward interaction robustness, including automatic judging, human-aligned ranking, verbosity control, and evidence-grounding brittleness.The cited benchmarks include MT-BENCH, CHATBOT ARENA, ARENA-HARD-AUTO, AlpacaEval, MT-EVAL, MULTICHALLENGE, LLMS GET LOST, and RMTBENCH.
Scientific benchmarks and domain-specific agents: Scientific benchmarks evaluate reasoning, research tasks, coding, materials-science tool use, data-driven discovery, tool-augmented reasoning, and chemistry agents, generally assuming the task is already specified.Examples include SCIBENCH, SCIEVAL, SCICODE, MATTOOLS, SCIENCEAGENTBENCH, SCIAGENT, and CHEMCROW.

3 SciConvBench

SCICONVBENCH evaluates multi-turn formulation of computational-science tasks across four domains by perturbing valid reference specifications with missing or conflicting information. It measures final resolution alongside conversation-grounded clarification and diagnostic behavior.

Benchmark scope: The benchmark spans fluid mechanics, solid mechanics, materials science, and partial differential equations, covering general numerical tasks and domain-specific simulator-tool prompts.Each domain represents a different class of scientific task formulation.
Task formulation: Each instance perturbs a clean task specification by marking ontology entries as MISSING or CONFLICT, then evaluates the model’s final specification against the planted issues.The ontology includes objectives, domains, physics or constitutive models, properties, conditions, numerical controls, outputs, and tool settings.
Interaction protocol: Agents ask one clarification question per turn within a fixed budget of 11 turns, while users respond only from each instance’s hidden reference specification.This protocol standardizes interactions across models and domains.
Dataset construction: The benchmark is built from vetted scientific tasks that are manually transformed into disambiguation or inconsistency-resolution conversations and filtered through expert review.Reviewers checked scientific meaningfulness, clear intended resolution, absence of trivial answer leakage, realism, and appropriate difficulty.
Evaluation: Evaluation separates final-output success from conversation-grounded success and scores both case-level completion and component-level resolution, clarification, or silent repair.The framework uses the transcript, final specification, and reference issue annotations because correct wording and dialogue paths can vary.
Evaluation: CGRR is the primary success criterion, while Capability, Robustness, and Usability summarize diagnostic dimensions rather than replacing the main metric.Robustness includes assumption avoidance, error detection, and memory consistency; Usability measures intent capture.

4 Experimental Setup

The experiments evaluate assistants in multi-turn interactions that begin with ambiguous or inconsistent scientific requests and end with a finalized task specification. Evaluation uses rubric-based judgments, supplemented by human validation and ablations of simulator choice and prompt wording.

Conversational evaluation framework: Each instance is a multi-turn interaction in which the assistant may ask one clarification question per turn before producing a final specification.The interaction ends after explicit finalization or when the turn-budget cap is reached.
Judge and human validation: Metrics are computed from saved conversations and final specifications using rubric-based judgments of semantic resolution, conversational grounding, and intent preservation.Headline numbers use GEMINI 2.5 PRO as the judge unless otherwise noted.
Ablations on user simulator and agent prompt: Two ablations vary the user simulator or paraphrase the assistant prompt while keeping the assistant fixed at GEMINI 2.5 PRO.The simulator ablation uses GEMINI 2.5 PRO, GPT-5.2, and CLAUDE SONNET 4.6 on an 80-case stratified subset.

5 Results

SCICONVBENCH results show that conversationally grounded resolution lags final resolution, with disambiguation especially difficult in fluid mechanics and inconsistency resolution following a different domain pattern. Component fragility, model-specific capability profiles, and robustness checks further characterize performance.

Resolution rates: Every model shows a non-zero FRR–CGRR gap, averaging 8.2 percentage points for Disambiguation and 14.7 percentage points for Inconsistency Resolution.The gap indicates that final resolution includes resolutions not grounded in the conversation.
Resolution rates: Fluid mechanics is consistently difficult for disambiguation, while inconsistency resolution is stronger for solids and PDEs and harder for materials.The two task types therefore exhibit different domain-wise difficulty patterns.
Ontology patterns: Numerics, solver choices, and governing-physics assumptions are the most fragile ontology components, with the lowest component-wise FRR.Figure 5 reports component-level FRR(k), CGRR(k), and SRR(k) across the scientific components defined in Equation (1).
Pareto view: GPT-5.2 has the strongest disambiguation profile, whereas GEMINI 2.5 PRO is more robust on inconsistency resolution.Across models, Robustness is the least stable axis, especially when moving from missing-information cases to planted conflicts.
Robustness across judges, prompts, and simulators: Robustness conclusions remain stable across judge choice, guided-prompt paraphrase, and user-simulator model on the 80-case subset.Table 2 summarizes these robustness checks.

6 Discussion

SCICONVBENCH shows that clarification comprises separable skills: models differ between eliciting missing information and resolving contradictions, with performance varying across scientific ontologies. The FRR/CGRR gap further indicates that models often silently infer repairs or defaults rather than grounding them in user clarification.

Model specialization: GPT-5.2 leads disambiguation at 52.7% CGRR, whereas GEMINI 2.5 PRO leads inconsistency resolution at 82.7% CGRR.The results indicate that no single model dominates both clarification regimes.
Ontology-dependent difficulty: Fluid mechanics is the main disambiguation bottleneck, with 29.8% best CGRR versus 68.5% in materials science and 72.1% in PDEs.Numerics and solver choices remain weak, with component-level FRR only around 10% to 21% across models.
Robustness trade-offs: GPT-5.2’s disambiguation robustness is roughly 87% to 94% across domains, but its inconsistency robustness falls to roughly 61% to 69%.The Pareto analysis shows that models may preserve usability and broad user intent while losing robustness.
Silent inference: CLAUDE SONNET 4.6 achieves 31.5% FRR but 0.0% CGRR on PDE inconsistency, indicating final repairs can occur without guided clarification.The FRR/CGRR gap reflects silent scientific inference, including unelicited solver choices, physical regimes, boundary conventions, or materials.

7 Conclusion

SCICONVBENCH evaluates whether scientific LLMs identify missing or inconsistent requirements through dialogue before final task specification. It covers four computational science domains and shows that final resolution rates exceed conversation-grounded resolution rates across domains.

SCICONVBENCH benchmarks conversational scientific task formulation across fluid mechanics, solid mechanics, materials science, and PDEs.
Unlike post-specification benchmarks, SCICONVBENCH tests whether models surface missing or inconsistent scientific requirements through dialogue before committing to a final specification.
Across five guided models, final resolution rate exceeds conversation-grounded resolution rate in every domain.

A Limitations · B Broader impacts · C Prompt Templates

SCICONVBENCH’s scope is limited by its domains, language, modality, task difficulty, dataset size, and lack of human clarification study. The benchmark’s broader purpose is to improve reliability by exposing silent assumptions, while its prompt templates define simulated-user behavior and forced finalization procedures.

A Limitations: The benchmark covers four computational-science domains using English-language, text-only prompts at undergraduate-to-early-graduate difficulty.Its absolute results should not be extrapolated to other domains, modalities, or research-level tasks.
A Limitations: The dataset contains roughly 1,000 cases because scientific task-formulation data are sparse and harder to construct than standard NLP corpora.A human clarification study has not yet been conducted.
B Broader impacts: SCICONVBENCH may help identify silent assumptions before they produce difficult-to-audit or irreproducible computational results.It measures whether models ask clarifying questions before finalizing scientific task specifications.
B Broader impacts: The principal negative impact is potential misuse or overinterpretation of benchmark scores.The benchmark is intended to support more reliable human-AI interaction in scientific settings.
C Prompt Templates: The appendix provides the exact prompt templates for the conversational evaluation framework, including guided and unguided assistant prompts reproduced in Section E.4.The prompts also include a side-by-side comparison of the two conditions on GEMINI 2.5 PRO.
C.1 Simulated User Prompt: The simulated user is instructed to answer strictly from the complete requirement, provide concrete values, flag unspecified details for reasonable assumptions, and remain consistent across clarifications.The simulated user is told to be concise, avoid unexplained reasoning, answer only what is asked, and not invent constraints.
C.1 Simulated User Prompt: The simulated-user query supplies the agent’s question and requests an answer based on the complete requirement.The context template instantiates the incomplete request, hidden complete requirement, and prior clarification history through question-answer exchanges.
C.2 Forced Finalization Prompt: When the turn cap is reached without explicit finalization, the assistant receives a suffix instructing it to stop asking questions and output the final specification from current information.The forced-finalization prompt also prohibits including the “[ COMPLETE]” tag.

D Capability, Robustness, and Usability. … E.4 Guided versus unguided comparison

The paper defines diagnostic measures for capability, robustness, and usability, then analyzes performance across prompt types, scientific domains, and guided versus unguided agents. Results show persistent silent-resolution gaps, stronger tool-use pressure, and larger benefits from guidance for inconsistency resolution than disambiguation.

D Capability, Robustness, and Usability.: Capability is measured by Clarification Recall, Clarification Precision, and Plan Completeness, with Detection Recall additionally restricting recall to planted conflicts.These metrics assess elicited information, question relevance, specification completeness, and conflict detection.
D Capability, Robustness, and Usability.: Robustness tracks Assumption Rate, Error Detection Rate, and Memory Consistency Rate, while usability uses Intent Capture Rate to separate intent drift from other failures.Assumption Rate is better when lower; Memory Consistency Rate equals one when the final specification does not contradict the dialogue.
E.1 General numeric questions versus tool-use prompts: The benchmark separates general numeric prompts from tool-use prompts because they exercise textbook or PDE reasoning versus scientific software invocation.Tool-use prompts presuppose concrete simulation tools or solver setups, whereas the general numeric group does not assume a specific simulation stack.
E.1 General numeric questions versus tool-use prompts: Silent resolution remains a major failure mode in both prompt splits, producing a persistent FRR-CGRR gap whenever information is missing.The gap is not attributable to either textbook-style prompts or tool invocation alone.
E.1 General numeric questions versus tool-use prompts: Tool-use prompts create wider disambiguation FRR-CGRR gaps through hidden solver, mesh, unit, and flag state, while inconsistency resolution remains comparatively easy.Planted conflicts are locally visible in the prompt, unlike many tool-specific assumptions.
E.2 Per-domain breakdown: Per-domain aggregation counts planted issues rather than cases, applies intent-capture gating at case level, and preserves the qualitative model ordering and FRR-CGRR gap pattern.Domains with many issues per case compress visually, while PDE results more closely match case-level numbers.
E.3 Full domain-level results: Tables 3 and 4 provide complete domain-level outcome and diagnostic metrics for disambiguation and inconsistency-resolution cases.Disambiguation covers missing task information; inconsistency resolution covers planted conflicts in the initial request.
E.4 Guided versus unguided comparison: +18pp guided CGRR in fluid mechanics and +11pp in materials science occur for inconsistency resolution, while guidance produces smaller gains in solid mechanics and PDEs.The comparison uses GEMINI 2.5 PRO, the model with fully scored unguided runs across every component.

E.5 Simulator ablation

The simulator ablation varies the simulated-user LLM while holding the assistant, judge, prompt template, and 80-case stratified evaluation subset fixed. Across simulator choices, the FRR–CGRR gap remains substantial, with only modest spread in the headline metrics.

Experimental setup: The ablation compares GEMINI 2.5 PRO, GPT-5.2, and CLAUDE SONNET 4.6 as simulated-user LLMs while fixing the assistant and judge to GEMINI 2.5 PRO.Each simulator uses the same prompt template and 80-case subset: 40 Disambiguation and 40 Inconsistency cases balanced across four domains.
Reported metrics: Table 5 reports FRR, CGRR, SRR, IC, MC, CR, and CP for each simulator, broken out by task.FRR, CGRR, and SRR are headline outcomes; CR and CP denote clarification recall and precision, while IC and MC denote intent capture and memory consistency.
Key findings: The FRR–CGRR gap persists in every table cell, ranging from 22.5pp for Inconsistency with CLAUDE SONNET 4.6 to 45.0pp for Disambiguation with GEMINI 2.5 PRO.The gap is identified as the central benchmark signal.

E.6 Judge ablation

On an 80-case stratified sample, both LLM judges showed substantial agreement with human annotations on headline metrics and near-perfect agreement on continuous clarification metrics. Agreement was weakest for Detect Rate in Inconsistency cases and Intent Capture.

Evaluation setup: The ablation compares Gemini 2.5 Pro and GPT-5.2 judges against human annotations on 80 balanced Disambiguation and Inconsistency cases.The sample is stratified across four domains and four outcome buckets: GROUNDED, SILENT, UNRESOLVED, and INTENT_FAIL.
Headline metrics: κFRR=0.64 (Gemini) / 0.70 (GPT-5.2), while κCGRR=0.47 for both judges, with 71.2% exact-match agreement.These are the two headline metrics used to drive the paper’s main results.
Continuous metrics: For continuous clarification metrics, both judges achieved Spearman ρ ≥0.90 on CR and CP, with MAE ≤0.06.The passage characterizes these correlations as near-perfect agreement with human scores.
Limitations: Agreement was weakest on Detect Rate for Inconsistency cases and on Intent Capture.Table 6 evaluates binary, ordinal, and continuous metrics using metric-appropriate agreement statistics.

E.7 Prompt-sensitivity ablation … F.2 Turn-cap statistics and forced finalization

The prompt-sensitivity ablation finds that scientifically equivalent system-prompt paraphrases preserve substantial grounding gaps while changing headline metrics only modestly. The benchmark also reports inference costs, ontology-level CGRR breakdowns, and turn-cap behavior, including frequent forced finalization by GPT-5.2.

E.7 Prompt-sensitivity ablation: Overall FRR ranges from 72.5–77.5%, while overall CGRR ranges from 42.5–46.2% across prompt paraphrases.The corresponding cross-paraphrase spreads are 5.0pp for FRR and 3.7pp for CGRR.
E.7 Prompt-sensitivity ablation: The FRR–CGRR gap persists in every prompt-paraphrase cell, spanning 20.0pp for Inconsistency, Variant A, to 45.0pp for Disambiguation, Original.The per-task FRR–CGRR gap shifts by at most 5pp between any two paraphrases.
E.7 Prompt-sensitivity ablation: The three scientifically equivalent prompts vary role framing, reasoning-step wording and order, and hard-constraint phrasing while preserving four behavioral contracts.These contracts include one question per turn, no compound questions, the literal [COMPLETE] sentinel, and a plain-sentence final specification.
F LLM API token usage and cost: The entire benchmark run costs approximately $120 in closed-weight agent API charges plus approximately $151 in judge API charges, totaling approximately $271.GPT-OSS-120B is self-hosted on 2×A100 GPUs and incurs no API cost.
F.1 Full ontology breakdown: The benchmark’s full ontology breakdown reports CGRR per ontology component for all five evaluator LLMs across both task types and four computational-science domains.The same values also drive the bar plot in Figure 5.
F.2 Turn-cap statistics and forced finalization: The conversational harness caps each case at Tmax = 11 assistant turns and force-finalizes cases without a final answer by that limit.Table 11 reports turns_needed distributions across the four domains and the percentage of cases hitting the cap.
F.2 Turn-cap statistics and forced finalization: GPT-5.2 force-finalizes 17.6 % of disambiguation cases and 19.4 % of inconsistency cases, whereas Claude Sonnet 4.6 essentially never hits the cap.Median conversation length is p50 ≤4 turns across all models.

F.3 Qualitative case studies · F.4 Human annotation instructions and judge rubric

The qualitative cases show that grounded clarification resolves planted ambiguities and inconsistencies, whereas silent repairs can yield coherent final specifications without conversational grounding. The human-annotation protocol evaluates these behaviors on an 80-case stratified subset using a seven-field rubric and blinded, calibrated scoring.

F.3 Qualitative case studies: Five case studies use the same GEMINI 2.5 PRO guided-mode assistant across failure modes, with human annotations and three independent LLM-judge rescoring sets per case.The cases come from the 80-case stratified human-annotated subset and cover all seven rubric fields.
F.3 Qualitative case studies: Grounded disambiguation explicitly elicits both missing alloy compositions before calculation, preserves intent, and earns FR=CGR=IC=MC= 1 with CR=CP= 1 from all judges and the human annotator.The final specification restates both clarified quantities, making the lever-rule calculation reproducible.
F.3 Qualitative case studies: Silent disambiguation can produce a scientifically correct final specification while failing conversational grounding because the assistant clarifies irrelevant gravity and mesh details instead of planted boundary ambiguities.The assistant silently selects the clamped edges and traction-free inner boundaries.
F.3 Qualitative case studies: Grounded inconsistency resolution surfaces both element-type and load-direction conflicts, obtains the user’s authoritative choices, and reshapes the final specification accordingly.Human and GPT-5.2 judges score FR=CGR=IC= 1, CR=CP= 1, and DR= 1.
F.3 Qualitative case studies: Silent inconsistency resolution discards a transient equation and invents boundary conditions without informing the user about the contradiction between the equation and steady-state description.The human annotator treats the end-state as resolved but not conversationally grounded.
F.3 Qualitative case studies: Inter-judge disagreement arises even after a grounded physics correction: the assistant rejects Bernoulli across a dissipative hydraulic jump and switches to the momentum equation.The human annotator and CLAUDE SONNET 4.6 agree on FR=CGR=IC= 1, CR= 1, and CP= 0.5.
F.4 Human annotation instructions and judge rubric: 80 cases are split evenly into 40 disambiguation and 40 inconsistency-resolution cases across four domains, with ten cases per task-domain cell and targeted outcome buckets.The subset uses one GEMINI 2.5 PRO guided-mode assistant and one expert annotator, with calibration on a separate five-case pool.
F.4 Human annotation instructions and judge rubric: The rubric scores seven fields from the prompt, planted issues, dialogue, and final specification, while blinding annotators to judge scores and calibrating wording before the main pass.Fields include binary FR, CGR, and MC scores plus continuous CR and CP measures; rationales and rater notes are retained for audit.