PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Sensitivity in Tool-Augmented Model Comparisons

Mateusz Modrzejewski, Tomasz Radzikowski

arXiv:2610.31148v1cs.CLcs.AIcs.LG

TL;DR

Reworded benchmark questions can change which model ranks higher, and accuracy gains alone do not show whether tools make those comparisons more stable. The paper uses matched conditions to separate readout, reasoning, and tool-call behavior, finding that tools often widen wording sensitivity despite deterministic reports.

  • Problem

    Question wording can change pairwise model rankings, and accuracy changes under tools cannot establish whether comparisons become more stable because several steps change together.

    Source: §1 · p. 1

  • Method

    A matched protocol compares readouts, reasoning, supplied reports, and tool loops across five wordings, six open models, and five domains.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Tools raise accuracy where programs answer questions but widen wording sensitivity for many model pairs; the widening varies with call frequency, interface, and instructions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Deterministic reports do not guarantee stable comparisons, so evaluations should report call frequency and comparison stability across several wordings alongside accuracy.

    Source: §9 · p. 7

    The five-wording spreads offer little inference to new wordings, and the domains are not a task sample, so prevalence beyond them is unknown.

    Source: §7 · p. 7

Abstract

from arXiv · show

Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.

1 Introduction

Tools can increase accuracy without stabilizing model rankings across question wordings: call behavior may vary with wording even when tool reports are deterministic. The paper introduces matched comparisons of answer readout, reasoning, tool access, and call behavior, then tests how those choices affect wording spread.

1 Introduction

  • A rewording can move the difference that ranks two models as much as changing the model, while tool use adds sensitivity because the model may not make the same call each time.An accuracy gain alone cannot isolate the answer readout, reasoning, call decision, and tool report as sources of wording sensitivity.Source: §1 · p. 1
  • With the same heavy-atom report available, Qwen3-8B answers correctly under all five wordings; when the model must obtain it, one wording leads to no call and a wrong answer.The report itself is unchanged across wordings; the difference is whether the model obtains it.Source: §1 · p. 1
  • The matched protocol compares readout, reasoning, supplied reports, plain-text tool loops, uninformative tools, and native calling under five instruction wordings and each written input form.It also varies interface, instructions, budget, and report reliability; the study applies the protocol to six open models in five domains.Source: §1 · pp. 1–2
  • Tools widen the wording spread of many model pairs against reasoning without tools, even as they raise accuracy where a program answers the question.In molecules, median spread is 0.17 with the plain-text loop versus 0.02 with supplied reports; widening depends on interface and instructions.Source: §1 · p. 2

2 Evaluation Conditions and Measures

  • The wording spread is the range across wordings of the accuracy difference between two models, computed on items present for both models in every wording.The protocol also tracks each model’s own accuracy range across wordings.Source: §2 · p. 2
  • Matched conditions distinguish first-token and answer-line reads, reasoning, reports supplied without a call, and tool loops; the loops append tool reports and end at an answer, six calls, or the token budget.Constant tools return “no information,” while the native interface uses structured tool-calling turns.Source: §2 · p. 2
  • The five domains vary between tools that answer directly and tools that return values the model must combine, while every wording shares the same canonical tool input and gold answer.Molecules and graphs use direct-answer operators; tables and mathematics require combining returned values.Source: Table 1 · p. 3
  • Call-frequency decomposition separates accuracy change into a component associated with call frequency and one associated with accuracy conditional on calling.Call frequency is the share of episodes with at least one call; the observed accuracy among called episodes is not the causal effect of calling.Source: §2 · p. 3

3 Experimental Setup

The setup spans five question-answering domains, six open instruction-tuned models, and matched evaluation conditions. It measures accuracy and wording spread with bootstrap-based comparisons while treating some significance flags as exploratory.

Domains

  • Five domains pair distinct tasks with operators: RDKit molecule and graph tasks, ABC-Eval music, FinQA tables, and MATH-500 mathematics.Molecule and graph gold answers come from the same libraries as their tools; operators answer the question directly in those controlled tasks.Source: §3 · p. 3
  • Domains were selected using the first 200 items: Qwen3-8B or Phi-4 had to score at most 0.90 and their pair spread reach 0.02, except music.GSM8K was excluded because those models scored 0.95 and 0.97; music was exempt from the selection rule.Source: §3 · p. 3

Models and harness

  • Six open instruction-tuned models were tested at fixed revisions, with greedy decoding and specified token budgets across domains.A separate Qwen3 size series and Qwen2.5-72B-Instruct were also tested on molecules, graphs, and music.Source: §3 · p. 3

Analysis

  • Spreads and accuracies use 95% percentile intervals from 1,000 item-bootstrap resamples; condition contrasts use paired resamples and Holm correction.A comparison is flagged wider or narrower when its corrected contrast excludes zero.Source: §3 · p. 3
  • Table 2 reports mean accuracy by condition and domain, weighting wordings and written forms equally; unparsed answers count as incorrect.The fixed readout is the direct answer in tables and mathematics, where the supplied-report condition is undefined.Source: Table 2 · p. 4
  • Simulated tests did not establish nominal error control when conditions differed in accuracy, so contrast counts are exploratory and size conclusions rely on raw paired differences and interventions.The smallest detected widening against reasoning was 0.09 in molecules and 0.10 in tables, larger than the reported repeat-run spread movements.Source: §3 · p. 4

4 Results

Tools improved accuracy when programs directly answered questions, but often widened pairwise wording sensitivity relative to reasoning without tools. The reported comparisons also show ranking reversals under tool use and a separate graph-domain effect from first-token answer scoring.

Repeatability and implementation checks

  • A separate music first-token run agreed on 0.95–0.99 of each of five models’ answers; two 200-item runs at batch size one gave identical answers.In music, operators returned gold answers on 0.99 of bar-count items, 0.63 of key items, and 0.53 of time-signature items.Source: §4 · p. 4

Accuracy gains and tool coverage

  • 0.23–0.56 accuracy gains over the fixed readout occurred in molecules, 0.11–0.24 on graphs, and 0.10–0.15 in music for every model.In music, tools helped on bar-count items but reduced accuracy by 0.04–0.10 on key items; in tables and mathematics, accuracy fell below reasoning without tools for most models.Source: §4 · p. 4

Answer-prefix scoring on graphs

  • 36 of 45 graph pairs had narrower spread when answers were read at an answer line rather than the first token; median spread fell from 0.25 to 0.05.Across letter domains, reductions correlated with accuracy hidden by the first-token read (Spearman 0.58), not with letter mass (−0.12).Source: §4 · p. 4

Pairwise sensitivity under tool use

  • 37 of 60 molecule pairs widened; median spread changes were 0.12 in molecules, 0.08 in tables, 0.06 in music, 0.04 in mathematics, and 0.03 on graphs.Between 0.73 and 0.93 of pairs moved upward across domains; Table 3 flags are exploratory, and zero flags do not establish equivalence.Source: Table 3 · p. 4, §4 · p. 5

Ranking reversals and selection loss

  • Under tools, 4 of 15 pairs reversed in music and molecules and 2 in mathematics, versus none under the fixed readout or reasoning without tools.Selecting the best model under one wording cost an average 0.03 accuracy in music when evaluated on the other four wordings, and less than 0.01 elsewhere.Source: §4 · p. 5

5 Tool Invocation and Interface Effects

Tool-driven comparisons are often more wording-sensitive when models decide whether to call, and the effect varies with call behavior, interface, and task. Supplied reports, alternative interfaces, and instructions help distinguish these effects.

Overall comparison

  • With the plain-text loop, tools widen many pairwise wording spreads against reasoning without tools; in letter domains, median spread exceeds both supplied reports and fixed readout.Tools also widen 54 of 60 molecule pairs relative to the fixed readout.Source: §5 · p. 5

Call frequency and conditional accuracy

  • In molecules, call frequency ranges from 0.94 [0.88, 1.02] to under 0.1 across domains, and calling models are correct in 0.97 to 1.00 of episodes under every wording.Llama-3.1-8B’s graph call frequency ranges from nearly every episode under one wording to fewer than a third under another.Source: §5 · p. 5

Interface, instruction and report-quality interventions

  • In molecules, the native interface and second tool description leave no widened pairs where the plain-text loop widens 10 of 12 in each group; effects of other interventions vary by model.A four-fold token budget reduces budget-cut episodes for Qwen3-8B and Phi-4 but leaves their call-frequency ranges in place; corrupting reports reduces widening stepwise to none.Source: Table 4 · p. 5, §5 · pp. 5–6

Supplied-report controls

  • With reports supplied, no letter-domain pair is wider than the fixed readout in 150 rows, and supplied reports never widen spreads in music.Reasoning with supplied reports also produces no wider spread in 0 of 21 graph and molecule rows; wrong reports can reduce accuracy by up to 0.26 without producing a wider spread in 105 rows.Source: §5 · p. 5

Model size

  • The Qwen3 series does not show a simple size ordering: 4B and 8B call frequencies vary with wording, models from 14B call under almost every wording, and the prior-generation 72B model has the widest measured range.The series varies architecture as well as size, so the comparison does not isolate size.Source: Figure 2 · p. 6, §5 · p. 6

Model-specific instruction effects

  • Output-format instructions affect models differently: removing the answer-alone clause eliminates Llama-3.1-8B’s widened graph pairs, while several models retain widened molecule pairs.Call frequency also responds to paraphrase, tool-description placement, and molecule representation; the five wordings’ call-frequency rankings barely agree across six models (Kendall’s W at most 0.13).Source: §5 · p. 6

Intermediate-result tools and stopping behavior

  • Where tools return intermediate results, calling behavior remains model-specific: constant tools widen 32 of 45 table pairs, and budget stops persist under the tested native interface.The native interface reduces graph widening from 4 to 1 of 9 pairs, while reducing table widening from 6 to 4 of 9; remaining widened table pairs include Mistral-Small-24B.Source: §5 · p. 6

6 Evaluation Recommendations

The paper recommends evaluating model comparisons across wordings while reporting accuracy, pairwise differences, and call frequency together. It also recommends controls and diagnostics that distinguish answer readout, report access, and tool-loop behavior.

Answer-scoring protocol

  • For multiple-choice items, compare first-token scoring with answer-line scoring and report the difference.Letter probability alone does not validate the readout.Source: §6 · pp. 6–7

Comparison protocol

  • Evaluate the same model comparisons under several wordings, reporting accuracy, pairwise differences, and call frequency together.Aggregate call frequency does not show whether models call on the same items.Source: §6 · p. 7

Supplied-report control

  • Run the supplied report under the same wordings to distinguish report access from tool use; stability with supplied reports does not establish what a model knows.Source: §6 · p. 7

Tool-loop diagnostics

  • Report token- and call-budget stops, rejected and repeated calls, and the interface used, especially for tools returning intermediate results.For developers, the target is calling that is consistent and appropriate under equivalent instructions; constant call frequency alone is insufficient.Source: §6 · p. 7

7 Limitations

The measured spreads have limited generalizability: they use five English wordings and a selected set of domains, tasks, and tool operators. Several model and interface results also have restricted replication or scope.

Prompt and measurement scope

  • Spreads are ranges over five wordings, and five wordings support little inference to unseen wordings.The range grows with ten wordings, and intervals reflect item sampling only.Source: §7 · p. 7

Task and tool scope

  • The five domains are not a task sample, and tools directly answer questions in molecules and graphs using operators written for those tasks.The findings therefore say little about tool discovery or argument construction, and prevalence outside these domains is unknown.Source: §7 · p. 7

Replication and model scope

  • The native interface and description variants were tested for only three models; mathematics lacks a full-size repeat, and three models use 8-bit weights.In tables, about half of a model’s predictions change between repeats while pair spreads move by at most 0.036.Source: §7 · p. 7

Interpretation and remaining scope

  • Call-frequency components are algebraic attributions, not causal effects; the wordings are English, and Qwen3-8B’s thinking-mode behavior remains future work.The supplied report is a control rather than an agent, and forced calling moves calling fully for only one model.Source: §7 · p. 7

8 Related Work

Prior work studies prompt sensitivity, answer readout, written input forms, and tool use. This paper connects these threads by measuring how wording affects call frequency and pairwise comparisons within an executed tool loop.

Prompt sensitivity and readout

  • Prior studies report that prompt paraphrases change accuracy and rankings, motivating evaluation across several prompts and estimation across templates [Sclar et al., 2023; Mizrahi et al., 2023; Maia Polo et al., 2024].Source: §8 · p. 7
  • Work on answer extraction finds first-token option probabilities may disagree with written answers, while heuristic extraction and LLM judging affect ranking agreement across templates [Wang et al., 2024; Hua et al., 2025].This paper contrasts first-token and answer-line readouts in the tool-loop setting.Source: §8 · p. 7

Written form of the input

Tool use and the decision to call

9 Conclusion

The authors find that tools can improve accuracy while widening pairwise comparison sensitivity to wording; the effect varies with tool interfaces and instructions.

9 Conclusion

  • Tools raise accuracy where a program answers the question and widen wording sensitivity in pairwise comparisons against reasoning without tools.In molecules, supplied reports leave comparisons most stable; native calling and a second tool description remove detected widening in the tested groups, while sensitivity persists on graphs and with intermediate tool results under the tested native interface.Source: §9 · p. 7

S1 The wordings

The wordings use five templates for each domain and written input form, varying how questions, inputs, and output instructions are presented. Tool descriptions remain unchanged across a domain’s wordings.

S1 The wordings

  • Five instruction templates are used for every written input form, with output-format clauses differing except in the variant without that clause.The same placeholders are filled with the item’s question, written input, form name, and options where applicable; the tool description precedes each wording unchanged.Source: S1 The wordings · p. 10

Symbolic music

  • Music prompts vary the placement of score descriptions, questions, and answer-format instructions across templates.The tool description lists available score-reading calls and permits at most six calls.Source: Symbolic music · p. 10

Molecules

  • Molecule templates place the question and molecular representation in different orders and vary the answer-format instruction.The molecule tool description offers property calls, including heavy-atom count and ring count, with at most six calls.Source: Molecules · p. 11

S2 Items, tools, models and harness

The evaluation covers five question-answering domains with domain-specific tools, scoring rules, and execution budgets. Its inclusion criterion selects domains with imperfect accuracy and wording spread under a fixed readout, except for music.

S2 Items, tools, models and harness

  • The benchmark includes 504 molecule and 504 graph items, 386 music items, 500 FinQA questions, and 336 MATH-500 problems.Molecule and graph items use four options; graph questions cover six tasks, and the music items fit a common token budget across three written forms.Source: S2 Items, tools, models and harness · p. 13
  • Letter answers are scored against the gold option, while numeric answers use domain-specific equality or tolerance rules.For FinQA, percentage conversion and rounding rules apply under specified conditions; stricter scoring preserves every table-row contrast direction and 78% of Holm outcomes.Source: S2 Items, tools, models and harness · p. 13
  • Domains enter if either Qwen3-8B or Phi-4 scores at most 0.90 and their wording spread is at least 0.02 on the first 200 items under the fixed readout.GSM8K fails the criterion with accuracies 0.95 and 0.97; music is exempt.Source: S2 Items, tools, models and harness · p. 13
  • Domain-specific tools use RDKit for molecules, NetworkX for graphs, music21 for music, table operators for FinQA, and SymPy-based tools for mathematics.Run alone, music tools match gold options on 0.99 of bar-count items, 0.63 of key items, and 0.53 of time-signature items.Source: S2 Items, tools, models and harness · p. 13
  • The harness limits calls to six and applies token budgets that vary by domain, with a separate native-interface setup.Under the native interface, calls are structured turns, the step budget is 128 tokens, and prose before a call block is permitted.Source: S2 Items, tools, models and harness · p. 13

S3 Statistical procedure

The statistical procedure estimates uncertainty in wording-spread contrasts with paired bootstrap resampling and Holm correction, then checks calibration and robustness under alternative analyses.

S3 Statistical procedure

  • Intervals use 1,000 paired bootstrap resamples over items, and Holm correction applies across model pairs within each domain, written form, and contrast.The paired resample drives both models, all wordings, and both conditions of a contrast.Source: S3 Statistical procedure · p. 13
  • Under a null of equal, nonzero wording effects, the bootstrap rejects at 0.05 in 0.025 of cases overall, with rates up to 0.04 except for tools versus fixed readout in molecules at 0.12.The within-item exchange test rejects 12 of 180 simulated contrasts before Holm correction and none after it; neither procedure is shown to hold nominal rates when conditions differ in level.Source: S3 Statistical procedure · p. 13
  • An alternative within-item exchange test counts 39 wider and 1 narrower pair, versus 34 and 2 under the bootstrap, with other contrast counts within 16% and no reversals.A no-wording-effect permutation test rejects more often because the fixed readout itself has a wording effect.Source: S3 Statistical procedure · p. 13
  • The standard deviation of accuracy differences across wordings corroborates the range measure: every pair wider by range is also wider by standard deviation.Cluster bootstrapping over shared graphs and tables moves a few counts but changes no direction.Source: S3 Statistical procedure · p. 13

S4 Supporting results

Supporting analyses show that call behavior accounts for much of pairwise comparison movement in molecules, graphs, and music, while interventions on wording, interface, and reports alter the observed spreads. Additional wording, model-size, and repeatability checks qualify how broadly these results generalize.

S4.1 The call term by comparison

S4.2 The forced call

  • Forced calls narrow 10 of 12 molecule comparisons, 5 of 9 graph comparisons, and 3 of 9 music comparisons, widening none.Call rates rose to at least 0.99 for Qwen3-8B in molecules, while ranges shrank but did not close for some other models; item-level correctness also changed more often when call decisions changed.Source: S4.2 The forced call · p. 14, S4.3 The decision item by item · p. 14

S4.4 Option position and permuted options

  • Option permutation moves accuracy by at most 0.04 and changes no Holm outcome among 150 letter-domain comparisons; molecule-format effects differ by representation.InChI raises fixed-readout accuracy over PubChem SMILES by 0.06 [0.03, 0.10] for Qwen3-8B, 0.01 [−0.03, 0.04] for Phi-4, and 0.05 [0.02, 0.08] for Llama-3.1-8B; its lead is concentrated on formula items.Source: S4.4 Option position and permuted… · p. 14, S4.5 The written form of molecules · p. 15

S4.6 Ten wordings and held-out wordings

S4.7 The tool description and its position

S4.9 The native interface by domain

S4.11 Wrong reports and the noisy supplied report

S4.13 Size

Related papers

Loading 2610.31148v1…