PaperlayerView exact revision on arXiv

Source-linked AI summary

When Verifiable Counts Depend on Wording: Auditing Wording Robustness in Instruction Following

Qishi Zhan, Seoyeon Jang, Zihan Dong, Minxuan Hu, Ziheng Chen, Tonghui Qu

arXiv:2610.05278v1cs.CLcs.AI

TL;DR

Verifiable instruction-following benchmarks commonly express each constraint through a single template, leaving unclear whether scores remain stable across equivalent wordings. WISE uses matched wording variants and finds that compliance and model comparisons can shift with construction and constraint. It therefore reports performance across forms rather than relying on one template score.

  • Problem

    Verifiable instruction-following benchmarks commonly use one fixed template per constraint, although the same operational target can be expressed in multiple ways.

    Source: §1 · p. 1

  • Method

    WISE holds task, target, checker, model, decoding configuration, and output scope fixed while varying wording across exact-count, keyword-inclusion, and word-range constraints.

    Source: §1 · p. 2

  • Results

    Matched verifier-equivalent instructions change measured compliance, with the direction and size depending on the predicate and construction rather than one surface form being uniformly harmful.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    WISE supplements conventional scores with mean and worst-form compliance, wording gaps, failure direction, and ranking stability.

    Source: §6 · p. 8

    WISE is instantiated mainly on three English count and quantity predicates, and its JSON structural probe is intentionally narrow.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Verifiable instruction-following benchmarks often express each constraint through one fixed template. We test whether scores remain stable when the operational requirement is unchanged but its wording varies. We introduce WISE, a matched evaluation suite and reporting protocol instantiated on exact word count, keyword inclusion exactly once, and an inclusive 8--12 word range. Across 100 matched tasks, up to thirteen models from seven providers, and repeated generations scored over the complete visible output, wording alone produces substantial compliance shifts. In an avoidance-family panel, five avoidance and exclusion forms fall below the positive baseline, while constructional controls also shift compliance substantially: in the nine-model control panel, compliance is 54.9% for the original positive form, 48.2% for a longer positive form, 36.7% when the target appears later, and 33.8% for AVOID1. A strict JSON-structure probe shows wording sensitivity beyond counting, with a different direction of effect. Effect sizes, failure directions, weakest forms, and model rankings vary across realizations. Under the most disruptive exclusion form, the top-ranked model changes and 24.1% of strictly ordered model pairs reverse. Human validation further shows that unanimous agreement on an exact-count interpretation can coexist with substantially different model behavior. WISE supplements conventional scores with mean and worst-form compliance, wording gaps, failure profiles, and ranking stability.

1 Introduction

WISE tests whether verifier-equivalent instructions yield stable compliance when their wording changes, holding the task and evaluation conditions fixed. The paper finds construction-specific shifts in compliance and rankings, motivating reports beyond a single template score.

1 Introduction

  • The study tests exact word count, keyword inclusion exactly once, and an inclusive 8–12 word range using transparent checkers on complete visible outputs.Exact word count is the primary probe; the keyword and range extensions test lexical counting and a broader acceptable interval.Source: §1 · pp. 1–2
  • Wording sensitivity is construction-specific: avoidance and exclusion forms can be disruptive, while affirmative controls and a strict JSON probe show that polarity alone does not explain the effects.The direction of wording effects can vary by constraint.Source: §1 · p. 2
  • A single-template score reflects both satisfaction of the operational predicate and compatibility with the selected linguistic realization.Wording changes can also alter failure profiles and comparative model rankings.Source: §1 · p. 2
  • WISE isolates wording effects by holding the task, target, checker, model, decoding configuration, and output scope fixed.It reports formwise compliance alongside mean and worst-form performance, wording gaps, failure direction, model-level variation, and ranking stability.Source: §1 · p. 2

2 Related Work

Prior work studies verifiable instruction following, negation, and prompt sensitivity. WISE isolates a matched contrast in which the operational predicate stays fixed while only its linguistic realization changes.

Negation in language models

  • Negation research examines semantic representation and inference, whereas this study tests constrained generation across direct negation, complex negation, avoidance, and exclusion under a fixed target and checker.Prior work reports challenges with negation in cloze probes, inference, and generation [Kassner and Schütze, 2020; Hossain et al., 2020, Truong et al., 2022; Truong et al., 2023].Source: §2 · p. 2

Verifiable instruction-following evaluation

  • Verifiable instruction-following benchmarks use deterministic checkers for length, keyword, formatting, and structural requirements, while later benchmarks extend to fine-grained, compositional, and unseen constraints.The cited examples include IFEval [Zhou et al., 2023], FollowBench [Jiang et al., 2024], InFoBench [Qin et al., 2024], and IFBench [Pyatkin et al., 2025].Source: §2 · p. 2

Prompt sensitivity and evaluation reliability

3 Problem Setup

WISE compares wording forms within matched task–model conditions and evaluates the complete visible output with a deterministic verifier. It summarizes both compliance and variation across forms, including changes in model rankings.

3 Problem Setup

  • For each task, model, wording form, and repeated generation, the base request and operational target stay fixed while only the target’s wording realization changes.The matched design holds the verifier, decoding configuration, output scope, and number of generations fixed, allowing differences across forms to estimate wording sensitivity.Source: §3 · p. 3
  • The verifier evaluates the complete visible output, counting whitespace-delimited tokens and retaining headings, Markdown, framing text, and wrapper content.The keyword condition requires exactly one case-insensitive whole-word occurrence; the range condition accepts 8–12 words inclusively.Source: §3 · p. 3
  • WISE summarizes form-level compliance with mean and worst-form performance and a wording gap, alongside model-specific measures and failure profiles.Model-specific summaries distinguish shared responses from pooled averages over divergent systems.Source: §3 · p. 3
  • Model rankings are compared across wording forms using Kendall’s τb, strict pairwise reversals, top-three overlap, and maximum absolute average-rank change.Pairs tied under either form are excluded from the reversal rate.Source: §3 · p. 3

4 Experimental Setup

The experiments use matched tasks and wording forms, repeated generations, and complete-output verification across count and quantity constraints, with a strict JSON probe as a structural extension. Model and uncertainty analyses are reported for the panels available under each comparison.

Tasks and target counts

  • WISE contains 100 base tasks balanced across target counts of 5, 8, 10, and 12 words, with three independent generations per model–item–form combination.Tasks were screened for factual stability, answerability without specialized knowledge, and compatibility with all four target counts.Source: §4 · pp. 3–4

Wording forms

  • The exact-count panel compares six natural realizations, and its control panel contrasts the original positive form with a longer positive form, a later-target form, and AVOID1.The forms preserve the same operational predicate; the matched control retains the same tasks, target counts, repetitions, token cap, and complete-output verifier.Source: §4 · p. 4

Models

  • The study evaluates thirteen instruction-tuned models from seven providers, while the keyword and range extensions use the seven models available when those panels were run.Exact-count experiments use all thirteen models; the primary extension contrasts use the six shared non-Gemini models.Source: §4 · pp. 4–5

Main metric and uncertainty

  • The primary outcome is strict exact-count compliance over the complete visible output, with mean absolute count error and under-target and over-target rates also reported.Exact-count uncertainty uses a provider–model–item hierarchical bootstrap with 2,000 replicates, while keyword and range panels use item-cluster and model–item bootstraps, respectively.Source: §4 · p. 4

Cross-constraint and structural extensions

  • The extensions reuse the 100 base tasks for keyword and range panels, and add a strict JSON probe requiring parseable output with exactly the requested key set and no extra prose.In the JSON probe, the base request and target field specification are held fixed while wording varies across POS, NEG, and AVOID.Source: §4 · pp. 4–5

Cross-constraint extensions

  • The cross-constraint figure shows AVOID with the lowest pooled compliance across exact count, keyword inclusion, and the 8–12 range; NEG effects vary by constraint.The NEG interval includes zero for the range, and exact-count AVOID denotes AVOID1.Source: Figure 2 · p. 5

5 Results

Across count and quantity settings, AVOID has the lowest pooled compliance, while direct negation varies by constraint. The results show that wording effects depend on construction and constraint type, and can also alter model rankings and human–model agreement.

5.1 Exact-Count Wording Sensitivity

  • AVOID1 falls 20.5 percentage points below POS in the original exact-count panel, while AVOID2 and CNEG remain close to the positive baseline.NEG and NEVER occupy an intermediate regime.Source: §5.1 · p. 5, Table 2 · p. 6

5.2 Constructional Controls

  • Compliance follows POS > POS_LONG > POS_END > AVOID1; later target placement lowers compliance by 18.2 points, versus 6.7 points for longer affirmative wording.The controls show that polarity alone does not account for the observed differences.Source: §5.2 · p. 5, Table 3 · p. 6

5.5 Transfer Beyond Exact Count

  • 77.4% under POS versus 55.3% under NEG and 44.3% under AVOID for keyword inclusion; for the 8–12-word range, compliance is 89.4% under POS, 85.8% under NEG, and 80.6% under AVOID.The wider range attenuates, but does not eliminate, wording sensitivity.Source: §5.5 · pp. 6–7

5.3 Avoidance and Exclusion Family Robustness

  • All tested avoidance and exclusion forms fall below POS, with decreases ranging from 12.2 to 30.5 points; failure directions and model vulnerabilities vary by form.AVOID1 and AVOID4 produce errors in both directions, AVOID3 and EXCLUDE2 more often undershoot, and EXCLUDE1 has the highest rate of extremely short outputs.Source: §5.3 · pp. 5–6, Table 4 · p. 7

5.4 Wording Alters Comparative Model Rankings

  • Across forms, 7.4%–24.1% of strictly ordered model pairs reverse relative to POS; EXCLUDE1 causes the largest disruption and changes the top-ranked model.Claude Opus 4.8 ranks first under POS but fifth under EXCLUDE1, while Claude Sonnet 4.6 becomes top-ranked.Source: §5.4 · p. 6, Table 5 · p. 7, Figure 3 · p. 8

5.6 Transfer to a Structural Constraint

  • Under strict JSON checking, AVOID has the highest pooled compliance; unparseable output dominates failures, while wrong key sets are essentially absent.This direction differs from the count experiments, so the evidence supports wording sensitivity rather than a claim that avoidance always hurts compliance.Source: Table 6 · p. 7, §5.6 · p. 7

5.7 Human Interpretation of Wording Forms

  • All five exact-count forms receive at least 90% exact-N judgments, yet unanimous judgments for POS and EXCLUDE2 coexist with a 12.2-point difference in model compliance.The wording effects therefore cannot be explained solely by whether people recognize the intended exact-count requirement.Source: §5.7 · p. 7

5.8 Statistical Robustness

  • After Holm correction, NEG, NEVER, and AVOID1 remain significant in the exact-count panel; both keyword contrasts and range AVOID remain significant, while range NEG is inconclusive.AVOID2 and CNEG do not remain significant after correction in the original exact-count panel.Source: §5.8 · pp. 7–8

6 Discussion and Conclusion

The findings show that verifier-equivalent wording can change scores in ways that are specific to construction and constraint, rather than uniformly driven by avoidance or negation. WISE reports these effects across declared matched variants, while its worst-form measure remains conditional on the chosen variant set.

Discussion and Conclusion

  • Wording sensitivity is construction-specific: affirmative controls also reduce compliance, and the JSON probe reverses the avoidance effect seen in count settings.Human judgments reinforce the result: POS and EXCLUDE2 receive unanimous exact-N judgments but differ by 12.2 percentage points in model compliance.Source: §6 · p. 8
  • A single-template score reflects both satisfaction of the operational predicate and compatibility with the selected linguistic realization, so it may yield unstable compliance estimates and rankings.WISE holds task, target, checker, model, decoding configuration, and output scope fixed while varying wording.Source: §6 · p. 8
  • WISE is a reporting protocol, not a universal worst-form leaderboard, because worst-form compliance depends on the chosen variant set.It tests stability across a declared matched variant set.Source: §6 · pp. 8–9
  • Across studied settings, wording can change measured compliance, but the direction and size depend on the predicate; paired contrasts are therefore more informative than one robustness number.Avoidance wording lowers compliance in exact-count and keyword settings, has a weaker effect for the permissive range, and shows a different direction for JSON.Source: §6 · p. 9

Limitations

WISE is limited by its narrow constraint inventory, wording design, verifier, model coverage, and small human-validation sample. Its findings therefore characterize the tested settings rather than a broader set of constraints or populations.

Limitations

  • WISE mainly covers three English count and quantity predicates; its JSON probe is narrow, leaving other constraint families untested.The proposed extensions include ordering, capitalization, citation format, tool-use schemas, and multi-constraint composition.Source: Limitations · p. 9
  • The natural wording forms vary jointly in lexical choice, scope, syntax, and pragmatic form, so the design cannot isolate each feature’s contribution.The affirmative controls partially address this, but a larger factorial design would be needed to estimate those contributions separately.Source: Limitations · p. 9
  • Exact-count and range rates depend on whitespace-delimited counting over complete visible output; alternative tokenization or normalization could change absolute rates.The reported conclusions are relative wording sensitivity under the stated verifier.Source: Limitations · p. 9
  • The exact-count study covers thirteen models, but keyword and range panels cover seven, with cross-constraint comparisons limited to six shared non-Gemini models.Absolute rates are not directly comparable across providers, and two Gemini adapters with saved-output floor behavior are excluded from primary wording contrasts.Source: Limitations · p. 9
  • Human wording validation used five annotators, a small English-speaking sample that cannot establish how broader populations or languages would judge the forms.The ratings contextualize clarity, naturalness, ambiguity, and complexity.Source: Limitations · p. 9

Ethical Considerations

The materials document WISE’s ethical scope alongside its model coverage, failure analyses, statistical procedures, and Gemini output diagnostics. They also specify where saved-output uncertainty constrains the main wording contrasts.

Ethical Considerations

  • WISE uses short, non-sensitive factual and explanatory prompts and is intended to complement, not replace, broader capability evaluations.The work uses hosted language-model APIs, incurring inference-related computational and environmental costs.Source: Ethical Considerations · p. 9

A Model and API Details

  • The exact-count study uses thirteen models, while keyword and range extensions use seven models available when those runs were executed.Table 7 lists model identifiers and access dates.Source: A Model and API Details · p. 12, Table 7 · p. 13

B Paired Failure Examples

D Surface-Behavior Audit

  • Across eleven primary non-Gemini models, AVOID3 and EXCLUDE2 errors concentrate below target, while AVOID1 and AVOID4 err in both directions.EXCLUDE1 has the broadest distribution and 9.3% of outputs contain at most two words, versus 0.5% under POS; meta-discourse openings occur in fewer than 0.4% of outputs under every form and do not explain the main compliance differences.Source: D Surface-Behavior Audit · p. 13, Figure 4 · p. 13

F Ranking Stability Metrics

  • Figures 5 and 6 visualize model-specific compliance gaps and under-target, exact, and over-target rates in the avoidance-family panel.The ranking metrics include Kendall’s τb, top-three Jaccard overlap, maximum absolute rank displacement, and strict pairwise reversals relative to POS; ties are excluded from reversal rates.Source: E Additional Figures · p. 13, Figure 5 · p. 14, Figure 6 · p. 14

G Statistical Robustness

  • Multiplicity is assessed within each primary experimental family using Holm correction, with bootstrap p-values drawn from the same draws as the confidence intervals.Table 10 reports the complete results.Source: G Statistical Robustness · p. 14, Table 10 · p. 15

H WISE Audit Summary

  • Table 11 summarizes the original exact-count panel, avoidance-family panel, constructional controls, keyword constraint, inclusive length range, and JSON probe.Figure 7 presents original-form contrasts against POS with provider–model–item bootstrap 95% confidence intervals.Source: H WISE Audit Summary · pp. 14–15, Figure 7 · p. 15, Table 11 · p. 16

I Gemini Saved-Output Diagnostics

  • Both Gemini models meet the saved-output floor criterion, with very short outputs and near-zero exact compliance at targets 8, 10, and 12.Raw response bodies, finishReason, and per-part thought flags were not retained, so response-part extraction cannot be fully excluded as a contributor; both models are excluded from primary wording-specific contrasts.Source: I Gemini Saved-Output Diagnostics · pp. 15–16, Figure 8 · p. 15, Table 12 · p. 16

J Cross-Constraint Pooled Results

The cross-constraint pooled-results tables report analyses for the six non-Gemini models shared by the keyword and range extensions, alongside corresponding seven-model analyses that include Gemini 2.5 Flash.

J Cross-Constraint Pooled Results

K Constructional-Control Model-Level Results

The constructional-control results use a retained nine-model panel, while the JSON-structure probe reports its model- and form-level results. The stated main JSON failure mode is unparseable complete visible output.

K Constructional-Control Model-Level Results

L JSON-Structure Model-Level Results

M Length-Range Model-Level Results

The model-level sections report complete form-by-form results for range, keyword, and exact-count constraints, alongside bootstrap sensitivity analyses. Human validation finds broad exact-count agreement while model compliance still differs across wording forms.

M Length-Range Model-Level Results

  • The length-range model-level results report 6,300 unique generations, with Claude Sonnet near ceiling and Gemini 2.5 Flash at floor.The experiment had zero API errors, missing rows, or duplicate keys.Source: M Length-Range Model-Level Results · p. 16

N Keyword Model-Level Results

  • The keyword section reports complete model-by-form results; Table 15 covers keyword-exactly-once robustness across seven models, including Gemini 2.5 Flash.The table reports percentages and defines zero and multiple keyword occurrences as separate outcomes.Source: N Keyword Model-Level Results · p. 16, Table 15 · p. 17

O Exact-Count Model-Level Results

  • The exact-count section reports complete model-by-form compliance values in Tables 21 and 22, including Gemini models separated from the main text for their floor effects.The tables cover the original six-form and avoidance-family panels.Source: O Exact-Count Model-Level Results · p. 17

P Bootstrap Sensitivity Analyses

  • Model–item and equal-model item bootstrap sensitivity analyses on the non-Gemini set preserve the key contrasts from the main hierarchical bootstrap.The sensitivity analyses address the small provider count and unbalanced hierarchy.Source: P Bootstrap Sensitivity Analyses · p. 17

Q Human Wording Validation

  • Five annotators completed 24 randomized wording-and-target items, yielding 120 judgments for each measure.They judged exact-N interpretation and rated clarity, naturalness, ambiguity, and complexity.Source: Q Human Wording Validation · p. 17

Q.1 Annotator-Level Exact-Count Judgments

  • POS and EXCLUDE2 receive exact-N judgments in all twenty annotator and target combinations; the four remaining forms receive exact-N judgments in 90% to 95% of combinations.Source: §Q.1 · p. 17

Q.2 Inter-Annotator Agreement

  • Exact-N judgments show moderate agreement (Fleiss’ κ = 0.54), while clarity and naturalness ratings have ICC values of 0.79 and 0.75.Ambiguity and complexity ratings have ICC values of 0.68 and 0.64; the authors treat these ratings as descriptive evidence while retaining uncertainty about fine-grained differences.Source: §Q.2 · pp. 17–18
  • EXCLUDE2 lowers pooled model compliance by 12.2 percentage points relative to POS, despite unanimous exact-N judgments for both forms.AVOID1, AVOID4, and EXCLUDE1 also receive 90% exact-N judgments while producing considerably larger compliance reductions.Source: §Q.2 · p. 18

R Reproducibility Details

The reproducibility materials preserve task and wording manifests, full prompts, generations, verifier outputs, and analysis summaries. Matched examples illustrate how affirmative, avoidance, exclusion, and JSON wording can produce distinct visible-output failures.

R Reproducibility Details

  • The panels use three generations per model and form, with six forms for exact-count tasks, four for constructional controls, and three for keyword, range, and JSON probes.The JSON probe uses a strict full-output parser.Source: R Reproducibility Details · p. 18
  • Exact-count and range checkers count whitespace-delimited tokens after stripping output, so headings, Markdown markers, and wrappers count when whitespace-separated.Multiple spaces and newlines collapse under splitting; empty or whitespace-only output counts as zero.Source: R Reproducibility Details · p. 18
  • The release includes raw generations, verifier results, counts, compliance and failure indicators, bootstrap summaries, model profiles, paired examples, and audit summaries.The 100-task manifest records item IDs, base tasks, and target counts; separate manifests record wording templates, and raw files include instantiated instructions and complete prompts.Source: R Reproducibility Details · pp. 18–19
  • Matched examples show a longer affirmative form failing where POS succeeds, a late-position target with compliant POS output, and avoidance outputs that can either overshoot or undershoot.The late-target example identifies a compliant matched POS generation; the avoidance examples show that failure direction is not uniform.Source: R Reproducibility Details · p. 20
  • An exclusion example shows extreme-short answer suppression, while a JSON example shows Markdown fencing causing strict full-output parsing to fail despite valid-looking JSON.The JSON comparison also indicates that avoidance wording can reduce wrapper failures in that constraint setting.Source: R Reproducibility Details · p. 21
Loading 2610.05278v1…