Detect Before You Disambiguate: Perturbation-Based Input Ambiguity Detection in LLMs

1 Introduction

• A model that silently commits to one reading can give a confident answer to a question the user did not ask.The authors cite studies finding that current models often fail to recognize ambiguity.Source: §1 · p. 1
• Detecting ambiguity is a prerequisite for trustworthy question answering and can precede clarification or enumeration of plausible interpretations.The paper frames detection and mitigation as two stages.Source: §1 · p. 1
• LLM-generated rephrasing can resolve an ambiguous input before answer disagreement is measured, weakening the ambiguity signal.Understanding an ambiguous question entails resolving it, and the rephrased question may encode one interpretation.Source: §1 · p. 1
• The proposed principle is to use semantics-blind transformations that stay close to the input without becoming a disambiguation step.The paper motivates this trade-off through the information bottleneck, without strictly optimizing that objective.Source: §1 · pp. 1–2
• The framework pairs a white-box detector using model internals with a black-box rule-based detector, then uses detection to drive lightweight re-asking.The white-box method targets important tokens; the black-box method measures answer-distribution dispersion.Source: §1 · p. 2
• The paper reports more reliable separation of ambiguous from unambiguous questions than output-side sampling and paraphrase-based baselines across open-domain, conversational, and visual QA.It also reports downstream gains when the detection output is used for mitigation.Source: §1 · p. 2

2 Related Work

• Ambiguity research spans open-domain, long-form, conversational, scientific, domain-specific, and visual QA, with benchmarks and methods for detection, clarification, and clarification-based answering.The passage names AmbigQA, ASQA, Abg-CoQA, Abg-SciQA, and work on proactive clarification and multimodal ambiguity.Source: §2 · p. 2
• Prior work often studies responses after ambiguity is recognized, including abstention, clarification, and clarification-based QA pipelines.This paper focuses on detecting whether the original input is ambiguous before resolution.Source: §2 · p. 2
• Output-side sampling can reflect either model unreliability or multiple valid interpretations, while input-side perturbations may remove ambiguity before measurement.The framework therefore requires label-preserving perturbations, instantiated with black-box or white-box methods.Source: §2 · p. 2

3.1 Overall Framework

• The framework samples answers from the original and perturbed questions and scores the resulting answer set with EigV(Agree).It draws five answers from the original and one from each perturbation; low semantic agreement indicates likely ambiguity.Source: Figure 1 · p. 3, §3.1 · pp. 3–4
• Small, label-preserving interventions are intended to expose answer instability without turning perturbation into a disambiguation step.The information-bottleneck view motivates the design, but the paper does not optimize a strict objective or estimate mutual information.Source: §3.1 · p. 3
• Valid perturbations make small surface changes, preserve the ambiguity label, and avoid semantic rewrites or interpretation choices.The framework applies these principles to original questions before sampling answers.Source: §3.1 · p. 3

3.2 White-Box Perturbation

• White-box perturbation uses model-internal token importance to select positions for low-capacity interventions, rather than rewriting the question.The methods include Vanilla Gradient, DiffMask, and Logit Lens; selected tokens can be masked, deleted, or replaced.Source: §3.2 · p. 4

3.3 Black-Box Perturbation

• The black-box detector applies fixed rules based on surface features and pools answers across the original question and its variants for one dispersion score.Rules include constrained synonym substitutions, contraction changes, register prefixes, and guarded wh-word transformations; pooled scoring is intended to estimate answer movement across variants.Source: §3.3 · pp. 4–5

4.1 Experimental Setting

• Evaluation spans four ambiguity-annotated English QA benchmarks and three model families, comparing two detector variants with four uncertainty-based baselines.The benchmarks cover conversational, contextual text, and visual QA.Source: §4.1 · p. 5

4.2 Main Results (RQ1)

• Across all three model families, the two variants rank among the top two on every model block, with up to 0.07 higher average AUROC than the strongest baseline.At least one variant achieves the best average AUROC in each model family.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Analysis of Different Importance and Perturbations (RQ2)

• The three token-importance methods achieve similar AUROC, with gaps of around 0.02 between the best and worst and Vanilla Gradient slightly more stable.The comparison uses Vanilla Gradient, Logit Lens, and DiffMask on ABG-CoQA and AmbigQA.Source: §4.3 · p. 7, Figure 2 · p. 7
• Pooling answers across all applicable perturbation rules achieves the strongest performance, consistent with ambiguity signals being distributed across heterogeneous perturbations.The comparison includes individual rules and a cross-variant setting that pools the original question with all rule variants.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4 Mitigation Performance (RQ3)

• Exact Match rises by +8.15 to +18.07 points with UAR across every subset of both datasets compared with the Original baseline.UAR reuses detection artifacts and spends one additional forward pass; it also improves results on gold non-ambiguous subsets.Source: §4.4 · pp. 7–8, Table 2 · p. 8

4.5 Ablation Study

• On AQuA, LLM paraphrasing drops AUROC from 0.708 to 0.637, while LLM-executed fixed rules also underperform deterministic semantics-blind rewrites.With paraphrasing, absolute uncertainty falls below the EigV(Agree) baseline; the fixed-rule comparison spans text and vision settings.Source: Figure 4 · p. 8, §4.5 · p. 8, Table 4 · p. 9

4.6 Failure Case Analysis

• On AQuA, 154 of 164 false negatives are referential or deictic; false positives include counting questions and cases requiring fine-grained visual perception.The authors suggest missed cases can arise when a model silently commits to a salient referent, while answer instability can reflect model or perceptual uncertainty.Source: §4.6 · pp. 8–9

4.5 Ablation Study

• Average cosine similarity between original and perturbed inputs ranges from 0.883 to 0.999 across datasets and model families, indicating limited semantic drift.For visual QA, the comparison uses the edited question because the image remains unchanged.Source: §4.5 · p. 8, Table 3 · p. 9

• Perturbation-based detectors distinguish ambiguous from unambiguous questions more effectively than output-side sampling and paraphrase-based baselines, while evidence-augmented re-asking improves answer accuracy on under-specified questions.Source: §5 · p. 9

Limitations

• The black-box rule library is manually designed for English, while the white-box detector requires model internals that may be unavailable through closed-source APIs.Applying the black-box method to other languages requires reconstructing its rules.Source: Limitations · p. 9

A Detail Technology of Uncertainty Estimation

• The estimator forms a pairwise semantic-agreement matrix over sampled answers, then uses its degree matrix to compute a symmetric normalized graph Laplacian.Source: A Detail Technology of Uncertainty… · p. 13
• Smaller Laplacian eigenvalues indicate semantic clusters among sampled answers, while a larger computed value signals more semantic modes and thus ambiguity.Source: A Detail Technology of Uncertainty… · p. 13

B.1 More Results on Different White-box Importance

• Across text-only datasets and models, white-box importance methods have similar AUROC, with Vanilla Gradient slightly better and more stable across settings.Source: §B.1 · p. 13

B.2 Comparison with Ambiguity-Aware Prompting

• The selected white-box importance method outperforms simple attention and predictive entropy on Abg-CoQA and Qwen3.Source: Figure 5 · p. 13
• Both perturbation-based detectors achieve higher mean AUROC across four datasets than ProCoT, which itself outperforms Direct Prompt.Source: Table 5 · p. 13, §B.2 · p. 13

B.3 Controlled Analysis of UQ-Augmented Re-asking

• Across text-only datasets, white-box importance methods perform stably, with Vanilla Gradient slightly better than the alternatives.Figure 6 compares different white-box importance methods.Source: Figure 6 · p. 14
• UAR consistently outperforms Hint and SelfSample on Qwen3-4B in Exact Match, indicating benefits from detector-derived evidence beyond generic re-prompting or additional sampled context.Table 6 compares UAR with the two controlled re-prompting baselines.Source: §B.3 · p. 13, Table 6 · p. 14

B.4 Sensitivity to Perturbation Budget

• Detection performance generally improves as more black-box perturbation rules are included and saturates around nine rules on AmbigQA with Qwen3-4B.Table 7 reports ambiguity-detection AUROC under different perturbation budgets.Source: §B.4 · p. 14, Table 7 · p. 14

B.5 More Ablation

• Simply using attention or predictive entropy does not improve results over the paper’s chosen white-box importance methods.Attention aggregates per-layer attention flow; predictive entropy scores tokens using next-token distribution entropy.Source: §B.5 · p. 14

C Detail Methods for Black-Box Perturbation

• The black-box rules are semantics-blind: they use surface features such as part of speech, orthography, or the leading wh word, not model-generated readings.The appendix organizes the rules by edit granularity.Source: C Detail Methods for Black-Box… · p. 14

C.1 Group A: Single Word Level

• Single-word rules substitute verbs, nouns, adjectives, and adverbs with filtered WordNet synonyms, reinflecting forms where needed, and also expand or contract function-word abbreviations.The synonym rules constrain candidates and preserve grammatical features such as tense, number, or degree.Source: §C.1 · p. 14

C.2 Group B: Sentence and Phrase Level

• Sentence- and phrase-level rules add short register or instruction prefixes, or rephrase leading wh words with safeguards for wh determiners.Examples include conversational openers and replacing “Who” with “Which person.”Source: §C.2 · p. 15