PaperlayerView exact revision on arXiv

Source-linked AI summary

What Is a Clarification? Estimating Model-Dependent Input Ambiguity

Jingqin Tang, Tim Tomov, Sebastian Schmidt, Dominik Fuchsgruber, Stephan Günnemann

arXiv:2610.20052v1cs.CL

TL;DR

Ambiguous inputs can admit multiple answers, but a particular model may know and answer only some of them. The paper formalizes a correspondence between answers and clarification classes, then uses an answer–clarification–answer cycle to estimate model-accessible answers. CIRCLECLAR recovers answer sets across QA and executable Text-to-SQL, with reported benefits for ambiguity detection and response selection.

  • Problem

    Input ambiguity is commonly treated as a property of the question, although a particular model may know and answer only some of its valid alternatives.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    The paper formalizes clarifications through uniqueness, faithfulness, and answer equivalence, then estimates model-realizable answers by checking recovery through answer–clarification–answer cycles.

    Source: §1 · p. 2

  • Results

    CIRCLECLAR achieves the best answer-set F1 in seven of twelve model–dataset settings, beats direct-generation baselines across QA settings, and performs strongly on executable Text-to-SQL.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    Recovered model-realizable support predicts model-dependent ambiguity and provides a basis for choosing whether to answer, clarify, or abstain.

    Source: §1 · p. 2

    CIRCLECLAR’s recall is limited by candidate generation, and successful recovery does not guarantee factual correctness because a model may consistently reproduce an error.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

Ambiguous inputs can admit multiple interpretations, each potentially yielding a different answer. Existing work largely treats this ambiguity as a property of the input, yet for a particular language model, only some alternatives may be known and answerable. We therefore distinguish world ambiguity, capturing all alternatives admitted by an input, from model ambiguity, capturing those accessible to the responding model. To characterize these alternatives, we formalize natural-language clarifications through three desiderata: \emph{uniqueness}, \emph{faithfulness}, and \emph{answer equivalence}. We show that these desiderata induce a one-to-one correspondence between valid answers and equivalence classes of clarifications, implying that any interpretation can be represented either by its answer or by the clarifications that uniquely express it. We leverage this correspondence to introduce \circleclar{}, a black-box method that estimates the set of answers available to a model by checking consistency along an answer--clarification--answer cycle. Across open-domain QA and executable Text-to-SQL, \circleclar{} achieves strong recovery of these answer sets. In QA, the recovered support yields the strongest overall performance in model-dependent ambiguity detection and enables effective selection among answering, clarifying, and abstaining. Together, these results demonstrate the practical value of a model-dependent view of ambiguity and its characterization through an answer-clarification correspondence.

1 Introduction

The paper distinguishes ambiguity admitted by an input from the alternatives accessible to a particular model. It formalizes clarifications and uses their correspondence with answers to estimate the model-accessible alternatives.

1 Introduction

  • World ambiguity is fixed by the input, while model ambiguity depends on which valid alternatives the responding model knows and can answer.The same question can therefore present different accessible alternatives to different models.Source: §1 · pp. 1–2
  • Valid clarifications are characterized by uniqueness, faithfulness, and answer equivalence, yielding a one-to-one correspondence between valid answers and clarification classes.An interpretation can be represented either by its answer or by clarifications that uniquely express it.Source: §1 · p. 2
  • CIRCLECLAR estimates a model’s answer set by testing whether candidate answers return after an answer–clarification–answer cycle.It is evaluated on open-domain QA and executable Text-to-SQL.Source: §1 · p. 2
  • CIRCLECLAR achieves the best answer-set F1 in seven of twelve model–dataset settings and outperforms direct-generation baselines across all QA settings.Its recovered support also predicts model-dependent ambiguity and supports choosing whether to answer, clarify, or abstain.Source: §1 · p. 2

2 Related Work

Prior work represents, estimates, and acts on input ambiguity in several ways. Comparatively less attention has focused on how ambiguity depends on the responding model’s knowledge.

2 Related Work

  • Input ambiguity has been represented through disambiguated questions, distinct executable programs, structured interpretation–answer pairs [16], and ground-truth distributions over semantic answer classes and relative likelihoods.Source: §2 · pp. 2–3
  • Estimation approaches use clarification ensembles for uncertainty, embedding-based clustering, semantic diversity, sampled-answer distributions, or interpretation-to-answer stages.Source: §2 · p. 3
  • Clarification systems and benchmarks study ambiguity detection, question generation, and clarify-or-answer policies,,,,,.Models may identify ambiguity when asked but still answer by default in ordinary QA [18]; BAG uses sampled responses to guide answering, clarifying, or abstaining [1].Source: §2 · p. 3
  • APA treats ambiguity as model-dependent through changes in average token entropy, while this work also formalizes two ambiguity types and estimates actionable clarifications and answers.Source: §2 · p. 3

3 Why Input Ambiguity Should be Model Dependent

The section argues that input ambiguity alone does not capture what a particular model can realize. It defines model-realizable support and reports that this support varies across model scale and tracks world support imperfectly.

3 Why Input Ambiguity Should be Model Dependent

  • Input ambiguity is usually studied independently of the model, although ambiguity relevant to a response depends on which alternatives the model can realize.The paper treats ambiguity as model-dependent, following Kim et al..Source: §3 · pp. 3–4
  • World answer support contains all correct answers admitted by the question, whereas model-realizable support contains valid answers the model can produce.For the unspecified Jungle Book adaptation, a model may know both film-specific answers or only one.Source: §3 · p. 4
  • Larger, more capable models tend to realize more of the ambiguity and show higher input-level Spearman correlation between model-realizable and world support sizes.Even the strongest model remains substantially below the mean world-support size; Figure 2 reports mean size and Spearman correlation with 95% bootstrap confidence intervals.Source: §3 · p. 4, Figure 2 · p. 4
  • A model that realizes only one answer faces a different response decision from one that realizes several, so the appropriate choice can depend on model-specific support.The possible choices include answering, clarifying, or abstaining.Source: §3 · p. 4

4 A Clarification Framework for Estimating Model-Dependent Ambiguity

The framework formalizes valid clarifications and proves that clarification classes correspond to answers. CIRCLECLAR uses this correspondence to test candidate answers through repeated recovery under clarifications.

4 A Clarification Framework for Estimating Model-Dependent Ambiguity

  • The framework targets model-realizable answer support, formalizes valid clarifications, and uses their equivalence to the answer space to construct a black-box estimator.The paper notes that LLMs without top-p or top-k sampling assign non-zero probability to every sequence, and in practice considers outcomes with non-negligible probability mass.Source: §4 · p. 4

4.1 Formally Defining Clarifications

  • A valid clarification must uniquely determine an answer and preserve the original question’s meaning; different phrasings that induce the same answer count as equivalent.The three desiderata are uniqueness, faithfulness, and answer equivalence.Source: §4.1 · p. 5
  • Under these desiderata, and assuming every valid answer has a clarification, each valid answer corresponds to exactly one clarification class.Thus an interpretation can be represented either by its answer or by the class of clarifications that induce it.Source: §4.1 · pp. 5–6

4.2 CIRCLECLAR: Estimating Model-Realizable Answer Support

  • CIRCLECLAR enumerates candidate answers, generates a clarification for each, and retains candidates whose recovery probability under repeated re-answering meets threshold τ.Reliable recovery provides evidence that a candidate belongs to model-realizable support.Source: §4.2 · p. 6

5 Experiments

The experiments evaluate CIRCLECLAR’s recovery of model-realizable answer sets across open-domain QA and executable Text-to-SQL, then test whether those sets support model-dependent ambiguity detection and response selection.

5 Experiments

  • The evaluation covers 1,000 AmbigQA questions, 100 curated AmbigQA-HQ questions, and 96 ambiguous AMBROSIA Text-to-SQL requests.Reference model support includes answers that the evaluated model reliably recovers from their ground-truth clarifications; baselines include adapted ICE, 50-sample generation, and one-shot enumeration.Source: §5 · pp. 6–7, §5.1 · p. 7

5.1 Estimating the Model-Realizable Answer Support MT (x)

  • CIRCLECLAR performs strongly across datasets, with particularly large gains over baselines on smaller models and AmbigQA-HQ.The evaluation reports mean per-question set F1 and exact set match for QA and Text-to-SQL.Source: §5.1 · p. 7, Table 1 · p. 7
  • CIRCLECLAR’s gains are most consistent in precision, while one-shot enumeration performs remarkably well on larger models.The authors also report that unambiguous-question evaluations show the method largely avoids hallucinating alternatives and outperforms other methods on smaller models.Source: §5.1 · p. 7
  • Consistency filtering substantially increases precision across nearly all settings, with a generally small recall decrease; its precision gain diminishes with model scale.The filtering step removes candidates, and larger models hallucinate fewer invalid initial candidates.Source: Figure 3 · p. 7, §5.1 · p. 8

5.2 Downstream Uses of Model-Realizable Support

  • CIRCLECLAR achieves the highest overall AUROC for model-dependent ambiguity detection, with its clearest advantage on AmbigQA-HQ.The detection score is the cardinality of the recovered answer support; the authors attribute weaker standard AmbigQA results to noisy or incomplete annotations.Source: §5.2 · p. 8, Figure 4a · p. 8
  • CIRCLECLAR performs best overall across models and datasets when selecting whether to abstain, answer, or clarify.The comparison evaluates balanced accuracy; direct application of the decision rule offers little consistent improvement over Direct (Generic), while BAG is near the constant-policy baseline in most settings.Source: Figure 4b · p. 8, §5.2 · p. 9

6 Discussion

The discussion identifies limits in candidate discovery, correctness guarantees, and benchmark coverage. It argues that model-realizable ambiguity complements a world-centric view and can be represented through answer-equivalent clarification classes.

6.1 Limitations

  • CIRCLECLAR may miss answerable interpretations that candidate generation fails to produce, and its consistency check cannot distinguish confidently repeated factual errors from correct knowledge.Evaluation also depends on ambiguity annotations that may omit valid interpretations or include unsupported ones; AmbigQA-HQ mitigates but cannot guarantee complete coverage.Source: §6.1 · p. 9

6.2 A Model-Dependent View of Ambiguity

  • The paper frames ambiguity as model-dependent: an input may permit alternatives that a particular model cannot realize, and valid answers correspond one-to-one with equivalence classes of clarifications.The defined model-realizable support excludes interpretations a model recognizes but cannot answer, as well as linguistic ambiguity it detects without resolving.Source: §6.2 · p. 9, §6.1 · p. 9

A Proof of Proposition 4.1

Under D1–D3, valid clarification equivalence classes correspond one-to-one with answers, provided every answer has a valid clarification. The proof establishes that the map is well-defined, injective, and surjective.

A Proof of Proposition 4.1

  • D1 and D2 make each valid clarification’s unique answer a valid answer for the original input.D3 ensures that equivalent clarifications induce the same answer, so the map from clarification classes to answers is well-defined.Source: A Proof of Proposition 4.1 · p. 13
  • If two clarification classes map to the same answer, D3 makes their clarifications equivalent, proving injectivity.Thus the classes must be identical.Source: A Proof of Proposition 4.1 · p. 13
  • Every answer has a valid clarification by assumption, so each answer is the image of a clarification class and the map is surjective.Together, injectivity and surjectivity establish the bijection.Source: A Proof of Proposition 4.1 · p. 13

B Construction of AmbigQA-HQ

AmbigQA-HQ was constructed to provide a more reliable and complete reference answer set for evaluating model-realizable answers. The authors selected genuinely ambiguous questions and manually checked, repaired, and completed their rewrites and answers.

B Construction of AmbigQA-HQ

  • AmbigQA-HQ addresses reference-set errors that can change both answer membership and set size, including unresolved or unsupported rewrites and incorrect answers.The original annotations could also include rewrites that change the question or multiple rewrites yielding the same answer.Source: B Construction of AmbigQA-HQ · p. 13
  • AmbigQA-HQ contains 100 questions: 96 selected from AmbigQA and four newly constructed, with answers verified against authoritative sources.The newly constructed questions cover additional ambiguity patterns.Source: B Construction of AmbigQA-HQ · p. 13
  • The authors selected questions only when the original wording supports more than one answer, excluding rewrites that change what the question asks.Excluded rewrites could split a multi-part request or add an absent restriction.Source: B Construction of AmbigQA-HQ · p. 13
  • For each question, the authors checked rewrite fidelity and answer correctness, then removed, split, or merged rewrites and added missing plausible readings.They also made small wording changes where needed.Source: B Construction of AmbigQA-HQ · p. 13

C Additional Answer-Set Recovery Results

Additional results report macro precision and recall alongside macro set F1 and exact set match. CIRCLECLAR shows a more consistent advantage in precision than in recall.

C Additional Answer-Set Recovery Results

  • The additional metrics are macro precision and recall, complementing macro set F1 and exact set match.Source: §C.1 · p. 15
  • CIRCLECLAR shows a more consistent advantage in precision than in recall.Table 4 complements macro set F1 and exact set match with macro precision and recall.Source: §C.1 · p. 15
  • When estimated support is empty, precision is one only if reference support is also empty; otherwise it is zero.Recall is one whenever reference support is empty.Source: §C.1 · p. 15

C.2 Sensitivity Analysis

Sensitivity analyses vary candidate enumeration, recovery sampling, and the recovery threshold, while keeping model-specific reference sets fixed. Results differ by model and dataset, and performance on single-answer questions is also evaluated.

C.2 Sensitivity Analysis

  • The sensitivity analyses use defaults B = 6, R = 10, and τ = 0.6 unless a parameter is varied, with reference sets held fixed.Source: §C.2 · p. 15
  • Increasing B improves GPT-model performance on AmbigQA-HQ but declines for Qwen; on AmbigQA, larger B decreases performance for all except nearly constant GPT-5.5.The effect of R is generally smaller and non-monotonic; larger recovery budgets do not uniformly improve performance.Source: §C.2 · p. 15
  • Higher τ generally improves Qwen models, while GPT-5.5 declines on AmbigQA-HQ; the default threshold is τ = 0.6.Source: §C.2 · p. 15

C.3 Answer-Set Recovery on Single-Answer Questions

  • CIRCLECLAR achieves the highest set F1 for three of four models on 100 unambiguous AmbigQA questions; GPT-5.5 methods perform similarly, with Sampling-50 slightly strongest.Source: §C.3 · p. 15

D Content-Aware Evaluation of Set-Induced Response Policies

The evaluation compares response modes and answer content, using progressively stricter criteria for whether estimated answer supports match the reference. CIRCLECLAR is generally strongest on AmbigQA-HQ once answer identity is considered, while results on original AmbigQA are more mixed.

D Content-Aware Evaluation of Set-Induced Response Policies

  • The evaluation compares estimated and reference supports by whether they induce the same action: ABSTAIN for empty support, ANSWER for one answer, and CLARIFY for at least two.For content-aware evaluation, the same fixed estimates are rescored; direct policies are omitted because they do not output answer sets.Source: D Content-Aware Evaluation of… · p. 15
  • The four success criteria progress from matching the response action to exact recovery of the reference answer set.Intermediate criteria require the correct unique answer or at least two matched reference answers, then require every prediction to be reference-supported.Source: D Content-Aware Evaluation of… · pp. 15–17
  • Each criterion’s success rate is averaged unweighted across the empty-, singleton-, and multi-answer reference-support regimes.Figure 7 reports these means for the four methods.Source: D Content-Aware Evaluation of… · p. 17, Figure 7 · p. 18
  • CIRCLECLAR is generally strongest on AmbigQA-HQ under content-aware evaluation, while results on original AmbigQA are more mixed.Action-only rankings are mixed; predicting the correct response mode does not necessarily mean recovering the correct answers.Source: D Content-Aware Evaluation of… · p. 17, Figure 7 · p. 18

E Experimental Settings and Prompts

This appendix describes the experimental procedures and prompts for constructing and assessing model-specific answer sets in open-domain QA and executable Text-to-SQL. It also specifies SQL answer-equivalence checks and two direct response-policy baselines.

E Experimental Settings and Prompts

  • CIRCLECLAR uses τ = 0.6 across models and datasets; GPT-5.4-mini at temperature 0 groups outputs and judges equivalence, with stricter MT(x) thresholds of 0.9 for QA and 0.8 for Text-to-SQL.Evaluation matches semantically grouped predictions to model-specific reference groups and computes precision, recall, set F1, and exact set match.Source: E Experimental Settings and Prompts · p. 18

E.1.1 Prompt Templates

  • QA answer enumeration requests distinct canonical answers under reasonable readings, while prohibiting explanations, clarifying questions, aliases, and formatting variants.Table 6 summarizes QA generation budgets and set-construction rules; the supplied passage does not report its cell values.Source: §E.1.1 · p. 19
  • QA semantic clustering groups answers only when they mean the same answer, keeps differences in date, number, or specificity separate, and does not judge factual correctness or add answers.ICE uses Hou et al.’s [5] released AmbigQA few-shot prompts without modification.Source: §E.1.1 · p. 19
  • QA answer-conditioned clarification prompts request one non-identical question that makes the target answer unique under a reasonable interpretation without changing the answer type.The prompt also says not to include the target answer unless it already appears in the original question.Source: §E.1.1 · p. 20
  • Resolved-question QA generation returns one canonical standalone answer, while equivalence judgments match predictions to exactly one authoritative reference group or return NONE when unclear or ambiguous.Pairwise equivalence judgments count aliases for the same entity as equivalent, but reject merely related or conflicting answers.Source: §E.1.1 · pp. 20–21

E.2.1 Prompt Templates

  • Text-to-SQL enumeration requests queries for all identified reasonable interpretations, with read-only SELECT or WITH statements and no duplicate or formatting variants.Table 7 summarizes SQL generation budgets and set-construction rules; the supplied passage does not report its cell values.Source: §E.2 · p. 21, Table 7 · p. 22, §E.2.1 · p. 22
  • SQL outputs are grouped by interpretation and requested meaning, keeping A-only, B-only, and A+B results distinct when they represent separate readings.The prompt considers input, schema, SQL, and typed results together, and keeps outputs separate when equivalence remains uncertain.Source: §E.2.1 · p. 22
  • SQL clarification prompts ask for one non-cosmetic request that expresses the target query’s reasonable interpretation and rules out alternatives without mentioning SQL or changing the requested output type.The clarification must not be identical to the original request, and the prompt forbids outputting REJECT.Source: §E.2.1 · p. 23
  • SQL-generation prompts request one read-only SQLite query for a resolved request, with no explanation or extra columns.The prompt supplies the database dump and request and requires query-only output.Source: §E.2.1 · p. 23
  • The ICE clarification prompt asks for up to eight distinct, faithful, database-answerable rephrasings, or an empty list when the request is clear.It prohibits answering, writing SQL, seeking more information, or adding readings unsupported by the wording and database.Source: §E.2.1 · p. 24
  • SQL group judgments compare meaning against exactly one authoritative reference group, allowing lossless reshaping but rejecting changed interpretations or ambiguous matches.The prompt says to consider complete typed results and requested facts rather than SQL syntax or accidental result equality.Source: §E.2.1 · p. 24
  • Consistency checks preserve the source SQL’s requested facts and interpretation, allowing lossless derivations but marking missing or changed content not equivalent.They use deterministic relation checks as evidence, not as the final semantic decision, and permit an uncertain judgment when evidence is insufficient.Source: §E.2.1 · pp. 24–25

E.3 Direct Response-Policy Baselines

  • Both direct baselines receive only the unresolved question and are queried once at temperature T = 0; the generic baseline chooses among answer, clarify, and abstain.It answers when sufficiently clear and known, clarifies when the user can resolve ambiguity, and abstains when the answer is unknown.Source: §E.3 · p. 25
  • The model-dependent baseline clarifies only when it can reliably provide different answers for at least two plausible readings; it answers if only one is answerable and abstains if none are.Its decision depends on the distinct answers the model can provide, not ambiguity alone, and it assumes no search, tools, or reference evidence.Source: §E.3 · pp. 25–26
Loading 2610.20052v1…