PaperlayerView exact revision on arXiv

Source-linked AI summary

Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions

Alberto Testoni, Raquel Fernández

arXiv:2402.06509v1cs.CLcs.AI

TL;DR

Dialogue systems struggle to ask effective clarification questions, and it is unclear whether human clarification behavior reflects model uncertainty. Using CoDraw, the paper compares these uncertainties and proposes model-uncertainty-triggered clarification, which outperforms several baselines on task success.

  • Problem

    The relationship between human clarification behavior and model uncertainty is under-explored, while dialogue systems struggle to generate effective clarification questions.

    Source: §1 · p. 1, §7 · p. 9

  • Method

    The paper studies human–model uncertainty in CoDraw and uses a thresholded model-uncertainty module with template-based clarification generation.

    Source: §1 · p. 2

  • Results

    The proposed approach significantly outperforms several strong baselines, including one based on human clarification supervision, on task success.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings highlight the importance of enabling conversational agents to assess and act on their own uncertainty through clarification questions.

    Source: §1 · p. 2, §7 · p. 9

    The controlled setup focuses on the size attribute and template-based generation, so further work is needed in more complex tasks and model architectures.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Clarification questions are an essential dialogue tool to signal misunderstanding, ambiguities, and under-specification in language use. While humans are able to resolve uncertainty by asking questions since childhood, modern dialogue systems struggle to generate effective questions. To make progress in this direction, in this work we take a collaborative dialogue task as a testbed and study how model uncertainty relates to human uncertainty -- an as yet under-explored problem. We show that model uncertainty does not mirror human clarification-seeking behavior, which suggests that using human clarification questions as supervision for deciding when to ask may not be the most effective way to resolve model uncertainty. To address this issue, we propose an approach to generating clarification questions based on model uncertainty estimation, compare it to several alternatives, and show that it leads to significant improvements in terms of task success. Our findings highlight the importance of equipping dialogue systems with the ability to assess their own uncertainty and exploit in interaction.

1 Introduction

This paper studies when collaborative dialogue agents should ask clarification questions by comparing human and model uncertainty. It proposes uncertainty-driven clarification and reports higher task success than strong baselines.

  • Human questioning supports conversational grounding, cognitive development, and language learning, motivating clarification as a central capability for interactive agents.Source: §1 · p. 1
  • Clarification questions help dialogue participants address ambiguity and underspecification, but generating effective questions remains difficult for modern neural dialogue systems.Source: §1 · p. 1
  • Model uncertainty does not closely correspond to human clarification-seeking behavior, weakening human clarification decisions as supervision for resolving model uncertainty.The paper directly compares the two approaches in CoDraw, an under-explored relationship in prior work.Source: §1 · p. 2
  • The proposed Drawer uses model uncertainty to trigger template-based clarification questions, with a threshold controlling both uncertainty sensitivity and questions per dialogue.The question-answer pair is added to the dialogue history before the next drawing action.Source: §1 · p. 2, Figure 1 · p. 2
  • The uncertainty-based model is significantly more effective for task success than several strong baselines, including a model trained on human clarification supervision.Source: §1 · p. 2

2 Related Work

Prior work studies clarification through human dialogue annotations, learned decision modules, and model-uncertainty signals. This paper positions its contribution as a direct comparison between human and model uncertainty in collaborative dialogue.

  • Clarification research has developed annotated conversation datasets and schemes for identifying the form and function of clarification requests.Resources include CoDraw-iCR, TEACh, and annotated Minecraft dialogue datasets.Source: §2 · p. 3
  • Madureira and Schlangen (2023b) use CoDraw-iCR annotations to train a neural model for deciding when to clarify, whereas this paper uses them to study human–model uncertainty relations.Source: §2 · p. 3
  • Existing systems either train external models to decide when to ask or use model uncertainty, such as action entropy or probability differences, to trigger clarification.Source: §2 · p. 3

3 Task & Dataset

CoDraw is a collaborative instruction-following task in which a Teller describes a target clipart scene and a Drawer reconstructs it. The dataset includes roughly 10,000 dialogues, and success is measured by image similarity.

  • The Teller describes an abstract target scene, while the Drawer selects and places cliparts from a gallery on a canvas.Source: §3 · p. 3
  • CoDraw contains around 10k dialogues averaging 7.7 turns, with target scenes containing approximately six cliparts on average.Source: §3 · pp. 3–4
  • The Drawer can remove, resize, or flip objects and typically responds with confirmation feedback or follow-up questions before the next turn.Source: §3 · p. 4
  • Task success is measured with a 0–5 similarity score comparing the target image with the Drawer’s final reconstruction across cliparts and their attributes.Source: §3 · p. 4

4 Computing Uncertainty in CoDraw

The Drawer estimates uncertainty over clipart selection and attributes before acting. The paper uses score-based, entropy-based, and ensemble-variance measures for different prediction types.

  • The baseline Drawer assigns scores to 58 available cliparts and predicts their attributes from the Teller’s latest message and the current canvas state.At inference, cliparts with scores above 0 are drawn.Source: §4.1 · p. 4
  • The baseline model reaches an average CoDraw similarity score of 3.31 (sd=0.67) on turn-by-turn test-set instructions.Source: §4.1 · p. 4
  • For clipart selection, lower assigned scores indicate greater uncertainty about selecting a given clipart.Source: §4.2 · p. 4
  • For size and orientation, entropy over the discrete attribute-label distributions measures uncertainty, with higher entropy indicating greater uncertainty.Source: §4.2 · p. 4
  • For position, variance across five Drawer models trained with different random seeds represents uncertainty in continuous x- and y-coordinate predictions.Source: §4.2 · p. 4

5 Model vs. Human Uncertainty

Using CoDraw annotations, the study tests whether Drawer-model uncertainty predicts when human players ask clarification questions. Model uncertainty has limited predictive power and remains poorly aligned with human clarification behavior despite improving task performance and calibration.

Human uncertainty

  • Around 40% of CoDraw dialogues contain at least one clarification question, averaging 2.2 requests within that subset.Human clarification behavior is measured from Drawer turns annotated as clarification requests and their mentioned attributes.Source: §5 · pp. 4–5

Method

  • Model uncertainty is evaluated by logistic regression over clipart selection, size, orientation, and position uncertainties to predict clarification decisions in the next human follow-up turn.The study uses clarification questions as a signal of human uncertainty because eye gaze and reaction-time cues are unavailable in CoDraw.Source: §5 · p. 4, §5.1 · p. 5

Results

  • 0.188 AP versus a 0.142 random baseline (p < 0.05) shows model uncertainty predicts human clarification questions above chance, but only weakly.The result averages five initialization seeds at epoch 15; performance remains low in absolute terms.Source: §5.2 · p. 5
  • As the Drawer trains, similarity score rises and uncertainty falls, but Average Precision stays low across epochs, so improved task performance does not improve prediction of human clarification behavior.The same lack of predictive power appears when epistemic and aleatoric uncertainty are estimated with model ensembles.Source: §5.2 · p. 5, Figure 2 · p. 5

Analysis

  • Human players have more knowledge and resources than the relatively simple Drawer, while speaker-specific clarification strategies introduce variability that may be difficult to predict.An exploratory analysis of underspecified first instructions further examines this variability in the CoDraw dataset.Source: §5.3 · pp. 5–6

6 Asking Clarification Questions

QDrawer asks template-based clarification questions when uncertainty about clipart size exceeds a threshold, targeting the most uncertain objects. Across CoDraw experiments, uncertainty-driven questioning outperforms human-timing, Decider, and random-turn alternatives, while improving task success and uncertainty calibration.

6.1 The Uncertainty-based QDrawer

  • QDrawer asks a size question when entropy over small, medium, and large labels exceeds a threshold, targeting up to two highest-entropy cliparts.Questions use fixed templates, and ground-truth annotations provide the Teller’s answers before the next drawing action.Source: §6.1 · p. 7

6.2 Baselines

  • Human-timed questions only marginally improve size accuracy and do not significantly improve similarity, whereas the external Decider reaches 80.1% accuracy and 3.34 similarity.The Decider selects turns based on human clarification behavior, while size uncertainty selects which cliparts to target.Source: §6.3 · pp. 7–8, §6.2 · p. 7

6.3 Task Success Results

  • Up to 87.3% size accuracy and 3.40 similarity score make uncertainty-driven QDrawer outperform the Silent Drawer and alternative clarification strategies.The Silent Drawer reaches 76.9% accuracy and 3.31 similarity, while the Decider reaches up to 80.1% and 3.34.Source: §6.3 · pp. 7–8, Table 1 · p. 7

6.4 Clarification Questions’ Contribution

  • Clarification exchanges add +11% absolute size accuracy and +0.09 similarity for uncertainty-driven QDrawer dialogues containing at least one question.The corresponding similarity increase is not significant when questions are asked at the same positions as human players.Source: §6.4 · p. 8
  • QDrawer’s uncertainty estimates are better calibrated than Silent Drawer estimates, reducing ECE from 0.10 to 0.05 and Brier score from 0.36 to 0.28.Both improvements are statistically significant with p < 0.001.Source: §6.4 · p. 8

6.5 How Many Questions to Ask

  • Using model uncertainty improves accuracy even when only a few questions are asked, outperforming random-turn questioning and the Decider under comparable question counts.Lowering the entropy threshold consistently improves QDrawer performance, allowing the question rate to be controlled.Source: §6.3 · p. 8, §6.5 · p. 9

7 Conclusions

The paper finds that human and model uncertainty are poorly correlated, then shows that model-uncertainty-driven clarification questions outperform baselines and human-decision-based methods in CoDraw.

  • Human and model uncertainty are poorly correlated, possibly because human clarification strategies vary widely.Source: §7 · p. 9
  • The proposed method extracts model uncertainty, generates clarification questions, and varies the uncertainty threshold to control question frequency.Source: §7 · p. 9
  • The proposed approach outperforms several baselines and methods trained on human clarification decisions.Source: §7 · p. 9
  • The findings support further research on integrating model uncertainty into natural language generation to address ambiguity and underspecification.Source: §7 · p. 9

Limitations

The experiments isolate clarification effects by focusing on size questions with template-based generation, but this controlled setup limits claims about more complex tasks, architectures, costs, and dialogue naturalness.

  • The controlled setup focuses on the size attribute and template-based generation, so confirmation in more complex tasks and model architectures remains necessary.Source: Limitations · p. 9
  • Focusing on size ensures answer correctness, whereas other attributes such as position have less direct mappings from language to space coordinates.Source: Limitations · pp. 9–10
  • Template-based question generation may not scale to settings where visual grounding and question-answer evaluation are more complex.Source: Limitations · p. 10
  • The study controls clarification cost only by limiting the number of questions and does not analyze effects on perceived dialogue naturalness.Source: Limitations · p. 10

A Similarity Score (v2)

The revised CoDraw similarity score changes component weighting and attribute handling to better match the properties analyzed in the experiments.

B Model vs. Human Uncertainty

The analysis evaluates whether model uncertainty predicts human clarification decisions in CoDraw and finds limited predictive value, including for ensemble-based uncertainty measures.

D The CoDraw Dataset and Models

The CoDraw dataset is publicly available, anonymized, and distributed under a Creative Commons Attribution-NonCommercial 4.0 license, alongside released Silent Drawer models.

E QDrawer Details

QDrawer shares the Silent Drawer architecture while modifying its loss to address clarification exchanges on previously introduced cliparts. The evaluation reports logistic-regression and random-baseline comparisons across average precision, F1, uncertainty correlations, and dialogue examples.

QDrawer Details

  • QDrawer has the Silent Drawer architecture, 3,347,390 parameters, and a 15-epoch training configuration following the original paper.Source: E QDrawer Details · p. 14
  • The modified loss function enables QDrawer to properly address clarification exchanges about cliparts added in previous dialogue turns.Source: E QDrawer Details · pp. 14–15

Evaluation

  • Table 3 compares logistic-regression average precision against a random baseline across initialization seeds and training epochs, alongside final CoDraw similarity scores.Source: Table 3 · p. 16
  • Table 4 reports F1 scores for the logistic-regression model and a random baseline.Source: Table 4 · p. 16
  • Table 5 reports uncertainty-type correlation coefficients for configurations defined by initialization seed and training epoch.Source: Table 5 · p. 16

Qualitative Examples

  • Figure 5 presents dialogue examples and clarification questions from Teller, Silent Drawer, and QDrawer, highlighting clarification exchanges in green.Source: Figure 5 · p. 18

Related papers

Loading 2402.06509v1…