Do we need to answer that question? Salience and Answerability of Potential Questions in Naturalistic Dialogue

1 Introduction

• Naturalistic data complicates QUD operationalisation because relevant questions arise at multiple levels and interactional phenomena make the set of QUDs or potential questions difficult to identify.These phenomena include turn-taking, speaker intentions, and shared situational context.Source: §1 · p. 1
• Building on Wu et al. (2024), the study tests whether the salience–resolution relationship extends to naturalistic dialogue across interactional settings.The authors apply the prior methodology to dialogues with varying interactional organisation.Source: §1 · p. 1
• The authors release a corpus of 7,124 generated questions annotated for salience and answerability by two annotators.The corpus is based on English dialogue and supports investigation of the QUD/PQ approach in conversation.Source: §1 · p. 1

2 Background

• The QUD framework represents discourse as evolving questions whose possible answers define relevant contributions, and potential questions can model candidate issues that may become active as discourse unfolds.QUDs have been used retrospectively to reconstruct discourse and predictively to model expectations.Source: §2 · p. 2, §2.1 · p. 2

2.1 Questions and Discourse structure

• Prior dialogue research includes controlled tasks and relatively structured interactions, leaving the framework’s generalisation to less organised conversation uncertain.The study addresses this gap with a diverse set of naturalistic dialogues.Source: §2.1 · p. 2
• Wu et al. (2024) generated and annotated potential questions in news and TED talks, finding positive salience–answerability correlations; whether the relationship holds in dialogue remains unclear.Their reported correlations were ρ = .59 for news and ρ = .74 for TED talks.Source: §2.1 · p. 2

2.2 The Multifaceted Nature of Dialogue

• Dialogue varies in participant number and organisation, and situated context, turn-taking, and participation management broaden potential questions and may reduce turn predictability.Duologues tend to have more stable trajectories than multilogues, which add turn-management and discourse-development complexity.Source: §2.2 · pp. 2–3

3 Source Corpus: Naturalistic dialogues

• The corpus samples 60 spoken BNC dialogues across six categories organised around participant number and expected interactional organisation.The categories distinguish spontaneous, organised, and public discussions.Source: §3 · p. 3, Table 1 · p. 4
• Spontaneous conversations are generally shorter and have briefer turns than more structured interactions.More structured interactions include longer explanations or arguments.Source: §3 · p. 3
• Transcript imperfections include unknown-speaker turns and about 2% unclear-token markers, which sometimes complicate annotation.The corpus also contains occasional speaker attribution errors.Source: §3 · pp. 3–4

4 Method: Automatic question generation and human annotation

• For each anchor utterance, the method generates potential questions from preceding dialogue context and correlates their salience with how much they are answered later.The pipeline comprises question generation, manual annotation, and correlation analysis.Source: §4 · p. 4

4.1 Automatic question generation

• The authors generate three questions per selected anchor and choose Mistral for subsequent experiments after a small pilot found its questions more salient on average and more often valid than Gemma’s.Llama was dismissed after generating blatantly sexist questions; the authors caution that the pilot was too small for strong conclusions about model choice.Source: §4.1 · pp. 4–5, Figure 3

4.2 Guidelines adaptations for dialogue

• Dialogue-specific guidelines define answerability on a three-point scale and retain the salience scale while adding categories for transcription issues, shared knowledge, and question relevance.Answerability measures how much a question is addressed after the anchor, without separately scoring questions already answered in preceding context.Source: §4.2 · p. 5

4.3 Annotations

• Two annotators independently rate each question, agreement is assessed with multiple kappa and alpha metrics, and salience analyses retain both perspectives alongside an agreement-only subset.Answerability scores are averaged to produce a finer-grained estimate.Source: §4.3 · pp. 5–6

• 7,124 Mistral-7B-generated questions are released with salience and answerability annotations.Source: §5 · p. 6
• Each corpus row records its dialogue, anchor turn, question, annotator scores, agreed gold scores, and comments.Source: §5 · p. 6
• Because the BNC license prevents including dialogues, metadata enables their reconstruction from the BNC.Source: §5 · p. 6

6.1 Salience: Which questions should be answered?

• Salience agreement is fair to moderate and lower than Wu et al. (2024), with transcription artefacts and missing non-linguistic context contributing to disagreements.Source: §6.1 · p. 6
• Radio conversations have the highest inter-annotator agreement, while public television has the lowest, where annotators lacked the audience’s visual context.Most dialogue-group pairs differ significantly in agreement.Source: §6.1 · p. 7, Figure 6 · p. 8
• Salience agreement is highest for Scores 2 and 7, which mark transcription issues and close paraphrases, and lowest for Scores 1 and 8, involving contextual relatedness and whether a question must be answered.The latter judgments rely on discourse coherence, pragmatics, shared context, and intentions that transcripts do not reveal.Source: §6.1 · p. 7, Figure 5 · p. 8
• Annotators often agree that questions are at least loosely connected to prior context, but judgments about whether they are essential vary more.The latter can require fine-grained dialogue understanding and non-linguistic context.Source: §6.1 · pp. 7–8

6.2 Answerability: Which questions are answered?

• Answerability agreement is higher than salience agreement, and annotators agree most on Score 0, identifying questions left unanswered.Distinguishing partial from full resolution is ambiguous, especially for complex or implicit questions.Source: §6.2 · p. 9
• Answerability varies less across dialogue types than salience, although open-domain multilogues differ from all other types; low average scores indicate most questions go unanswered.Source: §6.2 · p. 9, §6.1 · p. 9

6.3 Dialogue predictability

• Salience and answerability correlate positively across all settings, but the association is two to three times weaker than for news and TED talks (ρ = 0.59–0.74).The authors note that testing other question-generation models would help validate this finding.Source: §6.3 · p. 9, Figure 7 · p. 10
• Answerability is relatively homogeneous below salience 6, while salience levels 6–8 are associated with significantly higher answerability; Score 2 has lower answerability across conditions.Most Score 2 questions ask for transcription clarification and are unlikely to be answered in dialogue.Source: §6.3 · p. 10

7 Conclusion

• The study adapts Wu et al. (2024) to 60 British National Corpus dialogues spanning a range of interactional settings.It investigates whether salience predicts later question resolution in dialogue.Source: §7 · p. 10
• Salience is positively related to answerability, but the relationship is substantially weaker than in news articles and TED talks and weaker still in open-domain multilogues.The authors suggest dialogue is less predictable than monologic discourse, and that disagreement may reflect challenges specific to dialogue types.Source: §7 · p. 10
• The work offers one of the first large-scale operationalisations of QUD-driven approaches to modelling naturalistic dialogue, while moderate salience-annotation agreement underscores task complexity.The authors argue that annotator perspective should be considered when studying dialogue structure.Source: §7 · p. 10
• Future work should examine when questions are answered and by whom, and develop automatic models for predicting dialogue salience.Source: §7 · p. 10

Limitations

• The study covers English spoken British National Corpus dialogues collected in the early 1990s, and some transcripts contain missing, unclear, or misattributed speech.The data reflects a specific linguistic, cultural, and technological context; transcript imperfections sometimes complicated question generation and annotation.Source: Limitations · p. 11
• Potential questions were generated with a single LLM, so the findings support one plausible operationalisation rather than an exhaustive account of dialogue QUDs.Different models may generate different questions from the same context, and human agreement on QUD generation or inference is only moderate.Source: Limitations · p. 11
• Annotators saw transcripts with limited contextual information and had to reconstruct participants’ assumptions, intentions, and shared knowledge.The authors report that these reconstructions can vary substantially across observers; a more diverse annotator pool may reveal greater variation.Source: Limitations · p. 11

Ethical Considerations

• Generative AI was used only to improve linguistic quality, not to generate scientific content, formulate hypotheses, design experiments, or interpret results.References were selected and added manually.Source: Ethical Considerations · p. 11

A Corpus selection

• Corpus selection constrained conversation type, participant count, and length, with duologues limited to two participants and multilogues to five or more.Some dialogues were manually cut at a topic boundary when suitable shorter examples were unavailable.Source: A Corpus selection · p. 13

B Prompt for question generation

• The adapted prompt asks for three questions about the last utterance that a dialogue participant would be curious about and cannot answer from the prior context.It treats special tokens as contextual information rather than part of the utterance itself.Source: B Prompt for question generation · p. 13

C Annotations

• The annotations appendix reports annotator and gold average scores, support counts, and pairwise significance tests, including analyses of salience and answerability.Source: C Annotations · p. 14

C.1 Grades

• Table A2 reports average salience scores and gold-support counts, while Figure A9 identifies pairs of groups with significant distribution differences.Figure A9a shows both annotators’ results, with each triangle to be analysed separately.Source: §C.1 · p. 14
• Gold annotations show no significant salience differences between dialogue groups, but the restricted gold subset limits the tests’ statistical power.The authors nevertheless note that salience appears sensitive to dialogue type and participant count, with duologues showing more salient questions than multilogues.Source: §C.1 · p. 14
• Most groups differ significantly in salience distributions; open-domain duologues and multilogues are similar, while public radio multilogues resemble organised duologues.Public television multilogues resemble public duologues, though the statistical strength of these patterns varies by annotator.Source: §C.1 · p. 14

C.1.1 Answerability.

• Most questions are not explicitly answered in the subsequent dialogue, as average answerability scores are low across all groups.Source: §C.1.1 · p. 14
• Answerability distributions are more stable across dialogue categories than salience distributions.Source: §C.1.1 · p. 14

C.2 Dialogue predictability

• Answerability is relatively stable below salience score 6, except score 2, while scores 6–8 have significantly higher answerability.The differences among scores 6–8 are significant except between scores 7 and 8 for annotator 1.Source: §C.2 · p. 16, Figure A11 · p. 16, Table A3 · p. 16
• Figure A12 reports mean answerability by salience level and regression lines across the full 0–8 scale; correlations are slightly higher than on the reduced scale [1, 3, 5, 6, 8].Source: §C.2 · p. 16, Figure A12 · p. 17
• Both annotators rank dialogue groups similarly by ρ, with open-domain duologues and public television multilogues swapping second and fifth place.Source: §C.2 · p. 16, Table A4 · p. 17
• Gold-annotation rankings differ from annotator rankings, preserving only the top two groups: organised duologues and public duologues.Source: §C.2 · p. 16, Table A4 · p. 17
• Most group differences in salience–answerability correlation are not significant; open-domain multilogues have the lowest ρ, while public television multilogues differ significantly from most groups.Open-domain duologues and organised multilogues are the groups most similar to public television multilogues; differences are even fewer for gold annotations.Source: §C.2 · p. 16, Figure A13 · p. 17, Table A4 · p. 17