The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering

1 Introduction

• The study tests whether grounding behaviors increase trust despite not necessarily making responses more faithful to dialogue and input rationale.The authors warn that this mismatch can lead to unjustified trust.Source: §1 · p. 1
• The authors analyze linguistic properties and model responses on TopiOCQA, comparing GPT-3 variants with retrieve-and-generate systems using automatic metrics and human evaluations.Human evaluation covers plausibility, groundedness, faithfulness, response preference, and perceived trust across controlled linguistic phenomena.Source: §1 · pp. 1–2
• GPT-3 variants are less faithful than DPR+FiD according to expert annotations and most automatic metrics.Users nevertheless prefer open-book GPT-3 over the fine-tuned model, even though half of the preferred responses are unfaithful.Source: §1 · p. 2
• Users trust responses with high lexical alignment significantly more, but prefer responses without pronouns and with short answers over fragments.The introduction also reports that elliptical responses are dis-preferred.Source: §1 · p. 2

2.1 Dataset and Models

• The study uses TopiOCQA’s development set of 205 dialogues and 2,514 turns, comparing open-book DPR+FiD and DPR+GPT-3 with closed-book GPT-3.GPT-3 was not tuned to resemble the target distribution, partly to keep the work focused and approximate end-user exposure to a model without direct prompt control.Source: §2.1 · p. 2

2.2 Dialogue Phenomena

• Lexical alignment is measured by unigram overlap with the question; syntactic forms include short single-sentence responses, declarative sentences, fragments, and rare multi-sentence answers.Pronouns are identified by their presence in subject or direct-object position.Source: §2.2 · pp. 2–3
• GPT-3 variants are more verbose and lexically aligned, while DPR+FiD produces shorter, more fragmented responses with more pronouns.The fine-tuned model more closely follows the human-response distribution; GPT-3 variants have limited conditioning on the target distribution through dialogue context in the prompt.Source: §2.2 · p. 3

3 Study of Faithfulness

• The paper defines faithfulness as entailment by a nonempty subset of knowledge together with a nonempty contextual explicature.The definition considers both dialogue history and knowledge.Source: §3 · p. 3

3.1 Automatic Evaluation

• K-F1++ measures unigram overlap with knowledge after removing response tokens that also appear in the question, discounting lexical alignment.The evaluation also used reference-based similarity metrics, knowledge-overlap metrics, and entailment metrics.Source: §3.1 · p. 3

3.2 Human evaluation studies

• Human evaluations separately judged response plausibility, grounding to the gold passage, faithfulness to dialogue and passage, and pairwise preference.The preference task sampled 250 faithful and unfaithful instances; annotators saw the dialogue and gold passage.Source: §3.2 · pp. 3–4

3.3 Results

• DPR+FiD outperformed GPT-3 variants on all reference-based metrics, while DPR+GPT-3 outperformed the baseline on most reference-less metrics.GPT-3 was evaluated zero-shot, whereas DPR+FiD was fine-tuned on TopiOCQA.Source: §3.3 · p. 4
• The K-F1 to K-F1++ absolute difference was 2.3% for DPR+FiD versus 5.8% and 4.5% for GPT-3 variants, which retained superior K-F1++ grounding scores.The authors suggest greater question alignment in GPT-3 responses explains their larger differences.Source: §3.3 · p. 4
• GPT-3 variants, with closed-book GPT-3 worst, produced less-grounded and more-unfaithful outputs than DPR+FiD in expert judgments.The faithfulness judgment task used 500 instances and two expert annotators.Source: §3.3 · p. 4, Table 3 · p. 4
• The authors leave fact-checking of extrinsic hallucinations—potentially true extra information that is not faithful to the input source—to future work.This bounds the study's treatment of extra information beyond the source.Source: §3.3 · p. 4
• DPR+GPT-3 was preferred 70% of the time, with 48% of those preferences for unfaithful responses; GPT-3 was preferred 45% of the time, with 66% unfaithful.Annotators could see the gold passage, and could also reject both options.Source: §3.3 · p. 4, Table 4 · p. 4

Human Evaluation Experiment

• The authors connect surface form to preference and warn that it may reduce users’ need to check sources, risking trust in an imperfect system.The trust experiment tests whether linguistic phenomena relate more generally to users’ trust judgments.Source: §4 · pp. 4–5
• The experiment isolated linguistic phenomena by modifying 15 response pairs per phenomenon and collecting 20 trust judgments per pair.Annotators saw the dialogue and chose the response they trusted more, or neither.Source: §4 · p. 5

Results

• Users trusted responses with high lexical alignment significantly more than low-alignment responses, and preferred short answers over fragments and responses without pronouns.The authors note this differs from human-to-human dialogue findings and may reflect users seeking stronger evidence from a system.Source: §4 · p. 5, Table 5 · p. 5

• GPT-3 variants were less faithful than DPR+FiD by automatic metrics, a pattern confirmed by faithfulness annotators.Source: §5 · p. 5
• Users showed a significant preference for more conversational GPT-3 responses, even when those responses were unfaithful.Source: §5 · p. 5
• A separate experiment found that high lexical alignment increased users’ trust.Source: §5 · p. 5

• The analysis covers lexical alignment, ellipsis, and pronouns, omitting other grounding phenomena such as repairs.Source: Limitations · p. 5
• Lexical alignment was calculated from only the user question and system response, leaving multi-turn convergence unmeasured.The authors note this may reflect the OCQA domain or a bias in TopiOCQA.Source: Limitations · p. 5
• Crowdsourcing experiments were diagnostic on a given dataset and did not represent more organic real-use cases.The authors identify whole-dialogue extrinsic evaluation as an ideal but more costly alternative.Source: Limitations · p. 5

• Fluent but factually incorrect, lexically aligned text could be more convincing than more faithful text, creating a possible fake-news misuse scenario.Source: Ethics Statement · p. 5
• Crowd workers could complete at most 40 HiTs and received $0.29 per preference-task HiT or $0.20 per trust-study HiT.Source: Ethics Statement · p. 5
• The trust study involved 38 annotators and the preference task 115, from five named countries; AMT data were anonymized to provider specifications.Source: Ethics Statement · p. 6
• The authors obtained TopiOCQA from the public domain, did not intend to redistribute it without its authors’ consent, and assumed precautions against offensive content.Source: Ethics Statement · p. 6

• Only 27 of 558 GPT-3-preferred responses cited the document (4%), versus 76 of 359 baseline-preferred responses (21%).Across all feedback, 286 of 2,170 responses explicitly mentioned the grounded document.Source: A Additional Human Evaluation Results · p. 8
• Five and 20 annotations per item were used to calculate majority agreement in the preference and trust studies, respectively.Source: A Additional Human Evaluation Results · p. 8
• Some items had quite low annotator agreement, which the authors identify as an avenue for future work.Source: A Additional Human Evaluation Results · p. 8
• Feedback suggests GPT-3 responses were often preferred for the amount and variety of information and conversational style.Source: A Additional Human Evaluation Results · p. 8

B.1 Faithfulness Judgment Task

• Annotators assess response quality by how well it represents the provided sources, while watching for fluent answers containing slight inaccuracies.The instructions warn annotators to read carefully, as if proofreading.Source: §B.1 · p. 8

B.2 Preference Task

• Figure 4 presents the preference interface, where annotators compare two answers in the conversation with a supporting document available.They may choose either answer, mark both high- or low-quality, and optionally explain their choice.Source: §B.2 · p. 8, Figure 4 · p. 10
• Preference judgments ask which answer is better, with instructions to consider the conversation and explain which parts of the question and answers mattered.Annotators may select both answers as high quality or both as low quality when neither is preferable.Source: §B.2 · p. 8

B.3 Study of Trust

• The trust task asks annotators to choose the more trustworthy of two responses using the dialogue only; all responses are stated to be correct.Annotators can indicate that they cannot decide and optionally explain why.Source: §B.3 · p. 8, Figure 5 · p. 11

B.1 Faithfulness Judgment Task

• Faithfulness judgments begin with plausibility; implausible responses end the annotation before further questions are asked.The interface then provides the source document for annotators to assess faithfulness.Source: §B.1 · p. 8, Figure 2 · p. 9, Figure 3 · p. 9
• Figures 2 and 3 show the staged faithfulness interface: plausibility is judged before the document becomes available for annotation.The plausibility screen stops the task when a response is not plausible.Source: Figure 2 · p. 9, Figure 3 · p. 9