Source-linked AI summary
Evaluating Verifiability in Generative Search Engines
Nelson F. Liu, Tianyi Zhang, Percy Liang
TL;DR
Generative search engines need verifiable citations so users can check generated statements, but existing systems may appear trustworthy while containing unsupported claims and inaccurate citations. Using human evaluation of four commercial engines, the paper finds low citation support and precision, while distinguishing verifiability from factuality.
Problem
Trustworthy generative search engines should cite statements comprehensively and accurately, motivating evaluation of citation recall and precision.
Method
The paper uses human evaluation to audit the verifiability of Bing Chat, NeevaAI, perplexity.ai, and YouChat, measuring fluency, perceived utility, and citation support.
Results
Only 51.5% of generated statements are fully supported by citations, and 74.5% of citations support their associated statements, despite generally fluent and informative responses.
Takeaways & Limitations
Low citation recall and precision, combined with fluent and helpful-seeming responses, increase existing systems’ potential to mislead users.
Takeaways & Limitations
The evaluation assesses verifiability rather than factuality, so it does not determine whether generated statements are true.
Abstract
from arXiv · showhide
Generative search engines directly generate responses to user queries, along with in-line citations. A prerequisite trait of a trustworthy generative search engine is verifiability, i.e., systems should cite comprehensively (high citation recall; all statements are fully supported by citations) and accurately (high citation precision; every cite supports its associated statement). We conduct human evaluation to audit four popular generative search engines -- Bing Chat, NeevaAI, perplexity.ai, and YouChat -- across a diverse set of queries from a variety of sources (e.g., historical Google user queries, dynamically-collected open-ended questions on Reddit, etc.). We find that responses from existing generative search engines are fluent and appear informative, but frequently contain unsupported statements and inaccurate citations: on average, a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence. We believe that these results are concerningly low for systems that may serve as a primary tool for information-seeking users, especially given their facade of trustworthiness. We hope that our results further motivate the development of trustworthy generative search engines and help researchers and users better understand the shortcomings of existing commercial systems.
1 Introduction
Generative search engines are rapidly attracting users by producing tailored answers with inline citations, but their trustworthiness depends on whether those citations comprehensively and correctly support the generated statements. A human audit of four commercial systems finds fluent, seemingly useful responses that often contain unsupported statements and inaccurate citations.
- Generative search engines directly generate tailored responses with inline citations, unlike conventional search engines that retrieve pre-existing webpages.
- Verifiability requires every external-world statement to be fully supported by citations and every citation to support its associated statement.These requirements correspond to citation recall and citation precision.
- The audit evaluates Bing Chat, NeevaAI, perplexity.ai, and YouChat using human judgments across diverse information-seeking queries.The query sources include historical Google user queries and dynamically collected Reddit questions.
- 51.5% of generated sentences are fully supported by citations, while 74.5% of citations support their associated sentence.These averages indicate low citation recall and citation precision despite generally high fluency and perceived utility.
- The study links fluent, seemingly useful responses with a facade of trustworthiness that may increase their potential to mislead users.The authors report an inverse relationship between perceived utility and citation precision, and hypothesize that copying or close paraphrase contributes to it.
2 Human Evaluation of Fluency, Perceived Utility, and Verifiability
The evaluation formalizes fluency, perceived utility, citation recall, and citation precision for assessing generative search responses and their verifiability. It treats support of external-world statements and accuracy of associated citations as complementary requirements.
- The framework evaluates fluency, perceived utility, citation recall, and citation precision, with high recall and precision indicating comprehensive and accurate citations.
- Fluency and perceived utility: The protocol measures fluency and perceived utility with five-point Likert judgments about cohesion and whether the answer is helpful and informative.
- Citation recall: Citation recall is the proportion of verification-worthy statements fully supported by their associated citations.Annotators identify verification-worthy statements and judge whether their citations fully support each one.
- Citation precision: Partially supporting citations count toward precision when their associated statement is fully supported by the union of citations and no citation fully supports it alone.This rule allows multiple partial citations to jointly support a statement.
- Citation precision: Citation precision is the proportion of generated citations that support their associated statements.The metric rewards accurate citation rather than indiscriminate citation, which can yield high recall but low precision.
- Citation F1: Citation F1 combines citation precision and citation recall using their harmonic mean, so high F1 requires both metrics to be high.
3 Evaluation Setup
The study evaluates four commercial generative search engines with a human annotation protocol across diverse query distributions and answer types. Differences in abstention behavior can complicate direct system comparisons.
- Evaluated generative search engines: The evaluation covers Bing Chat, NeevaAI, perplexity.ai, and YouChat, which generate responses by conditioning language models on queries and retrieved content.The study saves each system’s first complete single-turn response.
- Abstention behavior: Differing abstention rates make direct performance comparisons difficult, although the study does not find that higher abstention produces higher evaluation performance in practice.
- Abstention behavior: 22.7% of NeevaAI’s 1450 queries receive no response, whereas Bing Chat and YouChat generate responses for nearly every input query.NeevaAI’s response is designed for display within a conventional search-results page, unlike the conversational interfaces of Bing Chat and YouChat.
- Query distributions: Each system is evaluated on 1450 queries across diverse information needs and answer formats.
- Query distributions: The query set spans 12 distributions from sources including historical Google queries, Reddit questions, essay prompts, debates, and how-to queries.These distributions require short spans, long-form paragraphs, lists, or tables.
- Human evaluation protocol: Human annotators assess fluency, perceived utility, citation recall, and citation precision through a three-step annotation process.Annotators were recruited and trained through a qualification study, and agreement exceeded 82.0% pairwise agreement and 91.0 F1 for all judgments.
4 Results and Analysis
Human evaluation finds that existing generative search engines are fluent and appear useful, but often fail to cite comprehensively or accurately. Citation precision is inversely related to perceived utility, partly because systems copy or closely paraphrase cited webpages even when those passages are irrelevant to the query.
- Fluency and Perceived Utility: Responses averaged 4.48 for fluency and 4.50 for perceived utility, indicating that annotators generally found them fluent and helpful.Ratings were averaged across all systems and responses.
- Citation Recall and Precision: 51.5% of generated statements are fully supported by citations, while 74.5% of citations support their associated statements.These averages indicate low citation recall and precision across the evaluated systems.
- Citation Recall and Precision: Perplexity.ai achieved the highest average citation recall at 68.7, whereas Bing Chat achieved the highest average citation precision at 89.5.Recall and precision varied substantially across the four evaluated systems.
- Citation Recall and Precision: Citation recall was 58.5 for NaturalQuestions queries with long answers versus 47.8 for non-NaturalQuestions queries.Query distribution affected citation recall more than citation precision.
- Citation Precision and Perceived Utility: Citation precision and perceived utility were strongly inversely correlated (r = −0.96), with Bing Chat highest in precision but lowest in perceived utility.YouChat showed the opposite pattern, with lowest citation precision and highest perceived utility ratings.
- Citation Precision and Perceived Utility: Systems often copied or closely paraphrased cited webpages, increasing citation precision while lowering perceived utility when the copied material was irrelevant to the query.This pattern reflects a trade-off between faithfulness and abstractiveness.
5 Related Work
Related work develops methods and evaluation frameworks for generating and assessing supported answers. Contemporaneous evaluation also reports that ChatGPT and YouChat responses are coherent but frequently undersourced and inaccurate.
- Methods for Supported Generation: Prior systems use reinforcement learning from human preferences to answer questions while providing supporting evidence.One approach conditions responses on evidence retrieved from a separate system.
- Evaluation Frameworks: The attributed to identified sources framework evaluates whether statements are supported by provided evidence across several generation tasks.Attributed question answering additionally requires answers to point to supporting evidence text.
- Contemporaneous Evaluation: Domain experts found ChatGPT and YouChat responses coherent and concise but frequently undersourced and inaccurate.The paper reports a similar lack of citations for generated statements in YouChat responses.
6 Conclusion
The study audits the verifiability of four commercial generative search engines and finds fluent, seemingly informative responses alongside frequent unsupported statements and inaccurate citations. Citation precision is inversely correlated with perceived utility, suggesting that apparently more helpful responses often have less accurate citations.
- Human evaluation audited Bing Chat, NeevaAI, perplexity.ai, and YouChat for verifiability.
- 51.5% of generated statements were fully supported by citations, while 74.5% of citations supported their associated statements.
- Citation precision was inversely correlated with perceived utility, so responses that seemed more helpful often had more unsupported statements or inaccurate citations.
- Analysis suggests that copying or closely paraphrasing cited webpages inflates citation precision while reducing perceived utility when copied content is irrelevant.
Limitations
The paper evaluates verifiability rather than factuality, enabling users to inspect sources without determining whether generated statements are true. It also evaluates claims at the sentence level because no clear linguistic definition of a claim is available.
- The study assesses verifiability by checking citation support, not whether generated statements are factually true.Verifiability helps users inspect sources and draw their own conclusions about trust.
- Users may not check sources, so evaluating factuality remains an important direction for future work.
- The evaluation uses sentence-level claims because sentences can contain multiple claims and no clear linguistic definition of a claim currently exists.The authors identify finer-grained claim evaluation as future work.
A Example queries from each evaluated query distribution
The evaluation uses diverse query distributions and answer formats, then applies a multi-step annotation process covering response quality, verification-worthiness, citation support, and annotation guidelines. The appendix documents both query construction and the human-evaluation interface.
- Example queries from each evaluated query distribution: Each system is evaluated on 1450 queries spanning AllSouls, davinci-debate, ELI5, WikiHowKeywords, and seven NaturalQuestions subdistributions.The distributions include randomly sampled queries from each listed source.
- Example queries from each evaluated query distribution: Debate queries are generated from manually transformed Perspectrum propositions, expanded with text-davinci-003, and manually filtered for inappropriate content.
- Example queries from each evaluated query distribution: ELI5 titles become queries by removing ELI5-specific prefixes, while article titles are paraphrased into concise Google-style keyword queries.
- Annotation Interface: The annotation interface first collects five-point ratings of fluency and perceived utility, then filters statements for verification-worthiness.
- Annotation Interface: The final annotation step judges citation support for each statement and citation, including whether multiple citations jointly provide full support.Annotators used qualification guidelines before participating in the main evaluation.
E Annotation Quality
Annotation quality is assessed on a random sample of 250 query-response pairs using pairwise agreement and F1 against majority consensus. Agreement is high across all reported judgment types.
- 250 randomly sampled query-response pairs were used to compute inter-annotator agreement.
- Agreement was measured both between individual rating pairs and against the majority consensus using F1.
- More than 82.0% pairwise agreement and 91.0 F1 were achieved for all judgments.
F Fluency and Perceived Utility
Existing generative search engines generally produce fluent responses that appear informative and useful, but fluency declines for some information-aggregation queries.
- Existing generative search engines generally produce fluent text.Fluency is evaluated using five-point Likert ratings.
- Fluency is notably lower on NaturalQuestions queries with table-type long answers and no short answers.These queries often require aggregating information within or across citations.
- Responses from existing generative search engines generally appear informative and useful.Perceived utility is evaluated using five-point Likert ratings.
G Citation Recall and Precision
Human evaluation reports citation recall and precision across query distributions, with both metrics described as concerningly low.
- Citation recall is concerningly low across evaluated query distributions.Many generated statements are not fully supported by citations.
- Citation recall and precision are evaluated in systems that may serve as primary tools for fulfilling users' information needs.The passages emphasize that these systems already have millions of users.
- Citation precision is concerningly low across evaluated query distributions.Many generated citations do not support their associated statements.
H Citation F1
Citation F1 is reported for the evaluated generative search engines across each query distribution.
- Table 9 presents citation F1 for every evaluated generative search engine.
- Citation F1 is reported for each query distribution.
- The citation F1 results are presented as human-evaluation results for generated responses.