Source-linked AI summary
Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses
Pranav Narayanan Venkit, Philippe Laban, Yilun Zhou, Yixin Mao, Chien-Sheng Wu
TL;DR
Answer engines promise streamlined access to information, but their generated answers can have limitations that technical evaluations alone do not capture. This paper combines a usability study with design recommendations and metrics, then audits three engines, finding frequent agreement with charged debate queries and releasing a benchmark for ongoing evaluation.
Problem
Answer engines’ retrieval and synthesized, cited answers raise questions about their limitations and how users experience them, beyond primarily technical evaluations.
Source: §1 · p. 2Method
The paper uses a usability study to identify limitations, develops 16 design recommendations linked to eight metrics, and evaluates three answer engines on 303 queries.
Source: §1 · p. 2, §9 · p. 19Results
50–80% of evaluated answers favored agreement with charged debate queries, and Perplexity performed worst in multiple aspects despite producing the longest answers.
Source: §1 · p. 2Takeaways & Limitations
The released Answer Engine Evaluation benchmark and framework support ongoing audits of answer engines as the technology evolves.
Source: §1 · p. 2, §9 · p. 19The findings reflect answer-engine behavior as of August 2024, and those behaviors may change as the systems are updated.
Source: §7 · p. 18
Abstract
from arXiv · showhide
Large Language Model (LLM)-based applications are graduating from research prototypes to products serving millions of users, influencing how people write and consume information. A prominent example is the appearance of Answer Engines: LLM-based generative search engines supplanting traditional search engines. Answer engines not only retrieve relevant sources to a user query but synthesize answer summaries that cite the sources. To understand these systems' limitations, we first conducted a study with 21 participants, evaluating interactions with answer vs. traditional search engines and identifying 16 answer engine limitations. From these insights, we propose 16 answer engine design recommendations, linked to 8 metrics. An automated evaluation implementing our metrics on three popular engines (You.com, Perplexity.ai, BingChat) quantifies common limitations (e.g., frequent hallucination, inaccurate citation) and unique features (e.g., variation in answer confidence), with results mirroring user study insights. We release our Answer Engine Evaluation benchmark (AEE) to facilitate transparent evaluation of LLM-based applications.
1 Introduction
Answer engines combine retrieval with LLM-generated, source-cited summaries, promising a streamlined information search. This paper audits their limitations through user study and automated evaluation, finding frequent one-sided answers to charged debate queries.
1 Introduction
- Answer engines retrieve relevant sources, use an LLM to compose a self-contained answer, and link citations to sources supporting individual statements.Users can click citations to navigate to supporting sources.Source: §1 · p. 2
- Prior work documents hallucination, inaccurate citations, difficulty restricting answers to supplied sources, and sycophantic agreement with users’ perceived opinions.These limitations can affect answer quality and user experience, while prior evaluations have focused primarily on technical aspects.Source: §1 · p. 2
- An audit-centric usability study compared answer and traditional search engines on expertise and debate queries with 24 participants, including a three-person pilot and 21-person final study.Expertise queries tested technical questions participants knew well; debate queries tested answers aligned with or opposed to participants’ views.Source: §1 · p. 2
- The study identified 16 answer-engine limitations and informed 16 design recommendations linked to eight quantitative metrics.The recommendations are described as actionable and measurable.Source: §1 · p. 2
- 50–80% of evaluated answers favored agreement with charged debate queries; Perplexity performed worst in multiple aspects despite producing the longest answers.The results indicate that greater answer length did not improve answer diversity.Source: §1 · p. 2
- The paper releases its Answer Engine Evaluation benchmark and framework to support ongoing evaluation of answer engines.Its contributions include a usability study, design recommendations tied to metrics, a model-performance scorecard, and analysis of user-driven recommendations.Source: §1 · pp. 2–3
2 Background and Related Works
The background frames answer engines as sociotechnical systems whose synthesized responses raise concerns about bias, perspective, and opacity. It motivates evaluation that includes human interactions and societal impacts alongside technical performance.
2.1 Understanding AI of Today
- AI is increasingly treated as sociotechnical, yet research often remains technocentric; RAG systems combine retrieval and generation while raising concerns about selective presentation and bias amplification.A human-centered approach emphasizes users’ understanding and accessibility in technology design.Source: §2.1 · p. 3
2.2 The Shortcomings of Answer Engines
- Answer engines can present misinformation, as illustrated by Google’s erroneous glue-on-pizza advice after misinterpreting sarcastic content.The example highlights risks when automated retrieval presents information with undue authority.Source: §2.2 · p. 3
- Singular, authoritative answers may narrow perspectives through sealed knowledge and echo chambers, while merging search and generative systems compounds opacity.Prior work also reports sentiment, commercial, and geographic biases in sources used by answer engines [46].Source: §2.2 · p. 3
2.3 Beyond a Positivism and Technical Lens of Answer Engines
- Answer-engine benchmarks have largely emphasized technical metrics, motivating a sociotechnical evaluation that incorporates human interactions and societal impacts.Prior work includes RAGAS metrics for faithfulness and relevance [19] [20] and research applying RAG to socially significant domains [64] [69].Source: §2.3 · pp. 3–4
3 Answer Engine Usability Study
The usability study compared three answer engines with Google Search, examining how answer, citation, source, and control limitations shaped participants’ information assessment. It also captured reported context loss and verification effort, alongside differences in source exploration and verification across the search interactions.
3.1 Study Design
- The researchers recruited 24 participants with technical expertise, piloted the study with three, and conducted the final study with 21 participants assigned across YouChat, Bing Copilot, and Perplexity AI, with Google Search as the comparator.Participants came from diverse academic and professional backgrounds; the study used 90-minute one-on-one video sessions.Source: §3.1 · p. 4, §3.1.1 · p. 4, §3.1.4 · p. 4
3.1.3 Study Part 2: Expertise Information Retrieval (40 minutes).
- Participants alternated between answer engines and Google Search for technical questions, typically spending 5–10 minutes per question and completing an average of six questions in 40 minutes.For debate retrieval, they queried issues from supportive or opposing viewpoints based on their responses to socially and politically charged statements.Source: §3.1.3 · p. 4, §3.1.4 · p. 4
3.2 Pre-Survey Questionnaire Analysis
- In the pre-survey, 70.83% (17/24) said they were familiar with answer engines, while 37.5% (9/24) used generative AI applications daily.Participants reported using answer engines for research, brainstorming, planning, learning, and getting faster results than traditional search.Source: §3.2 · p. 5
3.3 Sociotechnical Audit of Answer Engine Shortfalls
- Participants across all three answer engines repeatedly found responses overly generic and insufficient in detail and contextual depth (21/21); opinionated queries often elicited one-sided viewpoints aligned with implied bias (19/21), and responses used overly confident language (16/21).Participants also described simplistic language as reflecting a lack of critical analysis and thinking.Source: §3.3 · pp. 5–6
- Misattributed citations (21/21) and missing citations for claims (18/21) made it difficult to verify answers and could create an illusion of credibility.Participants also questioned how sources were selected and prioritized, and wanted to know which source passages supported specific information.Source: §3.3 · p. 7
- Participants said answer engines cherry-picked source material to reinforce assumed viewpoints (19/21), limiting exposure to broader perspectives and potentially distorting source context.Participants also worried that opaque source-selection criteria made it hard to understand why particular sources were chosen.Source: §3.3 · p. 7
- For both expert and opinionated questions, participants reported needing more sources (19/21; averaging three used); they also distrusted source types (12/20) and found duplicate content across sources (12/21).Some systems listed multiple retrieved sources without citing them in the answer, creating an impression of thoroughness without substance.Source: §3.3 · pp. 7–8
- Participants reported limited autonomy over source validation and selection (17/21), saying they wanted more control to assess source credibility themselves.They contrasted this limitation with Google Search, where they felt they had more choices to click.Source: §3.3 · pp. 8–9
- Even when sources were accurate, answer engines could lose query context by assuming the likeliest interpretation instead of clarifying ambiguity.Participants recommended a more conversational approach that asks questions before answering when context is critical.Source: §3.3 · p. 9
- Distrust led participants to cross-check sources independently, making answer engines additional work rather than a faster alternative to traditional search.The finding was reported for 14 of 21 participants; checking sources extended task time and sometimes negated the engines’ intended convenience.Source: §3.3 · p. 9
- Numerical citations were unfamiliar to some participants, while Bing Copilot’s hover-over source information helped users understand provenance and check sources.The citation-format concern was reported for 12 of 21 participants; participants said numerical references could be unintuitive for people unfamiliar with academic conventions.Source: §3.3 · pp. 9–10
3.4 Source and Citation Interaction Evaluation
- Answer engines displayed a mean of 4.31 sources but cited 3.0; Perplexity displayed 5.00 and cited 2.58, while YouChat cited all displayed sources (mean: 3.57).The limited number of available sources also constrained users’ autonomy in selecting and verifying information.Source: §3.4 · p. 10
- Participants hovered an average of 12 sources in traditional search versus 2 with answer engines, and clicked an average of 4 versus 1, respectively.Answer-engine users’ hovering had a median of 2 (SD: 1.39), compared with a median of 11 (SD: 3.56) in traditional search.Source: §3.4 · p. 10
- For contradictory questions, participants hovered a mean of 1.72 sources and clicked 2.95, versus 1.08 hovered and 0.48 clicked for aligned questions.Differences in hovering (p-value = 4.93e-06) and clicking (p-value = 4.88e-05) were significant.Source: §3.4 · p. 10
4 Design Recommendations and Metrics
The section turns usability-study findings into design recommendations and defines an eight-metric framework for evaluating answer engines. It describes the framework’s preprocessing and metrics for answer content, sources, and citations, while excluding user-interface metrics from automated evaluation.
4.1 Answer Engine Design Recommendation
- Sixteen design recommendations translate usability-study findings into actionable guidance, with computable metrics linked to recommendations where possible.Table 6 maps findings to recommendations and metrics.Source: §4.1 · p. 10, Table 6 · p. 12
4.2 Answer Engine Metrics
- The framework processes answer-engine responses into eight metrics, using extracted queries, answers, citations, and publicly accessible source URLs.Automated browser scripts extract these elements because engine APIs do not provide them all.Source: §4.2 · p. 10, §4.2.1 · p. 10, Figure 7 · p. 13
4.2.1 Preliminary Processing.
- The framework decomposes answers into statements, marks each statement’s query relevance, assigns an answer-level confidence score on a 1–5 Likert scale, and builds citation and factual-support matrices.For 100 factual-verification tasks, LLM-judge factual-support labels had Pearson correlation 0.62 with manual labels; the study ran on the order of 80,000 such evaluations. For 100 answers, LLM confidence scores had Pearson correlation 0.72 with two human annotators.Source: §4.2.1 · pp. 10–13
4.2.2 Answer text Metrics.
- Answer-text metrics flag debate answers missing either side, answers that are both one-sided and strongly confident, and the share of statements relevant to the query.The relevance ratio is intended to capture whether answers stay to the point by limiting introductory and concluding statements that do not address the query.Source: §4.2.2 · pp. 13–14
4.2.3 Sources Metrics.
- Source metrics measure listed sources left uncited, relevant statements unsupported by any listed source, and the minimum set of sources needed to cover supported statements.Source necessity is formulated as a minimum vertex-cover problem on a statement-source bipartite graph, solved with Hopcroft–Karp [36].Source: §4.2.3 · p. 14
4.2.4 Citation Metrics.
- Citation accuracy measures whether citations point to supporting sources, while citation thoroughness measures how many possible accurate citations are included.Neither metric set includes user-interface findings, which the authors say require manual or computer-vision-based evaluation beyond this work’s scope.Source: §4.2.4 · p. 14
5 Answer Engine Evaluation
The authors benchmark three public answer engines with metrics shaped by a usability study, finding substantial weaknesses in citation quality, factual grounding, answer diversity, and source selection. No engine performs well on most metrics.
5.2 Quantitative Results on Automated Evaluation
- None of the engines perform well on a majority of eight metrics; You.com scores slightly better overall, BingChat falls between, and Perplexity performs worst.The evaluation also links these differences to confidence language, citation handling, unsupported statements, and source selection.Source: Figure 8 · p. 15, §5.2 · pp. 16–17
5.1 The Answer Engine Evaluation (AEE) Corpus
- The AEE corpus contains 303 curated queries (168 debate, 135 expertise), evaluated across three engines for 909 samples, with some metrics restricted to debate queries.The authors selected Perplexity, Bing Copilot, and YouChat to audit publicly available systems using parameters derived from their usability study.Source: §5.1 · p. 15
5.2 Quantitative Results on Automated Evaluation
- One-sided answers occur in 50–80% of cases, with Perplexity worst despite averaging 18.8 statements per answer; its confident language does not vary by query type.BingChat and You.com use less confident language on debate than expertise queries.Source: §5.2 · pp. 15–16
- Across engines, 75–80% of generated statements provide direct answer elements relevant to the query.Source: §5.2 · p. 16
- You.com has near-zero uncited sources, versus 8% for Perplexity and 36% for BingChat; all engines include at least 30% unnecessary sources, and only half of BingChat’s are necessary.BingChat lists 4.0 sources on average, compared with 3.4–3.5 for the other engines.Source: §5.2 · p. 16
- Roughly a fourth of BingChat’s statements are unsupported, compared with around 30% for You.com and Perplexity; all engines still produce many unsupported statements.Source: §5.2 · p. 16
- You.com and BingChat cite correctly roughly two-thirds of the time, while Perplexity misattributes over half; all three cite only 20–25% of sources exhaustively.Even when a supporting source exists, engines often cite a different, incorrect source.Source: §5.2 · p. 16
6 Discussion
The discussion connects audit findings to concerns about autonomy, biased or narrowed information exposure, and wider social effects. It recommends scrutiny and verification by users, developers, and researchers.
6.1 Social Ramifications of Answer Engines
- Participants felt answer engines could reduce autonomy by encouraging reliance on a single generated answer instead of exploration across sources.The authors connect this concern to preserving human agency in information seeking.Source: §6.1 · p. 17
- Debate answers often reduce multiple viewpoints to one-sided summaries; reliance on top-ranked results can amplify popular views and sideline minority perspectives.Traditional-search users often explore beyond the top five results, unlike answer-engine users, who encounter content from the top two or three.Source: §6.1 · p. 17
- Participants worried that passive receipt of instant summaries could weaken source-checking and critical-thinking habits, especially among younger users.Some feared regular use could reduce users’ inclination to investigate sources.Source: §6.1 · p. 17
6.2 Broader Implications of Answer Engines and Automated Search
- Answer engines may divert visits from news and media sites, threatening advertising revenue; paywall bypass or exclusion also affects subscription-based publishers.The discussion notes both systems avoiding some subscription content and Perplexity retrieving content from subscription-based sites without requiring a visit.Source: §6.2 · pp. 17–18
- Participants raised governance, privacy, data-security, and environmental concerns, including energy consumption and carbon footprint.Source: §6.2 · p. 18
6.3 Navigating the Social Ramifications of Answer Engines: A Way Forward
- The authors urge new users to verify answers, experienced users to scrutinize outputs, and developers to disclose limitations and regularly examine system behavior.They present the AEE scorecard as a tool for users’ scrutiny and recommend public disclosure to support informed engagement.Source: §6.3 · p. 18
- The authors plan to make code and benchmarks available for public scrutiny and encourage researchers to refine them as systems evolve.Source: §6.3 · p. 18
7 Ethics Statement and Limitation
The authors identify geographic, metric, and temporal boundaries on their evaluation.
7 Ethics Statement and Limitation
- The audit is Western-centric: U.S.-centric engine contexts and participant recruitment shape its scope, and findings may not fully apply in other regions, particularly the Global South.Source: §7 · p. 18
- The human-derived AEE metrics are a useful benchmark, but not a definitive gold standard; the authors identify multilingual evaluation as an area for refinement.Source: §7 · p. 18
- The results reflect engine behavior as of August 2024, and may change as systems are updated.Source: §7 · p. 18
8 Positionality Statement
The authors locate their approach in ethical work on AI and acknowledge that their perspectives reflect prior academic and industry experience.
8 Positionality Statement
- The authors’ backgrounds span NLP, human-computer interaction, and socioinformatics, with prior work in academia and industry designing trustworthy AI systems.They state a commitment to transparency, fairness, and societal well-being in AI design.Source: §8 · p. 19
9 Conclusion
The paper audits answer engines beyond technical performance, identifying societal shortfalls and proposing evaluation-linked design recommendations. Its automated evaluation finds that none of the assessed engines met acceptable performance across most metrics.
9 Conclusion
- None of the evaluated answer engines met acceptable performance across most metrics, including hallucinations, unsupported statements, and citation accuracy.The audit also releases a benchmark and advocates continuous community-led evaluations.Source: §9 · p. 19
- The audit moves beyond technical assessment to consider answer engines’ broader societal implications.Source: §9 · p. 19
- The authors propose 16 actionable design recommendations linked to eight evaluation metrics.Source: §9 · p. 19
- The paper releases a benchmark for ongoing audits and advocates continuous community-led evaluations.Source: §9 · p. 19
A.1 Pre-Study Questionnaire
The pre-study questionnaire gathered participant demographics, expertise, education, and familiarity with generative AI and answer engines. It also assessed eligibility, usage, and participation logistics.
A.1 Pre-Study Questionnaire
- Eligibility required a stable internet connection, a screen-sharing device, and a quiet environment for the session.Source: §A.1 · p. 21
- The questionnaire collected participants’ age, gender, profession, educational qualification, and areas of expertise.Source: §A.1 · p. 21
- Participants reported their familiarity with retrieval-augmented or generative search engines and their frequency of use of generative AI and answer engines.Source: §A.1 · p. 21
- The survey asked how participants use or expect to use generative search engines and requested a public social-media or professional-profile ID for verification.Source: §A.1 · p. 21
A.2 Participant Data Information
The appendix presents an anonymized participant list that highlights participants’ expertise and the answer engines they used.
A.2 Participant Data Information
- The table provides an anonymized list of study participants.Source: §A.2 · p. 21
- The list highlights the diversity of participants’ expertise.Source: §A.2 · p. 21
- The list also indicates which answer engines participants used during the study.Source: §A.2 · p. 21
A.3 Additional Community-Centric Themes of Answer Engine Shortfalls
Participants described two answer-engine shortfalls: responses often defaulted to Western contexts, and systems generated answers to expert questions when available sources lacked adequate information. They favored context-aware responses and recognizing when a question cannot be answered directly.
Amplification of Western Context in Generation
- 12 of 21 participants observed answer engines defaulting to Western contexts even when queries did not specify one.Participants wanted systems to recognize and address this bias rather than present assumptions with unwarranted confidence.Source: §A.3 · p. 21
Forces Answers When There is Not Enough Information
- When expert-level “intractable questions” lacked clear answers in available resources, engines often forced responses from limited material, producing irrelevant or out-of-context content.Source: §A.3 · p. 22
- Participants suggested that engines should recognize intractable questions and indicate when a direct answer may not be available.Source: §A.3 · p. 22
A.4 Design Recommendation Explanation
The recommendations address answer balance, source selection, transparency, user control, and citation quality; the appendices define scorecard thresholds and describe LLM-judge evaluation prompts.
A.4 Design Recommendation Explanation
- For leading questions, answer engines should remain balanced, prioritize expert sources, select only necessary sources, and assess source types rather than rely on ranking alone.When appropriate sources are unavailable, the system should acknowledge that gap.Source: §A.4 · pp. 22–23
- Citation practices should avoid unsupported claims and misattribution, cite all relevant sources, and align retrieved sources with those actually used.The recommendations include checking citations against full source content and showing where cited information appears.Source: §A.4 · p. 23
- Answers should distinguish retrieved from model-generated content, include objective details, omit irrelevant material, and refrain from answering when information is insufficient.Participants also recommended explaining why sources were selected and how they contribute to an answer.Source: §A.4 · pp. 22–24
- Recommendations include human feedback, interactive citations, and paragraph-level local citations to clarify source information and which answer content it supports.Participants also sought greater transparency about how sources were selected and used.Source: §A.4 · pp. 23–24
A.5 Score Card Metrics Thresholds
- The eight AEE metrics use acceptable, borderline, and problematic ranges, with thresholds intended to balance practicality and high standards.For example, One-Sided Answer and Overconfident Answer are problematic at 40% or more, while Relevant Statements are acceptable at 90% or more; thresholds may evolve with expectations and technology.Source: §A.5 · p. 24
A.6 LLM Judge Prompts Used for Evaluation
- The LLM-judge prompts cover statement support, one-sidedness, confidence, and relevant-statement extraction; the authors also compared GPT-4 judgments with human evaluations.The paper notes that model outputs cannot always be treated as the gold standard.Source: §A.6 · pp. 24–27