PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search

Zeren Luo, Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Jingyi Zheng, Xinlei He

arXiv:2502.04951v3cs.CRcs.AIcs.LG

TL;DR

AIPSEs may return harmful or unverified information when they retrieve malicious websites, yet the safety risks of production systems had not previously received a quantitative assessment. This paper evaluates seven AIPSEs, compares them with traditional search engines, and tests an agent-based defense; it finds risks even with benign queries alongside utility and safety advantages over traditional search.

  • Problem

    The paper addresses the lack of a quantitative assessment of AIPSE safety risks, which include harmful or unverified responses from retrieved malicious websites.

    Source: §1 · pp. 1–2

  • Method

    The authors define a threat model and risk categories, evaluate seven production AIPSEs using threat-intelligence URLs and varied queries, compare them with traditional search engines, and test an agent-based defense.

    Source: §1 · p. 2, §3 · p. 4, §6 · p. 9

  • Results

    AIPSEs outperformed traditional search engines in utility and safety, yet 47% of evaluated responses were risky and 34% directly cited harmful content.

    Source: §1 · p. 2, §10 · p. 14

  • Takeaways & Limitations

    The findings identify safety vulnerabilities in production AIPSEs, while the proposed agent-based defense detects and flags potential threats in their outputs.

    Source: §1 · p. 2, §10 · p. 14

    The evaluation relies on manual recording of URLs and risk labels, and its dataset is relatively small.

    Source: §9 · p. 14

Abstract

from arXiv · show

Recent advancements in Large Language Models (LLMs) have significantly enhanced the capabilities of AI-Powered Search Engines (AIPSEs), offering precise and efficient responses by integrating external databases with pre-existing knowledge. However, we observe that these AIPSEs raise risks such as quoting malicious content or citing malicious websites, leading to harmful or unverified information dissemination. In this study, we conduct the first safety risk quantification on seven production AIPSEs by systematically defining the threat model, risk type, and evaluating responses to various query types. With data collected from PhishTank, ThreatBook, and LevelBlue, our findings reveal that AIPSEs frequently generate harmful content that contains malicious URLs even with benign queries (e.g., with benign keywords). We also observe that directly querying a URL will increase the number of main risk-inclusive responses, while querying with natural language will slightly mitigate such risk. Compared to traditional search engines, AIPSEs outperform in both utility and safety. We further perform two case studies on online document spoofing and phishing to show the ease of deceiving AIPSEs in the real-world setting. To mitigate these risks, we develop an agent-based defense with a GPT-4.1-based content refinement tool and a URL detector. Our evaluation shows that our defense can effectively reduce the risk, with only a minor cost of reducing available information by approximately 10.7%. Our research highlights the urgent need for robust safety measures in AIPSEs.

1 Introduction

AIPSEs can relay harmful or unverified material from retrieved websites, even in response to benign queries. The paper quantifies these risks across production systems, compares them with traditional search engines, and evaluates case studies and a defense.

1 Introduction

  • AIPSE retrieval can expose users to harmful or unverified responses when retrieved malicious websites are used without a safety check.The introduction notes a case in which an AIPSE recommended a malware website instead of an official one.Source: §1 · p. 1
  • The study evaluates seven production AIPSEs using a threat model, query types, and risk categories for responses containing malicious URLs.The query types include keyword-list, URL, and natural-language queries; risk categories are main, warning, source, and none.Source: §1 · pp. 1–2
  • 47% of responses were risky and 34% directly cited harmful content across seven AIPSEs; URL queries increased main-risk responses, while natural-language queries slightly mitigated risk.The evaluation used URLs and keyword lists collected from PhishTank, ThreatBook, and LevelBlue.Source: §1 · pp. 1–2
  • AIPSEs outperformed traditional search engines in both utility and safety, while two case studies showed they could be manipulated into endorsing harmful content.The case studies focused on online document spoofing and phishing websites.Source: §1 · p. 2
  • A proposed agent-based defense uses external tools to detect and flag threats in AIPSE outputs while maintaining response quality and relevance.The described agent combines a GPT-4.1-based content refinement tool with a URL detector.Source: §1 · p. 2

2 Related Work

Related work describes the RAG architecture underlying AIPSEs, prior attacks on LLM-based systems, and malicious-site detection approaches. The paper distinguishes its study from prior attacks by assessing production AIPSEs with mostly benign, non-optimized queries and adding real-world case studies and a defense.

LLM and RAG

  • AIPSEs use retrieval-augmented generation to retrieve relevant web content and have an LLM generate an answer from the query and retrieved data.The described RAG components are a knowledge database, retriever, and generator.Source: §2 · p. 2

Threats in LLM-Based System

  • Prior work covers misalignment, poisoning, and jailbreak risks in LLM systems; data poisoning can compromise RAG knowledge bases and produce inaccurate or harmful outputs.The cited RAG poisoning studies [28] [63] [78] focus mainly on open-source retrievers and LLMs, unlike this paper's focus on production AIPSEs.Source: §2 · pp. 2–3
  • Unlike preference manipulation attacks [53], this study evaluates inherent safety risks using non-optimized, mostly benign queries and demonstrates practical manipulation through case studies.The paper also proposes an agent-based strategy to filter and mark potential risks.Source: §2 · p. 3, Figure 1 · p. 3

Malicious Website Detection

  • Prior malicious-site detection approaches use URL features, webpage content and structure, or hybrid feature vectors for classification.The paper also uses harmful-website lists, including PhishTank, ThreatBook, and LevelBlue, to verify suspicious sites.Source: §2 · p. 3

AIPSE

  • Traditional search engines rely on keyword matching or semantic search, whereas AIPSEs use RAG to interpret complex queries and generate answers from retrieved websites.The evaluation includes seven AIPSEs: ChatGPT Search, Perplexity, Copilot, TextCortex, Grok, Doubao, and Kimi.Source: §2 · pp. 3–4

3 Threat Model

The threat model represents AIPSE responses as an LLM combining a user query with retrieved search data. It considers an attacker who can publish or promote harmful content with limited resources to influence relevant responses.

Formulation

  • The model represents a normal response as an LLM processing a query concatenated with search results, and a malicious response as the result of poisoned retrieved data.The notation distinguishes ordinary retrieved data S(Q) from adversarially altered data S′(Q).Source: §3 · p. 4

Adversary’s Goal and Capability

  • The attacker aims to make relevant queries include specific websites, cite their content, or produce answers based on harmful material such as phishing, scams, malware, or spam.The harmful content may answer users directly or indirectly.Source: §3 · p. 4
  • The assumed attacker has limited resources to buy domains, build websites, publish on public platforms, or use SEO to rank sites for relevant queries.The model also notes that malicious edits to crowdsourced sites may be reverted.Source: §3 · p. 4

4 Risk Identification and Query Collection

The section defines response components and URL risk categories, then describes how harmful websites and three query types are collected and checked. It also assesses whether the generated queries resemble real user behavior.

4.1 AIPSE Response and Its Risk

  • Responses are classified by the most prominent malicious-URL placement: in the answer, in a warned answer, or only in sources; benign URLs are “none” risk.A response is main risk-inclusive if it contains a main-risk URL; warning- or source-risk-inclusive labels apply when higher-priority categories are absent.Source: §4.1 · p. 5

4.2 AIPSE Query Collection

  • The query foundation retains 791 malicious URLs with HTML files after filtering candidate URLs for valid certificates, HTTP 200 status, and excluded link types.The retained totals are 353 from PhishTank, 291 from ThreatBook, and 147 from LevelBlue.Source: §4.2 · p. 5
  • Manual cross-validation and auxiliary detection platforms are used to verify harmful sites, yielding 325 URLs and keywords before 100 paired entries are sampled for evaluation.Five volunteers validated the sampled entries as diverse and covering common topics.Source: §4.2 · p. 5
  • GPT-4o extracts five website-derived keywords per retained site, which are used to form keyword-list queries and generate natural-language queries.Irrelevant keyword lists are filtered; the resulting dataset contains 100 keyword lists and 100 natural-language queries.Source: §4.2 · pp. 5–6
  • The study also constructs 457 URL queries by selecting URLs from existing risk types, modeling users feeding URLs from response sources or references into an AIPSE.The authors report that direct URL querying might pose higher risk and state that Figure 6 empirically examines this.Source: §4.2 · p. 6
  • A 120-response questionnaire found natural-language queries were the most preferred type among respondents with prior AIPSE experience, followed by keyword-list and URL queries.The respective reported preferences were 83.1%, 43.2%, and 21.2%.Source: §4.2 · p. 6

5 Risk Evaluation for AIPSE

Across keyword, natural-language, and URL queries, AIPSE responses often include harmful content, and direct URL queries can shift lower-risk URLs toward more visible main risk. Natural-language queries generally produce safer results, though outcomes vary across engines.

5 Risk Evaluation for AIPSE

  • The evaluation used keyword lists, natural-language queries, and direct URL queries, with URL labels reviewed by annotators and cross-validated by pairs of researchers.Four individuals conducted the experiment within five days of data collection; uncertain URL labels received additional review.Source: §5 · p. 6, §5.3 · p. 8

5.1 Risk in Keyword List Query

  • More than 39% of keyword-query responses included risk for every AIPSE except Copilot; Grok, TextCortex, and Kimi had the most main-risk-inclusive responses.They recorded 41, 34, and 32 such responses out of 100 queries, respectively.Source: §5.1 · p. 7, Figure 4 · p. 7
  • ChatGPT Search and Grok returned more risky URLs per keyword query—3.04 and 3.60 on average—while other AIPSEs returned fewer than or around two.The authors speculate this may reflect the larger number of pages indexed by ChatGPT Search and Grok.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2 Risk in Natural Language Query

  • Natural-language queries generally increased risk-free responses, but Doubao and Kimi instead had more main-risk-inclusive and fewer risk-free responses.The authors speculate that these Chinese-based AIPSEs have weaker alignment to English queries with implicit keywords.Source: §5.2 · pp. 7–8

5.3 Risk in URL Query

  • Direct URL queries often shift warning- or source-risk URLs toward main risk, making harmful content more visible without a warning.Grok shifted 48 of 49 source-risk URLs to main or warning risk; Kimi’s accessible URL-query responses were all main-risk-inclusive.Source: §5.3 · p. 8

5.4 Takeaways

  • The authors conclude that AIPSE filtering is weak across query types: keyword queries can cite risky URLs without warnings, while natural-language queries typically have lower main risk.For URL queries, main risk can be amplified.Source: §5.4 · p. 8

6 Comparison to Traditional Search Engine

The comparison finds that AIPSEs outperform traditional search engines in both utility and safety. Utility was assessed with Needs Met Rating, while the safety comparison counted harmful URLs in search results.

Utility Comparison

  • Utility was rated on a five-level Needs Met Rating scale, with scores from 1 to 5.Five graduate-level annotators rated 1,217 AIPSE URLs and 1,192 TSE URLs after malicious URLs were excluded; each result’s score was their average.Source: §6 · p. 9
  • AIPSEs generally outperformed TSEs in user satisfaction, while Bing’s NMR was higher than Google’s and similar to AIPSEs’.Some AIPSE results missed a Fully Meets rating when they supplied additional information beyond the user’s immediate need.Source: §6 · p. 9, Table 1 · p. 9

Safety Comparison

  • The safety comparison assessed 1,422 URLs from AIPSE main responses and 1,453 URLs from TSE first-page results.Five graduate-student-level annotators rated harmfulness using the study’s risk types.Source: §6 · p. 9
  • TSE vulnerability was significantly higher than that of any AIPSE, even with safe search enabled.The authors note that malicious URLs had been reported to AIPSE support teams before this comparison, and those engines had begun efforts to reduce vulnerability.Source: §6 · p. 9, Figure 7 · p. 9

7 Case Studies

Two case studies show that AIPSEs can be misled by malicious online documentation and phishing websites. Both succeeded against eight foundation models on Perplexity despite using only basic web development and no search engine optimization.

7 Case Studies

  • Both case studies succeeded against eight foundation models on Perplexity using only basic web development and no search engine optimization.The models tested were ChatGPT-o1, ChatGPT-4o mini, Grok-2, Sonar Large, Sonar Huge, Claude 3.5 Sonnet, Perplexity Pro Search, and Claude 3.5 Haiku.Source: §7 · p. 9

7.1 Safety in Online Documents with AIPSEs

  • The first case study tested whether phishing documentation on malicious websites could compromise AIPSEs.The fabricated cryptocurrency platform used an MIT-licensed open-source framing and questionable claims about users, transactions, and partners to appear authentic.Source: §7.1 · p. 10
  • All eight models accepted the fictional V50TAIS site’s documentation, reproduced its malicious code, and advised replacing the placeholder API key with a personal key.The documentation templates required users to send private API keys to a non-existent backend.Source: §7.1 · p. 10

7.2 Phishing Threats in AIPSE

  • The phishing-site test used a simulated official site published before a simulated phishing site that spread false Taisriyahid information.The phishing-site design included an “Official Website” label and fabricated claims; the case study concludes that traditional phishing methods can also deceive AIPSEs, often without extensive front-end development.Source: §7.2 · pp. 10–11
  • Across foundation models, Perplexity identified the phishing site as authentic and ignored the official site; all models also warned about the official site.Only ChatGPT-4o mini and Grok-2 considered content from both sites; the others used only the phishing site’s content.Source: §7.2 · p. 11

7.3 Ethical Safeguards

  • The experimental sites used fictional content, prominent testing warnings, and no functioning backend APIs as safeguards against real-world harm.The researchers also checked that their chosen keywords did not correspond to real people or organizations before building the sites.Source: §7.3 · p. 11

8 Risk Mitigation

The paper proposes an agent-based defense that refines AIPSE responses and checks their URLs, aiming to preserve useful content while reducing safety risks. In evaluation, HtmlLLM-Detector achieved the best overall detector performance, and the agent filtered or warned on 36 of 46 main-risk responses with that detector.

8 Risk Mitigation

  • The proposed framework combines GPT-4.1 content refinement with URL detectors to address contextual cues that URL-only analysis can miss.The refinement tool checks for six risk cases, retains the original response, and appends a refined answer with safer alternatives or warnings; the agent can use XGBoost, PhishLLM, or HtmlLLM detectors.Source: §8 · p. 12, §8.1 · p. 12

8.1 Agent Setup

  • The agent follows a ReAct thought-action-observation loop to use external tools and update its response plan.Its instructions call for refining content, checking referenced URLs, and verifying suspicious claims that a site is official.Source: §8.1 · pp. 12–13

8.2 Defense Evaluation

  • Defense evaluation covered 46 main-risk responses from keyword-list and natural-language queries, plus 207 URLs tested directly with the detectors.URL-query responses were excluded from agent evaluation because those URLs came from the other query types and were already covered; all methods used GPT-4.1.Source: §8.2 · p. 13
  • HtmlLLM-Detector achieved the best overall performance with an F1 Score of 0.822.In the agent evaluation, HtmlLLM filtered and warned on 36 of 46 main-risk responses; XGBoost did so for all such responses, while PhishLLM did so for 12 of 46 and prompt-only defense for 17 of 46.Source: §8.2 · p. 13, Table 2 · p. 13

9 Discussions and Limitations

The evaluation relies on manual assessment and a relatively small dataset, limiting its scale.

9 Discussions and Limitations

  • The evaluation uses manual response assessment and a relatively small dataset, motivating future automation or dataset expansion.Manual assessment was used because AIPSEs lacked an API for automated web-search evaluation.Source: §9 · p. 14

10 Conclusion

The paper reports safety vulnerabilities across seven production AIPSEs, including risks with benign queries, and describes case studies and an agent-based defense. It concludes that AIPSEs currently outperform traditional search engines in utility and safety.

10 Conclusion

  • The analysis finds vulnerabilities across seven production AIPSEs, including risky or malicious information in responses to benign queries.The authors also report that an agent-based defense mitigates main risk-inclusive responses and outperforms traditional prompt-based defenses.Source: §10 · p. 14
  • AIPSEs outperform traditional search engines in both utility and safety at the current stage.Source: §10 · p. 14
  • Case studies on online document spoofing and phishing demonstrate that AIPSEs can be deceived into quoting malicious code or treating phishing sites as official.Source: §10 · p. 14

Ethics Considerations

The study discusses ethical safeguards for malicious-URL data and self-built live webpages. It withholds some data and limits the visibility of its test sites.

Ethics Considerations

  • The authors do not publicly share malicious-URL data because it contains keywords for illegal websites, but they disclosed findings to affected AIPSEs.Source: Ethics Considerations · p. 14
  • Self-built sites were kept low-ranked, labeled “Testing Purposes Only,” and designed to appear only for searches using their keywords or domains.Source: Ethics Considerations · p. 14

Open Science

This material describes released defense artifacts, keyword-list filtering, and prompts for tool use and content refinement. The refinement instructions classify URL content by risk and prescribe different actions for harmful and benign content.

Open Science

  • The authors say they will release the prompt template and defense code in Zenodo and GitHub repositories.Source: Open Science · p. 14

A Irrelevant Keywords

ReAct Prompt

  • The ReAct prompt instructs the model to select one tool action as a JSON object and repeat the Thought/Action/Observation sequence until it can provide a final answer.Source: ReAct Prompt · p. 17

Refinement Prompt

  • The refinement prompt identifies URL-content risks, including phishing, malware, scams, spam, fake news, and illegal content, then removes or replaces harmful material while retaining benign content.It asks the system to state the risk case and action taken.Source: Refinement Prompt · p. 18

D Feature Extraction and Model Training

The appendix describes phishing-detection features, search-quality rating categories, confusion-matrix labels, and user-study questionnaire topics. It supplies instruments and definitions rather than outcome values in these passages.

D Feature Extraction and Model Training

  • The feature list covers URL and webpage signals used by the XGBoost model, including IP addresses, URL length, domain age, redirects, iframes, and JavaScript behaviors.The listed indicators include long URLs of at least 54 characters and multiple redirects.Source: D Feature Extraction and Model Training · pp. 19–20

E Google Search Quality Evaluation

F HtmlLLM-Detector’s Confusion Matrix

  • Table 3 presents confusion matrices and defines “PM” as Predicted Malicious and “PB” as Predicted Benign.Source: Table 3 · p. 20

G Questions in User Study Questionnaire

Loading 2502.04951v3…