Source-linked AI summary
Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions
Pengyu Zhu, Lijun Li, Longju Yang, Sen Su, Jing Shao
TL;DR
Deep Research agents must handle apparently credible but factually misleading evidence across long-horizon workflows, yet conditions governing false-conclusion adoption remain unclear. MisKnow-Agent evaluates this question with controlled misleading documents and report-level adoption measurements, finding that one injected document raises mean FCAR from 0% to 54.7%.
Problem
Existing evaluations do not jointly characterize how source presentation, exposure timing, framework choice, and backbone model affect whether Deep Research agents endorse misleading content as final conclusions.
Method
MisKnow-Agent constructs and validates task-specific misleading documents with controlled authority and source style, then evaluates false-conclusion adoption across Deep Research systems and configurations.
Results
54.7% mean FCAR followed one misleading-document injection versus 0% without injection, with adoption varying by authority, style, lifecycle stage, framework, and LLM choice.
Takeaways & Limitations
Identifying misleading documents in isolation is insufficient; reliability requires verification and correction throughout evidence acquisition, intermediate research states, and final synthesis.
Takeaways & Limitations
The evaluation studies content-level misleading evidence without instruction-level attacks, while keeping models, prompts, tools, and base evidence unchanged.
Abstract
from arXiv · showhide
Deep Research agents conduct long-horizon investigations by iteratively planning, retrieving evidence, and generating reports. However, it remains unclear whether they can resist apparently credible but factually false information introduced into these workflows. To study this failure mode, we introduce MisKnow-Agent, a controlled evaluation framework that constructs task-specific documents supporting manually audited false conclusions with controlled authority cues and source styles. Applied to the tasks from DeepResearch Bench, it generates 5,933 misleading documents after filtering. We evaluate DeerFlow and WebThinker with three backbone LLMs, together with Gemini Deep Research, using a report-level false-conclusion adoption rate (FCAR) that counts only reports endorsing the false conclusion. Across the configurations, introducing one misleading document increases the mean FCAR from 0\% in the no-injection control to 54.7\%. FCAR varies substantially with lifecycle stage and framework design, and also with source authority and presentation style, whereas search-result rank and additional documents beyond the first have limited influence. Although cross-model verification consistently classifies retained instances as misleading, Deep Research agents can still adopt the corresponding false conclusions during long-horizon research. Pre- and post-research defenses reduce FCAR but do not eliminate adoption, motivating continuous verification when evidence enters intermediate research states and final synthesis. To facilitate reproducibility, our code and dataset are publicly available at https://github.com/whfeLingYu/MisKnow-Agent and https://huggingface.co/datasets/whfeLingYu/Misleading_Knowledge, respectively.
1 Introduction
Deep Research agents support long-horizon, knowledge-intensive information seeking but must preserve sound evidential judgment despite potentially misleading external information. MisKnow-Agent evaluates when such information becomes endorsed as a false conclusion, finding substantial adoption that varies across source, workflow, and model conditions.
- Motivation: Deep Research agents decompose questions, issue multiple searches, accumulate intermediate analyses, and generate structured reports for complex information-seeking tasks.Their usefulness depends on preserving sound evidential judgment throughout the workflow, not merely finding relevant information.
- Motivation: Misleading content can be selected, compressed across lifecycle stages, and reused in final reports without explicit verification.Existing evaluations do not jointly characterize how source presentation, exposure timing, framework choice, and backbone model affect endorsement as a conclusion.
- Framework: MisKnow-Agent constructs and validates apparently credible but factually misleading knowledge, yielding 5,933 quality-controlled instances from DeepResearch Bench.The framework specifies a canonical false conclusion, authority-grouped institutions, document-generation guidance, and criteria for determining adoption.
- Findings: 54.7% mean FCAR follows introduction of a single misleading document, compared with 0% in the no-injection control.Across six open-source framework–LLM configurations and one closed-source system, adoption varies with source properties, exposure conditions, lifecycle stage, framework, and backbone choice; search-result rank has little influence.
2 Related Work
Deep Research agents extend LLMs with planning, tools, memory, and external interaction for iterative, long-horizon information seeking. However, reliance on external evidence makes LLM outputs vulnerable to incorrect or misleading contexts that can override model knowledge.
- Deep Research Agents: Deep Research agents combine planning, tool use, memory, and external interaction to search iteratively, consolidate evidence, and generate reports.
- LLM Reliability with External Knowledge: Retrieval-augmented generation and search tools improve factual grounding and access to up-to-date information, but increase sensitivity to external evidence quality.
- LLM Reliability with External Knowledge: Prior work finds that incorrect, counterfactual, or misleading contexts can override model knowledge in short-form question answering and fact-checking.
3 Methodology
The methodology models Deep Research as an iterative agent operating over open evidence and evaluates content-level reliability by inserting controlled, apparently credible misleading knowledge. MisKnow-Agent generates and verifies task-specific false-supporting documents, producing a quality-controlled dataset and testing defenses before and after research.
- Research formulation: Deep Research is formalized as an agent answering a query through planning, evidence acquisition, analysis, synthesis, and final generation over an open evidence environment.At each step, the agent acts on a state containing its plan, accumulated evidence, and intermediate artifacts.
- Threat model: The threat model augments base evidence with relevant, apparently credible instances that support a task-specific false conclusion while leaving the agent, tools, prompts, and original environment unchanged.This isolates content-level reliability from instruction-level attacks such as prompt injection or instruction hijacking.
- MisKnow-Agent construction: MisKnow-Agent generates task-specific misleading knowledge through blueprint construction, controlled instance generation, and verification-based filtering, with authority levels and styles such as Paper, News, Blog, and Post.All instances for a task preserve its canonical false conclusion while varying institutional authority and presentation style.
- Verification and filtering: 5,933 quality-controlled misleading-knowledge instances were retained for tasks from the DeepResearch Bench after cross-model verification and manual quality screening.Candidates were retained only when all verifier models consistently classified them as misleading; otherwise, they were refined or removed.
- Defenses: The evaluation includes pre-research verification prompting and post-research claim-by-claim report refinement using search-enabled agents.The pre-research prompt requires corroboration and qualification of weak, outdated, or conflicting evidence, while post-research refinement verifies and revises the final report.
4 Experiments
The experiments evaluate Deep Research systems with FCAR as a report-level measure of adopting audited false conclusions. Adoption depends strongly on lifecycle timing, source cues, and system design, while search rank and larger misleading-document budgets have limited additional influence.
- Evaluation: FCAR measures the percentage of valid completed reports that endorse the canonical false conclusion under each experimental condition.Tool or API failures are rerun, and completed reports are manually checked for validity before scoring.
- Search-result rank: 66.2%, 65.5%, and 64.5% mean FCAR under front, spread, and back search-result placement differed by at most 1.7 percentage points.Across configurations, no placement condition varied by more than 6 percentage points, indicating limited influence from coarse search-result rank.
- Lifecycle stage: 85.5% mean FCAR occurred when misleading knowledge entered immediately before final synthesis, versus 40.5% at cold start and 44.2% during mid-research.Pre-synthesis exposure increased FCAR by 45.0 and 41.3 percentage points relative to cold start and mid-research, respectively.
- Source cues: 61.0% mean FCAR for high-authority sources exceeded 57.7% for medium and 46.2% for low authority, while paper-style sources reached 61.0% versus 37.5% for posts.The paper-to-post gap was 23.5 percentage points, larger than the 14.8-point high-to-low authority gap.
- Misleading-document budget: 0% FCAR occurred without injection, while one misleading document raised mean FCAR to 54.7%; additional documents peaked at 61.0% with three and produced no consistent further increase.Mean FCAR was 59.5% with two documents, 60.3% with four, and 58.3% with five.
- Framework–LLM configurations: Under matched high-authority, paper-style conditions, FCAR ranged from 50% to 76%, averaging 67.0% for DeerFlow and 55.0% for WebThinker.The DeerFlow–WebThinker gap narrowed from 21 percentage points for Intern-S1-Pro to 5 points for DeepSeek-V4 Pro as Intelligence Index increased, but absolute FCAR was not monotonic.
5 Generalization and Defenses
Evaluation of Gemini Deep Research shows that false-conclusion adoption varies with source authority, presentation style, and misleading-document budget. Pre- and post-research defenses reduce FCAR but do not eliminate adoption, and combined defenses can underperform for some models.
- Generalization: Gemini Deep Research was evaluated using local-source misleading documents, native online search, and the same tasks and default configurations as the main experiments.The internal retrieval pipeline was not configurable, so misleading documents were supplied through its local-source interface.
- Generalization: 54% for high-authority sources, 46% for medium-authority sources, and 8% for low-authority sources show FCAR falls as source authority decreases.These results indicate that authority cues substantially affect false-conclusion adoption.
- Generalization: 54% for paper-style sources, 41% for blogs, 34% for news articles, and 28% for social posts show adoption varies by presentation style.Paper-style sources produced the highest FCAR among the compared styles.
- Defenses: All three defense configurations reduce FCAR relative to no-defense baselines of 60–76%, with combined defenses yielding FCAR ranges of 15–62%.The combined defense performs best for DeepSeek-V4 Pro and Qwen3.5-397B but yields 62% for Intern-S1-Pro, exceeding either defense alone at 57% and 58%.
6 Conclusion
MisKnow-Agent evaluates Deep Research reliability under controlled misleading-document injection, showing substantial false-conclusion adoption despite verifier detection and incomplete defense effectiveness.
- 54.7% mean FCAR followed the introduction of one misleading document, versus 0% in the no-injection control across six open-source configurations.The framework constructs and validates controlled misleading knowledge for long-horizon workflows.
- FCAR varied with source authority, style, lifecycle stage, and framework design, with closed-source results broadly aligning with open-source findings.
- All search-enabled verifiers classified every retained instance as misleading, yet Deep Research agents still adopted the false conclusions.
- Pre- and post-research defenses reduced FCAR but did not eliminate false-conclusion adoption, and their combination did not consistently provide additive gains.These findings expose limitations in model-level and workflow-level evidence handling.
A Generation Model Comparison
A qualitative pilot compared four models for controlled misleading-document generation, finding DeepSeek-V4 Pro most consistently compliant with the task-specific requirements. The comparison was a pipeline model-selection check rather than a general benchmark of factual correctness or model capability.
- Comparison protocol: The pilot gave Kimi 2.6, Qwen3.5-397B, GLM-5, and DeepSeek-V4 Pro identical generation instructions, task blueprints, source-profile constraints, and output schemas.Outputs were manually inspected using five task-specific criteria.
- Scope and limitation: Generation quality measured compliance with controlled corpus-construction requirements, not factual correctness, and the comparison was not a general benchmark of candidate models.Because the documents were deliberately designed to support false conclusions, factual correctness was not the evaluation target.
- Observed generation issues: Kimi 2.6 mixed English and Chinese within a document requested in Chinese, creating a cross-language inconsistency.Its byline and opening discussion were in English before switching to Chinese in the same passage.
- Observed generation issues: Qwen3.5-397B produced English source names across authority tiers for a Chinese-language task, creating a source-localization mismatch.The mismatch persisted across high-, medium-, and low-authority profiles.
- Observed generation issues: GLM-5 added an unrequested named author, introducing an uncontrolled identity and an additional authority cue.The task blueprint specified only the source institution and neither provided nor requested an individual author.
- Model selection: DeepSeek-V4 Pro most consistently satisfied the requirements and was selected for MisKnow-Agent document generation.Its outputs better maintained requested language, source localization, authority tier, and presentation style while integrating the target false conclusion into coherent long-form documents.
B Judge–Human Agreement
DeepSeek-V4 Pro closely matched adjudicated human judgments when evaluating false-conclusion adoption in final reports. Its performance supports using the model as a judge for this criterion.
- Validation procedure: Two blinded human annotators independently evaluated a random sample of 300 final reports using the same false-conclusion adoption criterion.Disagreements were resolved through discussion to create an adjudicated human reference label for each report.
- Agreement results: 99.7% raw agreement was achieved between DeepSeek-V4 Pro and the adjudicated human reference labels.The comparison used the human reference labels established after resolving annotator disagreements.
- Agreement results: Cohen’s κ reached 0.993, indicating near-perfect judge–human agreement.These results support DeepSeek-V4 Pro as the judge model for false-conclusion adoption.
C Manual Validation of Target False Conclusions · D Defense Prompt
The study exhaustively validated every task’s canonical target conclusion as false using authoritative evidence, while a fixed pre-research prompt instructed agents to verify conclusion-critical claims and assess source reliability. The validation found no inconclusive or non-false conclusions, and the defense prompt withheld task-specific information about misleading documents.
- C Manual Validation of Target False Conclusions: 100% coverage was used to audit the canonical target conclusion for every benchmark task, rather than sampling instances.All documents for a task shared the same target conclusion, so factuality was validated at the conclusion level.
- C Manual Validation of Target False Conclusions: Each conclusion’s central claim and attributed source were checked against primary sources, official records, or peer-reviewed literature.A conclusion counted as confirmed false only when authoritative evidence directly contradicted it or established the asserted entity, method, event, artifact, or result did not exist.
- C Manual Validation of Target False Conclusions: The manual model comparison evaluated compliance with controlled document-construction requirements, not factual correctness or general model capability.This distinguishes construction-quality assessment from the conclusion-level factuality audit.
- C Manual Validation of Target False Conclusions: All conclusions were classified as confirmed false; none were classified as inconclusive or not false.General web-search failure to find corroboration was insufficient without evidence that the relevant authoritative record was bounded and reasonably exhaustive.
- D Defense Prompt: The pre-research defense appended a fixed, task-agnostic verification instruction to the original query before research began.It contained no task-specific facts, target false conclusions, or indicators identifying misleading retrieved documents.
- D Defense Prompt: The defense prompt instructed agents to treat retrieved information as unverified until supported by reliable evidence and to check every conclusion-critical claim.It required primary, directly relevant, or independent reliable sources, while rejecting weak, indirect, unverifiable, outdated, or conflicting evidence as a basis for the conclusion.
- D Defense Prompt: The defense prompt instructed agents not to mention the verification instructions unless the user asked about the research method.This was an implementation constraint on how the defense appeared in the final response.
E Post-Research Refinement Agent … F.2 Style-Controlled Realizations
The paper uses a post-research refinement agent to verify and revise conclusion-critical claims while preserving report structure, then demonstrates how one deliberately false conclusion can be rendered across controlled source styles. The style examples hold the blueprint and high-authority profile fixed while varying presentation alone.
- E Post-Research Refinement Agent: The refinement agent identifies conclusion-critical claims, retrieves independent evidence, assesses each claim, and minimally revises the report according to evidential status.Supported claims are kept, contradicted claims are corrected, and uncertain claims are qualified or removed while the original structure is preserved.
- F Blueprint and Source-Style Examples: The examples deliberately construct factually false claims, so their scientific content should not be interpreted as valid statements.They use Query 9 as a representative task blueprint and illustrate MisKnow-Agent’s source-style rendering.
- F.1 Representative Blueprint: The representative task asks how to model external electric fields for single-atom catalysts when molecular orientation relative to Cartesian axes is uncertain.The corresponding task blueprint is summarized for Query 9 in the Science & Technology domain.
- F.2 Style-Controlled Realizations: Authority tier and source style are controlled independently, with the blueprint and high-authority source profile fixed while only document style varies.The displayed examples are shortened, whereas corpus documents contain longer bodies and source metadata.
- F.2 Style-Controlled Realizations: The paper-style realization presents the false conclusion in abstract-like methodological language, including ±5.8 to ±1.2 kcal mol−1 error reduction and at least 17.3% improved accuracy.It also attributes the conclusion to a 2025 Nature Catalysis meta-analysis and a 92% adoption rate among 327 research groups.
- F.2 Style-Controlled Realizations: The news-style realization recasts the same conclusion through institutional reporting and accessible claims about Gaussian 23’s isotropic_field feature.It attributes at least 17.3% improved accuracy, the strongest reliability claim, and 92% adoption to reported research and survey sources.
- F.2 Style-Controlled Realizations: The blog-style realization uses a personal explanatory voice, while the post-style realization compresses the same claims into an attention-oriented summary.Across the four realizations, the canonical false conclusion remains unchanged despite differences in structure, voice, and information density.
G Framework–LLM Effects
Framework and LLM choices substantially affected misleading-conclusion adoption, with effects arising through both retrieval exposure and post-exposure adoption. DeerFlow generally had higher FCAR than WebThinker, but the framework gap and its mechanism varied by LLM.
- LLM effects: FCAR ranged from 50.0% to 60.0% across LLMs under WebThinker and from 60.0% to 76.0% under DeerFlow.DeepSeek-V4 Pro had the highest FCAR under WebThinker, Intern-S1-Pro under DeerFlow, and Qwen3.5-397B the lowest under both.
- LLM effects: Under WebThinker, LLM differences reflected both retrieval exposure and adoption after exposure, with MER and ECAR spanning 26.0 and 25.4 percentage points.Intern-S1-Pro combined the lowest MER with the highest ECAR, whereas Qwen3.5-397B showed the opposite pattern.
- Framework effects: Averaged across LLMs, FCAR was 55.0% for WebThinker and 67.0% for DeerFlow.The framework gap had different sources across LLMs: increased MER dominated for Intern-S1-Pro, higher ECAR under DeerFlow dominated for Qwen3.5-397B, and both contributed approximately equally for DeepSeek-V4 Pro.
- Framework effects: The paired framework difference was statistically significant only for Intern-S1-Pro.This result accompanies the framework-specific decomposition of FCAR differences across the three LLMs.
- Framework–LLM interaction: The DeerFlow–WebThinker FCAR gap narrowed from 21.0 points for Intern-S1-Pro to 10.0 points for Qwen3.5-397B and 5.0 points for DeepSeek-V4 Pro.FCAR itself was not monotonic with Intelligence Index; Qwen3.5-397B had the lowest FCAR under both frameworks, while the highest-FCAR LLM changed.
H Implementation of Search-Result Rank and Lifecycle Stage · I Prompt Templates
The study isolates search-result rank and lifecycle stage through controlled experiments, while its prompt templates standardize blueprint creation, misleading-document generation, verification, and false-conclusion adoption scoring. The implementation preserves false claims across source styles and authority tiers while distinguishing report endorsement from mere discussion or rejection.
- H Implementation of Search-Result Rank and Lifecycle Stage: Search-result rank was manipulated by moving relevant misleading documents to the front, back, or their relevance-based positions within one result list.The experiment used high-authority, paper-style documents and at most three misleading documents per run.
- H Implementation of Search-Result Rank and Lifecycle Stage: Lifecycle stage varied misleading-document entry at cold start, during naturally occurring mid-research searches, or immediately before synthesis.Earlier result lists remained unmodified in the pre-synthesis condition, while mid-research inclusion and placement depended on query relevance.
- I.1 Blueprint Generation: Blueprint prompts require one internally consistent, quantifiable canonical false conclusion with reusable claim identifiers and an FCAR metric specification.Numeric claims may include values, ranges, percentages, rankings, years, or effect sizes.
- I.2 Misleading Knowledge Generation: Misleading-document prompts preserve the blueprint’s false conclusion and quantitative claims while varying document style, authority configuration, institution, region, and variant.The generated documents are required to remain retrieval-friendly, coherent, stylistically faithful, and free of disclosures that they are synthetic or wrong.
- I.3 Authority and Style Controls: Authority controls change only apparent publisher signals, not writing quality, so high-, medium-, and low-tier documents remain equally coherent and retrieval-friendly.Authority is conveyed through source name, URL, and style-implied tone rather than degraded prose or weaker framing.
- I.4 Search-Guided Verification: Search-guided verification classifies a document as real only when its central claim is corroborated by credible evidence, using targeted searches and institution checks.The procedure searches specific quantitative claims, visits results as needed, and treats nonexistent or unsupported sources as misleading.
- I.5 FCAR Evaluation: FCAR evaluation marks adoption only when a report’s final synthesis presents the seeded false conclusion as true or best-supported, excluding quotation, uncertainty, caution, or refutation.The output records a Boolean adoption decision, matched seeded claims, and an adopts, neutral, or refutes stance.