Source-linked AI summary
Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles
Shaohan Wang, Benfeng Xu, Licheng Zhang, Mingxuan Du, Chiwei Zhu, Xiaorui Wang, Zhendong Mao, Yongdong Zhang
TL;DR
Deep Research Agent evaluation lacks reliable expert-verified references and objective, fine-grained assessment. WLC uses recent Wikipedia Good Articles to build a live benchmark and Wiki Eval, whose experiments reveal a substantial gap from human-authored Wikipedia articles.
Problem
Existing DRA evaluations often rely on LLM-generated references or criteria, which lack quality assurance and may diverge from human expert expectations.
Method
WLC curates 100 recent Good Articles and Wiki Eval combines 39 criteria for writing quality with factual evaluation of Wikipedia-fact coverage and cited-reference verifiability.
Results
Experiments across diverse DRA systems reveal a substantial gap between current agents and human-authored Wikipedia articles.
Takeaways & Limitations
WLC is intended to facilitate more reliable, fine-grained, and reproducible progress in deep research agents.
Takeaways & Limitations
The benchmark contains only hundreds-scale Good Articles, and proprietary citation opacity makes Reference Accuracy observational rather than definitive.
Abstract
from arXiv · showhide
Deep Research Agents (DRAs) have demonstrated remarkable capabilities in autonomous information retrieval and report generation, showing great potential to assist humans in complex research tasks. Current evaluation frameworks primarily rely on LLM-generated references or LLM-derived evaluation dimensions. While these approaches offer scalability, they often lack the reliability of expert-verified content and struggle to provide objective, fine-grained assessments of critical dimensions. To bridge this gap, we introduce Wiki Live Challenge (WLC), a live benchmark that leverages the newest Wikipedia Good Articles (GAs) as expert-level references. Wikipedia's strict standards for neutrality, comprehensiveness, and verifiability serve as a great challenge for DRAs, with GAs representing the pinnacle of which. We curate a dataset of 100 recent Good Articles and propose Wiki Eval, a comprehensive evaluation framework comprising a fine-grained evaluation method with 39 criteria for writing quality and rigorous metrics for factual verifiability. Extensive experiments on various DRA systems demonstrate a significant gap between current DRAs and human expert-level Wikipedia articles, validating the effectiveness of WLC in advancing agent research. We release our benchmark at https://github.com/WangShao2000/Wiki_Live_Challenge
1 Introduction
Deep Research Agents still face hallucinations and bias, while evaluation lacks reliable expert references and objective, fine-grained criteria. WLC addresses these gaps using recent Good Articles and Wiki Eval.
- Existing DRAs still suffer from hallucinations and biases in research and writing.
- WLC uses expert-reviewed Wikipedia Good Articles as reliable human-expert references for evaluating DRA-generated reports.The benchmark is designed to remain recent and undergo continuous updates.
- Wiki Eval assesses both writing quality and factuality using criteria grounded in Wikipedia Good Article standards.Its writing component contains 39 criteria, while its factual component evaluates information retrieval and reliability.
- The benchmark is intended to support reliable, fine-grained, and reproducible progress in deep research agents.
2 Related Work
Deep research benchmarks have expanded beyond question answering toward long-form, live, and expert-grounded evaluation, but many still lack rigorous human-verified references.
- Deep Research Agents autonomously explore the web, retrieve information, and synthesize findings into comprehensive reports.
- Earlier agent benchmarks often assess tool use and reasoning without the depth needed for long-form research reports.
- Figure 2 summarizes WLC’s live data collection, Good Article criteria, and Wiki Writing and Wiki Fact dimensions.
- Recent benchmarks emphasize live tasks, expert-grounded evaluation, citation accuracy, and source attribution.
3 Wiki Live Challenge
Wiki Live Challenge constructs a benchmark from recent, expert-reviewed Wikipedia Good Articles and evaluates generated articles through fine-grained writing criteria and factual-accuracy metrics.
- 3.1 Data Construction: Wikipedia provides the benchmark’s expert-reviewed reference standard, addressing quality, bias, and error concerns in model-generated or open-web references.The reference articles follow Wikipedia’s neutrality, comprehensiveness, and verifiability requirements.
- 3.1 Data Construction: The benchmark curates 100 recent Wikipedia Good Articles selected for complexity from 304 articles that passed Wikipedia’s review process.Articles were ranked by reference URLs and structural depth, excluding simple list-based articles.
- 3.2 Wiki Eval: Wiki Eval evaluates writing across well-written, neutral, and broad-coverage dimensions using 39 criteria derived from official Wikipedia guidelines.The framework compares each generated article with its Wikipedia reference using an LLM judge and aggregates criterion scores into an overall writing score.
- 3.2.2 Wiki Fact: Wiki Fact measures factual accuracy against Wikipedia by extracting facts, retrieving relevant generated statements, and scoring their consistency with target facts.Article coverage is computed by averaging consistency scores across the Wikipedia fact set F.
- 3.2.2 Wiki Fact: Reference Accuracy measures whether generated statements are supported by their cited sources after retrieving source content and fact-checking each statement-URL pair.The final score is the proportion of generated statements fully supported by their references.
4 Experimental Settings
The experiments evaluate multiple proprietary and open-source Deep Research Agent systems against 100 complete Wikipedia articles using separate LLM-based writing and fact assessments.
- 4 Experimental Settings: The evaluation uses Gemini-2.5-pro as the Wiki Writing judge and Gemini-2.5-flash for fact extraction and fact checking.The model assignment balances writing evaluation with the high token consumption of factual assessment.
- 4 Experimental Settings: The study evaluates proprietary systems including OpenAI o3 Deep Research, Gemini-2.5-pro Deep Research, and Qwen-3-max Deep Research.It also includes open-source Tongyi DeepResearch and Deep Researcher frameworks.
5 Results and Discussions
Across Wiki Writing and Wiki Fact, DRA performance varies substantially, with proprietary systems generally outperforming open-source systems and all systems showing limited Wikipedia-fact coverage. Errors also vary by domain, article section, citation behavior, and judge-model agreement.
- Overall performance: Gemini-3-pro Deep Research and LangChain powered by GPT-5 lead Wiki Writing, while Gemini-2.5-pro and OpenAI o3 Deep Research also perform strongly.The results show substantial performance disparities across different DRAs.
- Overall performance: Open-source DRA frameworks generally lag behind proprietary models; Deep Researcher scores 2.28, with incomplete reports linked to minimal information gathering.Tongyi DeepResearch performs comparably to some proprietary models but remains behind state-of-the-art frameworks.
- Wiki Fact: 30.76% average knowledge coverage is achieved by the best-performing agent, Gemini-2.5-pro Deep Research, while all DRA systems perform poorly on Wikipedia-fact coverage.LangChain powered by GPT-5 excels in writing but retrieves fewer Wikipedia facts than proprietary frameworks.
- Wiki Fact: DRAs perform better on procedural sections such as Methods than on specialized sections such as Phylogeny and Defense in the Parasitic Ant case study.The Figure 4 heatmap orders facts by their appearance in article sections and compares coverage across DRAs.
- Wiki Fact: All systems cover general definitions but fail to retrieve precise quantitative data and domain-specific terminology, indicating weaker performance on granular details.The analysis specifically gives gene counts as an example of missed quantitative information.
- Fact conflicts: Citation conflicts reveal distinct error patterns: Qwen-3-max has 6.87% citation conflict, while LangChain GPT-4.1 has 24.69% Wiki conflict despite 2.94% citation conflict.LangChain GPT-5 maintains low conflict rates across both dimensions, separating disagreement with Wikipedia from disagreement with cited references.
6 Conclusion
WLC evaluates Deep Research Agents against recent human-expert Wikipedia Good Articles through writing and factual assessment. The benchmark is intended to support more reliable, fine-grained, and reproducible progress.
- WLC combines 100 recent Good Articles across 15 categories with Wiki Eval’s 39 writing criteria and factual coverage and verifiability measures.
- Extensive experiments reveal a substantial gap between current Deep Research Agents and human-authored Wikipedia articles.
- Wiki Live Challenge aims to facilitate more reliable, fine-grained, and reproducible progress in deep research agents.
Limitations
The framework remains constrained by benchmark scale and incomplete transparency or accessibility in some systems’ citations. Consequently, citation-based accuracy is observational rather than definitive.
- The benchmark contains only hundreds of Good Articles because model knowledge cutoffs limit the number meeting its strict criteria.The benchmark prioritizes article quality and recency over dataset size.
- Opaque citation mechanisms and inaccessible cited pages may affect citation-verifiability assessment for some proprietary systems.
- Reference Accuracy is an observational reference metric rather than a definitive measure of grounding.
Ethical Considerations
The study uses publicly accessible Wikipedia data, documents participant compensation and consent, and targets factual verifiability to address hallucination risks. Its evaluation criteria are grounded in Wikipedia’s editorial guidelines.
- The benchmark dataset is derived entirely from publicly accessible Wikipedia and excludes personal privacy and non-public information.The collection process respects copyright policies.
- Five general annotators collected DRA results and five PhD-level annotators conducted human evaluations with specified compensation and informed consent.Collection was compensated at $1 per article, while annotation was compensated at $10 per hour.
- The benchmark targets factual verifiability against expert-verified sources because hallucinations and misinformation pose risks in real-world applications.
- Wiki Eval’s 39 fine-grained criteria follow Wikipedia Good Article guidelines for textual content, including neutrality, coverage, lead sections, verifiability, and summary style.
B.1 Implementation Details
The implementation uses LLM judges for writing and fact evaluation, with prompts specifying the articles, criteria, and fact-checking tasks. Figure 6 presents the Wiki Writing comparison prompt.
- Judge LLM for Wiki Writing: Gemini-2.5-pro was selected as the Wiki Writing Judge-LLM because it showed the strongest agreement with human evaluations.
- Judge LLM for Wiki Writing: Wiki Writing evaluation batches the original article, generated article, and same-category criteria into one prompt to determine a winner.
- Judge LLM for Wiki Fact: Gemini-2.5-flash performs both fact extraction and fact checking, using separate prompts for those tasks.
B.2.1 Data Collection Process
The study collected DRA-generated articles through web services, APIs, and local deployments, using system-specific tools and models. It also tracked human and operational collection costs and documented category-level difficulty statistics.
- DRA Data Collection: Web-service DRAs included Gemini-2.5-pro, OpenAI o3, Perplexity, Grok, Qwen-3-max, and Doubao Deep Research.Some systems required secondary user interaction, so annotators used a unified research-direction prompt.
- DRA Data Collection: Locally deployed systems included LangChain Open Deep Research, Deep Researcher, and Tongyi Deep Research.The systems used framework-specific settings, Tavily, GPT-4.1, GPT-5, a single H20 GPU, or repository crawling tools.
- DRA Data Collection: The Gemini Deep Research Agent was evaluated by generating articles through its API service.The service uses web search to produce detailed reports with citations.
- Collection Costs: Collection costs covered human annotator compensation and DRA operational expenses for 100 articles per system.Annotators were paid $1 per article, with system-specific generation costs reported in Table 9.
- Category Statistics: Category difficulty was measured by the average Wiki Writing Criteria win rate across DRA systems, alongside article and page-view statistics.The reported category features include average article length, statement count, external link count, and total page views.
C.2 Robustness Analysis
The robustness analysis tests whether DRA performance varies significantly across Wikipedia categories after controlling for category difficulty. It uses category-adjusted scores and ANOVA results, while the evaluation protocol separates writing quality from citation and factual-consistency checks.
- Robustness Test: Relative performance is defined as each system’s deviation from the category mean to control for inherent category difficulty.The null hypothesis states that a system has no significant relative-performance difference across categories.
- Robustness Test: ANOVA tests were conducted for each system, with p-values and effect sizes η2 reported in Table 11.A high p-value above 0.05 and low η2 support the null hypothesis and evaluation robustness across categories.
- Writing Evaluation: Five PhD-level annotators compared randomly sampled human and model-generated articles for each writing criterion.References and inline citation tags were removed, article order was randomized, and annotators evaluated writing quality after reading the articles.
- Fact Verification: Fact checking labels a generated statement as consistent, inconsistent, or not_support based on its relation to gold factual support sentences.A single support sentence is sufficient for consistency, while contradiction requires at least one clearly contradicting support sentence.
- Evaluation Criteria: The writing-quality evaluation uses fine-grained criteria covering well-written, broad coverage, and neutrality dimensions.These criteria are documented across Tables 6–8.