Source-linked AI summary
DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Tasks
Jingxuan Han, Wei Liu, Mingyang Zhu, Youpeng Wang, Ziwen Wang, Lin Qiu, Xuezhi Cao, Xunliang Cai, Zheren Fu, Licheng Zhang, Zhendong Mao
TL;DR
Existing search-agent benchmarks emphasize specialized tasks and coarse rubrics, limiting their realism and interpretability. DailyReport evaluates timely daily-search needs with decomposed cascade rubrics and user-centric scores, finding that 17 systems still struggle with factuality, rationality, and user satisfaction.
Problem
Existing benchmarks emphasize specialized, static tasks and coarse task-level rubrics, limiting realism and evaluation interpretability for search agents.
Method
DailyReport builds 150 real-world daily-search tasks with 3,546 rubrics and evaluates decomposed subtasks using cascade rubrics across disentangled dimensions and user preference.
Results
Across 17 agentic systems, agents perform well on instruction following but struggle with factuality and rationality, while user preference scores remain below acceptable levels.
Takeaways & Limitations
DailyReport characterizes current search agents’ strengths and limitations through interpretable dimensional and user-preference evaluations.
Takeaways & Limitations
DailyReport restricts tasks to benign requests to avoid rejection by safety mechanisms.
Abstract
from arXiv · showhide
Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses. For SAs evaluation, prior benchmarks mainly focus on specialized tasks that are unlikely to arise in real-world user scenarios. Moreover, their reliance on coarse task-level rubrics often limits evaluation interpretability. To bridge this gap, we introduce DailyReport, an open-ended benchmark to evaluate SA capabilities on daily search tasks. It contains 150 open-ended tasks with 3,546 associated rubrics, capturing widely discussed and timely information demands of real-world users. Each task is decomposed into subtasks and evaluated with cascade rubrics across disentangled dimensions. Through cascade performance attribution and user-centric aggregation, we derive highly interpretable scores for each dimension, along with a user preference score. Our results on 17 agentic systems show that current systems still fall short of users' expectations. To facilitate future research, our dataset and code are made publicly available at https://github.com/AGI-Eval-Official/DailyReport.
1 Introduction
DailyReport addresses the mismatch between specialized, static Search Agent benchmarks and users’ everyday, timely information needs. It introduces a 150-task benchmark with user-centric cascade evaluation and shows that current agents still fall short of user expectations despite strong instruction following.
- Motivation: Existing Search Agent benchmarks emphasize specialized, overprocessed domain questions that are unlikely to reflect real-world user scenarios.Their static designs also fail to capture evolving real-world information needs.
- Benchmark: DailyReport evaluates whether agents can satisfy everyday users’ timely and practical information needs through tasks derived from trending topics and user comments.The benchmark targets widely discussed information needs from authentic daily contexts, including platforms such as Weibo and Facebook.
- Findings: 17 agentic systems perform well in instruction following but struggle with factuality and rationality, while user preference scores remain limited.Solving-trace analysis diagnoses underlying failure patterns and guides future Search Agent advances.
- Benchmark: 150 tasks and 3,546 rubrics comprise DailyReport, supported by over 500 hours of human annotation.The tasks are grounded in real-world scenarios to reflect authentic user needs.
- Evaluation: The user-centric cascade evaluation pipeline computes subtask performance across disentangled dimensions and quantifies user preference interpretably.Cascade rubrics encode hierarchical priorities so failures in foundational subtasks can affect the usefulness of subsequent analysis.
2 Related Work
Prior benchmarks evaluate search agents through fixed-answer retrieval, reasoning, browsing, or broad fact aggregation, while LiveResearchBench targets daily demands but remains U.S.-centric. DailyReport addresses these gaps with up-to-date real-world tasks and cascade rubrics that support interpretable attribution and user-preference quantification.
- Search-agent benchmarks: Existing benchmarks target fixed-answer information retrieval and multi-step reasoning tasks for search-agent evaluation.The cited works include Chen et al. [2025], Li et al. [2025], Song et al. [2025], and Wu et al. [2026].
- Search-agent benchmarks: BrowseComp evaluates web-browsing capabilities, whereas WideSearch requires aggregating large volumes of atomic facts across wide contexts.These benchmarks represent browsing-focused and broad information-aggregation evaluation settings, respectively.
- Daily-search evaluation: LiveResearchBench aligns tasks with daily user demands but remains largely U.S.-centric, limiting regional coverage.This limitation motivates broader coverage of real-world daily search needs.
- Daily-search evaluation: DailyReport adopts up-to-date daily search tasks aligned with real-world user demands and uses cascade rubrics across disentangled dimensions.These design choices enable interpretable performance attribution and user-preference quantification for search-agent evaluation.
- Search-agent systems: LLM progress has accelerated search-agent development, including Deep Research Agents designed for challenging report-generation tasks.Related systems perform multi-step web search, local synthesis, or reinforcement learning with authentic web-search interactions.
3 DailyReport Benchmark
DailyReport is an expert-curated benchmark of 150 open-ended, real-world daily search tasks spanning 10 domains and 35 categories, with cascade rubrics designed for interpretable subtask-level evaluation. Its tasks derive from current trends, cover retrieval and analysis needs, and are constructed through expert formulation and hybrid annotation.
- Benchmark Characteristics: Tasks originate from trending topics and broad queries that combine multiple related sub-questions for report generation and practical information seeking.Sources include authentic trending topics and user discussions, while tasks are regularly updated as topics evolve.
- Benchmark Characteristics: 150 expert-curated tasks with 3,546 detailed rubrics span 10 high-level domains and 35 fine-grained categories aligned with real-world user demands.The benchmark targets daily user scenarios and predominant information needs.
- Task Construction: Experts formulate 150 tasks from topic reports and user comments under authenticity, clarity, and safety requirements to reflect timely, practical information needs.Topics are collected from major Western and Chinese platforms to provide diverse, regionally representative coverage.
- Task Construction: The benchmark contains 100 retrieval-centric tasks requiring objective information integration and 50 analysis-centric tasks requiring autonomous information identification for deeper analysis.Retrieval-centric tasks involve lightweight analysis, whereas analysis-centric tasks address broader subjective topics.
- Task Construction: Hybrid annotation first assigns tasks to 35 fine-grained categories and then consolidates them into 10 high-level domains representing broader fields.Fine-grained categories capture specific user interests, while high-level domains organize broader areas.
- Evaluation Design: Tasks are decomposed into subtasks and evaluated with cascade rubrics across three orthogonal dimensions, enabling interpretable performance attribution and subtask-importance-aware aggregation.Rubrics combine LLM generation with human expert refinement and can also support full LLM-based automation.
4 User-centric Cascade Evaluation
The evaluation uses cascade rubrics to score instruction following, factuality, and rationality at the subtask level, then aggregates these scores into interpretable dimensional and user-preference measures. Factuality and rationality are evaluated only when instruction-following requirements are satisfied, reflecting their dependence on response usability.
- Cascade Dimensional Evaluation: Cascade rubrics evaluate each subtask’s instruction following, factuality, and rationality with dimension-specific judge models.For subtask T_i, the agent response is evaluated against the corresponding rubric for each dimension.
- Instruction Following: Instruction-following scores are 0, 0.5, or 1, and the overall Ins score averages performance across subtasks.The values represent failure, partial satisfaction, and full satisfaction of each rubric.
- Factuality: Factuality is scored only when instruction following is satisfied; a judge verifies objective claims and references through web search to assign f_ack ∈[0, 1].This cascade attribution treats factuality as meaningful only when the required target content is present.
- Rationality: Rationality receives scores rat_k ∈{0, 0.5, 1} based on logical reasonableness, with the judge emphasizing subjective reasoning and analysis to reduce factuality coupling.The overall rationality score is formulated using a subtask indicator for rationality rubrics.
- User-Centric Aggregation: User-centric aggregation maps responses to four helpfulness levels and weights subtasks by creator-assessed importance to compute UserPref.Importance categories P0, P1, P2, and P2(a) encode the user-preference impact of leaving individual subtasks unsatisfied.
5 Experiment
DailyReport evaluates 17 agentic systems and finds that search-augmented LLMs perform best, while current systems remain weak in user preference, factuality, and complete subtask satisfaction. Further analyses identify retrieval and reference quality as key limitations, while showing stable evaluation results and domain-dependent performance.
- Experimental Setup: 17 agentic systems are evaluated across native DRAs, search-augmented LLMs, and LLMs with Claude Code, using Gemini-3-flash as the judge model.Reasoning mode is enabled for all evaluated models.
- Main Results: Search-augmented LLMs achieve the best overall performance, followed by Claude Code systems, while native DRAs score relatively lower; GPT 5.4-based configurations perform best.The results suggest that daily search benefits from direct web search combined with strong general-purpose LLMs.
- Main Results: UserPref remains below the acceptable level of 3 even for the highest-scoring system, and SubTask Pass remains low across systems.UserPref can exceed SubTask Pass when a system satisfies more high-importance subtasks while missing less critical ones.
- Capability Analysis: Instruction following is relatively strong across systems, whereas Factuality is the weakest dimension and Rationality remains imperfect.The factuality gap reflects difficulty acquiring accurate, timely evidence and avoiding hallucinated claims.
- Task and Domain Effects: Analysis-centric tasks show slightly higher instruction following and rationality but lower factuality, while domain performance is higher for structured topics than for dynamic or subjective information needs.Politics & Law and Industrial Economies score higher in UserPref, whereas Sports & Entertainment tends to score lower.
- Trace Analysis: Search-tool usage has the strongest association with overall performance, while reference citation is common but Reference_Support remains lower than Reference_Ratio.These findings motivate sufficient retrieval, higher-quality references, and better reference-claim alignment.
- Robustness and Validation: Repeated evaluations of three representative models produce low variance, demonstrating stable DailyReport results, while metric validation reaches 96.5% accuracy versus 88.4% for human annotation.Gemini-3-Flash is selected as judge because it balances evaluation accuracy and cost while comparing favorably with other evaluated judges.
6 Conclusion … A.2 Constraints Elaboration
DailyReport evaluates search agents on realistic daily search tasks through decomposed subtasks and cascade rubrics, while documenting substantial human involvement and explicit constraint categories. Its construction combines interpretable evaluation design with diverse annotation and review procedures.
- 6 Conclusion: DailyReport contains 150 tasks and 3,546 associated rubrics targeting timely information needs of real-world users.The benchmark evaluates search agents on open-ended daily search tasks.
- 6 Conclusion: Tasks are decomposed into subtasks and evaluated with cascade rubrics across disentangled dimensions.The design supports subtask-level evaluation through structured rubric construction.
- 6 Conclusion: DailyReport uses cascade performance attribution and user-centric aggregation to produce interpretable dimensional scores.The supplied conclusion passage identifies these as the mechanisms for deriving the benchmark’s evaluation outputs.
- A.1 Human Annotation: The task-construction pipeline involved over 500 hours of human annotation and review, with contributors compensated approximately USD 56–70 per day.Contributors had diverse regional, educational, platform-use, and domain backgrounds, and familiarity with Western and Chinese media ecosystems.
- A Construction Appendix: Annotators identified common information needs from trending posts and user comments while filtering unsafe, overly narrow, ambiguous, or unsuitable topics.Task writers then created realistic search tasks with clear scopes, factual requirements, and analytical components.
- A.1 Human Annotation: Additional reviewers checked tasks for clarity, realism, safety, search dependency, and category consistency.These checks followed task writing and formed part of the construction workflow.
- A.2 Constraints Elaboration: Constraint categories include Content Constraints for core output information and Scope Constraints for boundaries such as temporal, spatial, domain, source, or policy restrictions.The categories are used to decompose constructed tasks and derive corresponding subtasks.
B Evaluation Appendix · B.1 Meta Evaluation
The meta-evaluation tests whether DailyReport’s user preference score reflects real users’ perceived experience. Results show strong agreement with user ratings, including a Weighted Cohen’s Kappa of 0.859, while the appendix also defines measures for analyzing solving traces.
- B.1 Meta Evaluation: Figure 6’s diagonal concentration indicates strong consistency between the aggregated scores and real users’ perceived experience.Each heatmap cell counts sampled instances with the corresponding score pair.
- B.1 Meta Evaluation: Reference Accuracy measures the factual reliability of cited references, while Refer-Claim Consistency measures whether claims are supported by those references.These metrics are part of the appendix’s detailed solving-trace analysis.
- B.1 Meta Evaluation: Web Search counts retrieval-tool calls, whereas Web Content Mining counts webpage-fetching calls for accessing full content.Both call types are treated as important search-to-… measures in the solving-trace analysis.
- B.1 Meta Evaluation: 0.859 Weighted Cohen’s Kappa verifies high alignment between the aggregated user preference score and real user ratings.The score indicates that the proposed aggregation closely corresponds to users’ perceived experience.
- B.1 Meta Evaluation: High alignment holds for tasks with real user preferences of 1 and 4.Figure 6 shows that the aggregated user preference scores closely match these endpoint ratings.
- B.1 Meta Evaluation: High consistency is also maintained for real user preference scores of 2 and 3.The agreement pattern extends beyond the endpoint preference categories.
B.2 Search Analysis … B.3.2 Factuality
The benchmark evaluates search agents through instruction-following and factuality judgments while revealing weaknesses in citation accuracy and claim-reference consistency. Its factuality process verifies extracted objective claims using web search, web fetching, and cited references.
- B.2 Search Analysis: B.2 Search Analysis: Reference Accuracy remains particularly limited, suggesting that search agents may rely on inaccurate, unreliable, or inappropriate citations.Refer-Claim Consistency is also imperfect, indicating that retrieved references are not always used faithfully to support generated claims.
- B.3.1 Instruction Following: B.3.1 Instruction Following: The judge checks whether each decomposed subtask’s action, requested content, scope, quantity, format, and completeness requirements are satisfied.For example, requested entity lists must provide the required items while respecting the specified scope.
- B.3.1 Instruction Following: B.3.1 Instruction Following: A score of 1 means the subtask is fully satisfied with all essential requirements and constraints correctly followed.This is the highest category in the instruction-following rubric.
- B.3.1 Instruction Following: B.3.1 Instruction Following: A score of 0.5 denotes partial satisfaction when some non-critical requirements are missing or imperfectly handled.An example is returning all ten requested movie titles while omitting directors for some entries.
- B.3.1 Instruction Following: B.3.1 Instruction Following: A score of 0 applies when responses omit required content, answer irrelevantly, refuse without reason, or fail to perform the required action.This category represents an unsatisfied subtask.
- B.3.2 Factuality: B.3.2 Factuality: Factuality evaluates objective claims by extracting specific, verifiable statements and checking them through orchestrated web search and web fetching.The workflow uses Serper Search and Jina Reader to verify claim correctness.
- B.3.2 Factuality: B.3.2 Factuality: When reports cite sources, their referenced webpages are jointly evaluated with other retrieved sources to determine claim correctness and information consistency.This consistency measure tests whether agents faithfully synthesize information from retrieved webpages.
B.3.3 Rationality · B.4 LLM Configuration · B.4.1 Deep Research Agents
The benchmark evaluates rationality by independently judging extracted analytical content, while configuring native deep research models for autonomous operation and substantially longer execution times.
- B.3.3 Rationality: Rationality assesses whether analytical response components are logically sound and well supported.The evaluated components include explanations, comparisons, causal analysis, trade-off evaluation, and recommendations.
- B.3.3 Rationality: Rationality evaluation extracts explanation, comparison, causal-analysis, trade-off, and recommendation content from each response report.Factual claims already used for factuality verification are excluded.
- B.3.3 Rationality: The extraction procedure preserves independence between rationality and factuality evaluation dimensions.Claims used for factuality verification are excluded from the rationality-related parts.
- B.4.1 Deep Research Agents: Native deep research models receive specialized configurations tailored to their operational characteristics.These configurations address their integrated research capabilities and longer execution times.
- B.4.1 Deep Research Agents: Deep research models autonomously determine search strategies, execute queries, retrieve and analyze web content, and synthesize coherent reports without external tool definitions.Their web search and content synthesis capabilities are fully integrated.
- B.4.1 Deep Research Agents: Deep research execution involves multiple rounds of autonomous web exploration and content synthesis.This operational pattern motivates extended processing allowances.
- B.4.1 Deep Research Agents: 1,800 seconds is the timeout threshold for native deep research operations.The threshold was extended because these operations have substantially longer execution times.
B.4.2 LLMs with Web Search Tools … Claims Extract Prompt
The evaluation standardized web-search and Claude Code configurations, then used prompts enforcing tool-grounded reporting and strict, criterion-by-criterion scoring. A separate extraction prompt restricted claims to question-relevant content originating in the document.
- B.4.2 LLMs with Web Search Tools: B.4.2 configured standard LLMs with google_search for structured organic results and fetch_webpage for URL full-text extraction, while enabling extended reasoning.fetch_webpage primarily used the Jina Reader API for Markdown conversion.
- B.4.3 LLMs with Claude Code: B.4.3 enabled extended thinking for all Claude Code backend models with identical settings and provided MCP integrations for the tool ecosystem.The passage specifies consistent evaluation parameters across backend models and orchestration.
- System Prompt for Report Generation: The report-generation prompt prohibited parametric knowledge, required autonomous multi-angle research, and mandated a Markdown report of at least 2000 words in the question’s language.Every factual claim had to come from retrieved tools, with uncertainty explicitly acknowledged when tools could not provide information.
- System Prompt for Report Generation: The report prompt required sequential numbered citations for major claims and a final References section listing cited sources with titles, sites, and URLs.In-body citations use parenthetical numbers such as [1], [2], and [3].
- Instruction Follow Score Prompt: The instruction-following evaluator receives the Question, Document, and Criteria and is directed to score objectively and fairly.The input schema identifies each item as a string or list containing the user question, agent result, and scoring criteria.
- 3. Workflow: The workflow scores every criterion without modification: 1.0 for strict satisfaction, 0.5 for partial satisfaction, and 0.0 when related content conflicts with requirements.The evaluator must read all inputs, iterate through individual criteria, and verify each score for accuracy.
- 4. Caution: The caution rules require judging only the Document’s claims and stated time information, accepting explicit metric unavailability when allowed, and checking criteria explicitly rather than inferring from titles or references.Scoring must remain strict, body-based, and limited to the supplied criteria; reference materials are ignored.
- Claims Extract Prompt: The claims-extraction prompt restricts extraction to claim information relevant to the accuracy Question and originating exclusively from the Document.Its inputs are the Document and Question.
3. Workflow Step 1: Analysis and Clarification … Claims Integrate Prompt
The workflow first analyzes the document and question to identify relevant information, then extracts verifiable claims while preserving source content, and finally checks and integrates results into strict JSON output.
- 3. Workflow Step 1: Analysis and Clarification: The first step aims to accurately understand the input information.It requires deeply analyzing the document and question to identify all information related to the accuracy question.
- 3. Workflow Step 1: Analysis and Clarification: The analysis treats “delivery result” as referring to the document and uses headings at every level to identify relevant content.Headings may directly correspond to the question and can provide necessary subject context.
- Step 2: Location and Extraction: The extraction step locates target information, preserves its complete original wording in “fact”, and stores integrated sentences in “extract”.Integration may clarify pronouns, interpret chart content textually, or supplement missing background context without tampering with the source.
- Step 2: Location and Extraction: Each extracted item must be an objectively verifiable factual claim explicitly appearing in the document; fabricated or vague content is prohibited.Subjective evaluations, common sense, metaphors, suggestions, instructions, and hypothetical reasoning are excluded.
- Step 2: Location and Extraction: Claims must retain complete context, including subjects, quantities, times, locations, and relevant constraints rather than isolated or potentially ambiguous fragments.Examples require identifying the movie and actor, or stating both Huawei-phone compatibility and unsupported other Android phones.
- Step 3: Check and Integration: The final step verifies every field against the requirements and integrates results into a strict JSON object containing “fact” and “extract” entries.The workflow objective is to ensure that the complete process meets all notes before producing the final result.
- Step 3: Check and Integration: Even when no extraction results exist, the question must still be answered by outputting an empty list in “json_output”.The prescribed output format includes analysis followed by the extracted results.
- Appendix: Core Principles and Exclusion List: The workflow requires comprehensive, atomic claims: split independent body statements and table cells, merge sentences describing one factual point, and supplement headings as context.The “extract” field must not lose original information, and claim content must remain objective and verifiable.
1. Task Objective … Prompt for Reference Consistency Judgment
The workflow extracts relevant document passages, identifies and verifies claims, generates constrained search queries, and aggregates evidence into structured judgments. It emphasizes conservative deduplication, reference consistency, complete context, source quality, and strict JSON outputs throughout.
- 1. Task Objective: Claims are globally deduplicated and reassigned to the most specific matching accuracy question, while preserving distinct facts and using each claim ID only once.Exact duplicates are deleted; inclusion cases retain the refined claim unless it loses key information, and uncertain cases are kept.
- 3. Workflow Step 1: Claim Verification: Claim verification restores omitted context, removes exact duplicates, and forbids deleting claims merely because they cannot be verified online.Generalized claims are instead decomposed into specific factual components for query generation.
- Step 2: Identification and Decomposition: Claims requiring multi-stage or multi-angle verification are decomposed before generating independent yes/no queries with dependency IDs for progressive verification.Queries must preserve the claim and sub-question intent, and verifiable components must yield at least one query even when another query is removed.
- Step 4: Check and Integration: The workflow checks query completeness, semantic clarity, sub-question alignment, and limiting time or location information before integrating results into strict JSON.Queries must not omit, tamper with, or fabricate claim content, although only the claim portion relevant to the sub-question needs querying.
- Paragraph Extract Prompt: Relevant paragraphs are extracted for every question in complete, original form, with repeated extraction allowed when one paragraph answers multiple questions.The process reads each question, locates and extracts all related paragraphs, then performs a secondary correctness check.
- Rationality Judge Prompt: Rationality judgment evaluates each extracted paragraph against document context while excluding already verified claims and distinguishing textual errors from instruction-following omissions.It checks contradiction, common-sense violations, reasoning errors, and semantic confusion without relying on personal preferences or external factual searches.
- System Prompt for Fact-Checking: Fact-checking searches verification queries, judges each as True, False, or Unknown, and derives the claim judgment from all query outcomes using multiple authoritative sources and time constraints.A claim is True only when all queries are True, False when any query is False, and Unknown when at least one is Unknown without any False; multimodal-only evidence yields Unknown.
- Webpage Content Verification Prompt: Webpage evidence is evaluated per query, then aggregated into Correct, Incorrect, or Unknown judgments, with secondary verification required for preliminary materials.Only relevant webpage information supports a query decision, and claim-level aggregation follows the same all-correct, any-incorrect, otherwise-unknown rule.