PaperlayerView exact revision on arXiv

Source-linked AI summary

LifeSearch: Benchmarking Deep Research Agents for Real-World Life and Productivity

Yibo Wang, Senjie Jin, Yansong Ning, Xinbing Liang, Haotian Luo, Jiazheng Zhang, Yuxuan He, Will Wu, Clarenceai

arXiv:2610.18092v1cs.AI

TL;DR

Real-world life and professional productivity settings remain underrepresented in deep-research evaluation. LifeSearch builds a bilingual benchmark with expert-written rubrics and evaluates 12 agents; the strongest scores only 63.4%.

  • Problem

    Deep-research agents’ capabilities in real-world life and professional productivity settings are insufficiently evaluated by existing benchmarks.

    Source: §1 · p. 2

  • Method

    LifeSearch constructs 203 Chinese and English tasks across seven domains from real-user query seeds, with 4,014 expert-annotated rubrics, and evaluates agents using a unified harness.

    Source: §1 · p. 2, §3.3 · p. 5, §5 · p. 9

  • Results

    Across 12 agents, all systems fall below 70% across domains, and GPT-5.6 Sol scores 63.4%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LifeSearch offers an evaluation framework intended to support more reliable and effective DR agents for everyday life and professional work.

    Source: §5 · p. 9

    Iteration counts include tool actions beyond search, and cross-model comparisons do not isolate the causal effect of additional searches.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Deep research (DR) agents address open-ended information needs through multi-step reasoning, long-horizon web search, and evidence synthesis. However, existing benchmarks primarily focus on academic research or synthetic tasks, leaving DR agents' capabilities in real-world life and professional productivity settings insufficiently evaluated. To bridge this gap, we introduce LifeSearch, a challenging benchmark comprising 203 Chinese and English tasks derived from real-user query seeds across 7 concrete life and productivity domains. With more than 4,500 hours of human annotation, we construct over 4,000 expert-written, fine-grained rubrics, covering both report-content criteria for coverage and accuracy and concrete criteria specifying values, units, dates, and URLs. We conduct extensive experiments on 12 advanced DR agents under a unified evaluation harness, systematically analyzing their performance patterns and failure modes. We find that even the best-performing frontier agent, GPT-5.6 Sol, achieves a score of only 63.4%, leaving substantial room for improvement on realistic life and professional DR tasks. We select the grading configuration by comparing candidate LLM judges against human rubric labels, and a human ranking study shows strong alignment between aggregate benchmark scores and human judgments (Spearman's $\rho = 0.89$, Kendall's $\tau = 0.81$). LifeSearch provides a rigorous evaluation framework and foundation for advancing DR agents toward real-world life and productivity.

1 INTRODUCTION

Existing deep-research benchmarks offer limited coverage of real-world life and professional productivity settings. LifeSearch targets this gap with a bilingual benchmark and finds substantial performance headroom among frontier agents.

1 INTRODUCTION

  • Existing deep-research benchmarks provide limited coverage of real-world life and professional productivity, often focusing instead on academic research or synthetic tasks.Earlier evaluation approaches range from short-form exact-match questions to open-ended reports graded with structured rubrics or agent-as-judge systems.Source: §1 · p. 2
  • LifeSearch comprises 203 tasks across seven life and productivity domains, built from crowd-written query seeds refined by experts and assessed with more than 4,000 fine-grained rubrics.Construction involved more than 4,500 hours of human labor; the rubrics assess report coverage and accuracy, including concrete values, units, or thresholds.Source: §1 · pp. 2–3
  • GPT-5.6 Sol scores 63.4%, while all evaluated systems fall below 70% across domains; autonomous runs also fail through premature clarification, out-of-window evidence, and omitted requirements.Comparable token counts do not guarantee comparable quality: Claude Opus 5 and GLM-5.3 differ by 6.9 percentage points.Source: §1 · p. 2
  • The benchmark’s overview depicts a pipeline from crowd-worker query seeds through expert refinement and rubric construction to LLM-judge scoring.The introduction reports that 11 judge configurations were compared against 402 human-annotated rubric labels, and a human ranking study found Spearman’s ρ = 0.89 and Kendall’s τ = 0.81.Source: §1 · p. 2, Figure 2 · p. 3

2 RELATED WORK

Deep-research agents and their evaluation have developed from web-browsing assistants and short-answer benchmarks toward long-horizon search and rubric-based assessment. Yet real-world life and professional productivity tasks remain underexplored.

Deep Research Agent

  • Deep-research agents extend web-browsing assistants by interleaving multi-step reasoning with iterative search and synthesizing retrieved evidence into reports; later systems and benchmarks target harder, longer-horizon information seeking.Examples include GAIA (Mialon et al., 2023), WebArena, AssistantBench, Search-R1 (Jin et al., 2025), and BrowseComp (Wei et al., 2025).Source: §2 · p. 3

Deep Research Agent Evaluation

  • Evaluation has shifted from verifying short exact answers toward multi-criterion rubrics, agent-as-judge methods, active fact-checking, and structured-table deliverables.Examples include DeepResearch Bench, ResearchRubrics, DRACO (Zhong et al., 2026), Mind2Web 2 (Gou et al., 2025), DeepResearchEval (Wang et al., 2026b), LiveResearchBench (Wang et al., 2026a), and WideSearch.Source: §2 · p. 3
  • Despite these developments, most reviewed benchmarks focus on academic or synthetic tasks, and their rubrics lack the numerical grounding and independent verifiability needed for usable life or work deliverables.Source: §2 · p. 3

3 OVERVIEW OF LIFESEARCH

LIFESEARCH brings bilingual real-world life and productivity tasks together with expert-written rubrics and evaluates reports through weighted rubric judgments and a human-checked judge configuration.

3 OVERVIEW OF LIFESEARCH

  • LIFESEARCH comprises 203 tasks—102 Chinese and 101 English—across seven domains spanning specialized professional research and everyday information needs.Figure 3 summarizes task counts by domain and rubric statistics.Source: §3 · p. 3, §3.1 · p. 4, Figure 3 · p. 4

3.1 DATA COLLECTION AND TASK DOMAINS

  • Crowd-worker query seeds are filtered for intent and quality, classified by domain, rewritten to remove operational actions, and manually screened for complexity, diversity, clarity, and feasibility.General time-sensitive searches remain eligible during filtering.Source: §3.1 · p. 4

3.2 EXPERT-GUIDED TASK REFINEMENT

  • Experts decompose seeds into typically two to four subtasks, enrich them around selected capabilities, and rewrite them as coherent life- or work-scenario queries.Subtasks cover lookup, computation, comparison, or reasoning; enrichment targets search planning, information synthesis, conflict resolution, or uncertainty handling.Source: §3.2 · pp. 4–5

3.3 RUBRIC DESIGN

  • 4,014 rubrics (19.8 per task), built with over 4,500 annotation hours, span seven dimensions; about 40% specify concrete targets, while open-ended tasks allow alternatives.Rubrics assess requirements such as coverage, accuracy, evidence quality, reasoning, and presentation.Source: Figure 3 · p. 4, §3.3 · p. 5

3.4 EVALUATION METHODOLOGY

  • An LLM judge labels each rubric Pass/Fail, then weighted satisfaction is normalized per task and averaged equally across tasks.Rubric weights reflect their relative importance within a task.Source: §3.4 · p. 6

3.5 JUDGE MODEL SELECTION IN DR SETTING

  • 84.25% mean accuracy: Gemini 3.1 Pro with low reasoning effort and zero temperature was best among tested configurations and was selected for benchmark grading.Table 3 compares 11 configurations on 402 human-labeled rubrics.Source: Table 3 · p. 7, §3.5 · p. 7
  • 99.0% self-consistency can coexist with 75.62% mean accuracy, while majority voting improves mean accuracy by at most 0.42 percentage points.The selected evaluation configuration therefore uses a single judge pass per report.Source: §3.5 · p. 7

4 EXPERIMENTS

The experiments evaluate 12 frontier agents on LifeSearch using a shared harness, and find substantial performance headroom alongside distinct domain strengths and recurring failure modes. Report scores align strongly with human rankings, while retrieval and report length alone do not determine quality.

4.1 EXPERIMENTAL SETUP

  • The 12 agents run in a shared ReAct-style harness with web search, page fetching, and code execution; reports are graded once and scores average three runs.The reported score and resource statistics are per-task means; output tokens include generation throughout execution.Source: §4.1 · pp. 7–8, Table 4 · p. 8

4.2 MAIN RESULTS

  • 63.4% is GPT-5.6 Sol’s overall score, ahead of Claude Opus 5 at 56.1%, while every other system scores below 50%.The strongest agent scores below 50% on 41 of 203 tasks (20.2%).Source: Table 4 · p. 8, §4.2 · p. 8
  • GPT-5.6 Sol and Claude Opus 5 rank first and second in all seven domains, but no system exceeds 70% in any domain.GLM-5.3 scores 51.9% versus Hy4 Preview’s 47.1% on everyday facts, while Hy4 Preview scores 51.4% versus 49.0% on academic and professional literature; the authors describe these differences as descriptive, especially for the nine-task everyday-facts subset.Source: §4.2 · p. 8

4.3 PERFORMANCE PATTERNS

  • Mean interaction count correlates positively with overall score (Spearman’s ρ = 0.76), consistent with sustained evidence collection, though cross-model comparisons do not isolate search effects.GPT-5.6 Sol averages 86.2 iterations and scores 63.4%, whereas GLM-5.2 averages 11.0 iterations and scores 37.1%; iterations also include other tool actions.Source: §4.3 · pp. 8–9
  • Final-report length correlates positively with score (Spearman’s ρ = 0.60), but longer reports do not ensure quality without relevant, correct, and complete information.Doubao-Seed-2.1-Pro and Gemini-3.6-Flash produce 12,089 and 10,256 characters per report while scoring 43.1% and 34.7%; Hy4 Preview scores 49.7% with 9,959 characters.Source: §4.3 · p. 9

4.4 FAILURE ANALYSIS

  • Recurring failures include premature termination, temporal-scope violations, and incomplete factual coverage.In the autonomous harness, follow-up questions can end research without a report; current data can conflict with an earlier requested cutoff; and fluent summaries can omit requested entities, values, dates, sources, or edge cases.Source: §4.4 · p. 9

4.5 HUMAN AGREEMENT

  • On 30 tasks, rubric-based and human rankings correlate strongly (Spearman’s ρ = 0.89, Kendall’s τ = 0.81), with identical top-four ordering among seven systems.Annotators ranked anonymized reports for factual accuracy, completeness, and formatting quality.Source: §4.5 · p. 9

5 CONCLUSION

The paper presents LifeSearch as a benchmark for deep research agents on real-world life and productivity tasks. Its 203 bilingual tasks and 4,014 expert-annotated rubrics support evaluation across seven domains, while experiments show that the strongest tested agent scores only 63.4%.

5 CONCLUSION

  • LifeSearch comprises 203 Chinese and English tasks across seven life and productivity domains, with 4,014 expert-annotated rubrics assessing requirement coverage and factuality.The authors describe the benchmark as an evaluation framework for deep research agents on real-world life and productivity tasks.Source: §5 · p. 9
  • Across 12 frontier agents evaluated in a unified harness, the top system scores only 63.4%, and greater output volume does not consistently ensure higher quality.Source: §5 · p. 9
  • The authors report validating their grading protocol against human rubric labels and holistic report rankings, and hope LifeSearch supports more reliable and effective agents for everyday life and professional work.Source: §5 · p. 9

A RUBRIC DESIGN DETAILS

Table 6 expands the rubric overview with annotation fields, authoring requirements, and illustrative examples. Its examples concern fictional subscription plans A and B.

A RUBRIC DESIGN DETAILS

B. Annotation fields

Each rubric criterion is a complete, independently judgeable acceptance statement, introduced as “The response should . . .” followed by a concrete requirement. The annotation fields distinguish checkability, necessity, priority, and relative weight.

B. Annotation fields

  • A criterion is a complete, independently judgeable acceptance statement.Source: B. Annotation fields · p. 13
  • The criterion text begins “The response should . . .” and states the concrete requirement.Source: B. Annotation fields · p. 13
  • Type records whether a judgment is objective or subjective; verifiable prices are objective, while stylistic preferences are subjective.Source: B. Annotation fields · p. 13
  • Necessity marks whether a requirement is explicit in the query or reasonably implied.A requested price comparison is an example of an explicit requirement.Source: B. Annotation fields · p. 13
  • Priority retains a qualitative importance label as input to the judge, while weight records a requirement’s relative importance within its task.Priority labels are MUST HAVE or NICE TO HAVE; scoring uses the separately recorded numerical weight.Source: B. Annotation fields · p. 13

C. Rubric-writing requirements

The rubric-writing requirements emphasize independently testable criteria with explicit targets and scope. They also favor affirmative wording and allow reasonable alternatives for open-ended tasks.

C. Rubric-writing requirements

B DATA COLLECTION AND EVALUATION PROMPT TEMPLATES

The appendix provides translated data-collection templates and language-specific evaluation prompts. Together, they specify query screening, domain labeling, minimal task rewriting, rubric judging, and search-agent instructions.

B DATA COLLECTION AND EVALUATION PROMPT TEMPLATES

  • The appendix translates filtering, domain-classification, and rewriting prompts, and supplies Chinese and English judge templates plus the search-agent system prompt.Judges label rubric satisfaction; annotator-assigned weights are applied separately during score aggregation.Source: B DATA COLLECTION AND EVALUATION… · p. 14

B.1 INTENT AND QUALITY FILTERING

  • Filtering judgments assess task quality and search relevance, excluding unclear, non-search, or low-quality requests.Source: §B.1 · p. 14
  • A query is time-dependent for filtering when it presupposes a specific unstated recent event; general time-sensitive requests without that premise remain false cases.Source: §B.1 · pp. 14–15

B.2 DOMAIN CLASSIFICATION

  • The English-query classifier assigns one of seven domains according to the query’s core purpose, including literature, company research, policy, and everyday facts.Source: §B.2 · p. 15

B.3 TASK REWRITING

  • Rewriting removes local-file dependencies and engineering operations with minimal edits while preserving search intent, sources, recency, wording, and content organization.Source: §B.3 · pp. 16–17

B.4 ENGLISH JUDGE PROMPT

  • The English judge returns a binary criterion-satisfaction label and reason, failing responses that omit required content or provide incorrect information.Source: §B.4 · pp. 17–18

B.5 CHINESE JUDGE PROMPT

  • The Chinese judge likewise returns a binary satisfaction label and brief reason, with criterion type, necessity, and priority supplied as fields.Source: §B.5 · p. 18

B.6 DEEP RESEARCH AGENT SYSTEM PROMPT

  • The search-agent prompt treats user dates as hard cutoffs, requires tool use, and instructs agents to enumerate explicit sub-requirements before finalizing.The execution-time date is supplied through a placeholder.Source: §B.6 · p. 19

C RESPONSE LENGTH AND EVALUATION TIMING

The section describes how final-answer length is measured and contextualizes the timing of the model evaluations.

C RESPONSE LENGTH AND EVALUATION TIMING

D MODEL RESPONSES AND RUBRIC JUDGMENTS

Four recorded trajectories pair selected rubric judgments with full-report scores, illustrating both satisfied requirements and concrete omissions. The examples span workflow comparison, literature screening, repository profiling, and localization-method comparison.

D MODEL RESPONSES AND RUBRIC JUDGMENTS

  • The four examples show selected Pass/Fail judgments, while reported scores and passed-rubric counts refer to each complete report and rubric set.Quoted model claims are not independently verified in the examples.Source: D MODEL RESPONSES AND RUBRIC JUDGMENTS · p. 20

D.1 CASE 1: DIFY WORKFLOW AND CHATFLOW

  • 60.38%: Hy4 Preview passed 14/18 rubrics, but its Workflow start-node answer omitted that User Input and Trigger are mutually exclusive.Source: §D.1 · p. 20

D.2 CASE 2: WINTER-WHEAT LITERATURE SCREENING

  • 57.53%: Hy4 Preview passed 11/15 rubrics, yet its winter-wheat literature list included a paper explicitly about spring wheat.The example notes that a relevant search query does not ensure every selected paper meets the crop requirement.Source: §D.2 · p. 21

D.3 CASE 3: A TIME-BOUNDED REPOSITORY PROFILE

  • 41.55%: DeepSeek-V4 Pro passed 14/26 rubrics, but its API data dated August 21, 2026, fell after the requested July 20 cutoff.The report distinguished the GitHub watchers field from subscribers, but the live snapshot was out of scope by date.Source: §D.3 · p. 22

D.4 CASE 4: UWB LITERATURE AND METHOD COMPARISON

  • 62.07%: DeepSeek-V4 Pro passed 14/18 rubrics and supplied a Chan/WLS reference, but omitted the required comparison of three localization algorithms.Source: §D.4 · p. 23

E BILINGUAL TASK AND RUBRIC EXAMPLES

This appendix presents one Chinese and one English LifeSearch task with selected rubrics, illustrating the benchmark’s task wording and evaluation criteria.

E BILINGUAL TASK AND RUBRIC EXAMPLES

E.1 CHINESE TASK: KUNSHAN POPULATION TIME SERIES

  • The Kunshan task requests a five-year population time series, including demographic and district-level indicators, source formats, gaps, and changes in statistical definitions or district boundaries.Source: §E.1 · p. 24
  • The rubric makes the stated definition of resident population an explicit, objective must-have criterion.Source: §E.1 · p. 24
  • Providing reproducible source URLs for population data is an objective, implicit nice-to-have criterion.Source: §E.1 · p. 24

E.2 ENGLISH TASK: UNIVERSITY TUITION AND ADMISSIONS COMPARISON

  • The English task compares University of Oregon and UCF tuition and admitted-student GPA/SAT figures using specified sources, while flagging conflicts, gaps, and non-standard ranges.Source: §E.2 · p. 24
  • The rubric prioritizes official university and NCES/IPEDS data, permitting clearly marked CollegeRaptor or PrepScholar backups when official data are missing.Source: §E.2 · p. 24
  • The rubric defines SAT midpoint as the arithmetic mean of the 25th and 75th percentiles, not a reported median, and asks that the rule be stated.Source: §E.2 · p. 24
  • The response should use a table with one row per school to align the institutional indicators for comparison.Source: §E.2 · p. 24

Related papers

Loading 2610.18092v1…