PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Personalized Deep Research: Benchmarks and Evaluations

Yuan Liang, Jiaxian Li, Yuqing Wang, Piaohong Wang, Motong Tian, Pai Liu, Shuofei Qiao, Runnan Fang, He Zhu, Ge Zhang, Minghao Liu, Yuchen Eleanor Jiang, Ningyu Zhang, Wangchunshu Zhou

arXiv:2509.25106v3cs.CLcs.AIcs.IR

TL;DR

Existing deep research evaluations often omit whether research adapts to individual users. PDR-Bench pairs diverse tasks with authentic profiles and uses PQR to assess personalization, quality, and reliability; experiments reveal both strengths and limitations across systems.

  • Problem

    Existing evaluations often use close-ended benchmarks, while open-ended deep research benchmarks remain scarce and typically neglect whether research adapts to users.

    Source: §1 · p. 1

  • Method

    PDR-Bench pairs 50 tasks across 10 domains with 25 authentic user profiles, and PQR evaluates Personalization Alignment, Content Quality, and Factual Reliability.

    Source: §1 · p. 2

  • Results

    Experiments across open-source agents, commercial systems, search-equipped LLMs, and memory systems reveal both strengths and limitations in personalized deep research.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark and PQR framework provide a foundation for developing and evaluating personalized AI research assistants.

    Source: §6 · p. 10

    Persona and context annotations are Chinese-centric, and the main experiments used a selected subset of queries rather than the full benchmark.

    Source: A LIMITATIONS · p. 15

Abstract

from arXiv · show

Deep Research Agents (DRAs) can autonomously conduct complex investigations and generate comprehensive reports, demonstrating strong real-world potential. However, existing evaluations mostly rely on close-ended benchmarks, while open-ended deep research benchmarks remain scarce and typically neglect personalized scenarios. To bridge this gap, we introduce Personalized Deep Research Bench (PDR-Bench), the first benchmark for evaluating personalization in DRAs. It pairs 50 diverse research tasks across 10 domains with 25 authentic user profiles that combine structured persona attributes with dynamic real-world contexts, yielding 250 realistic user-task queries. To assess system performance, we propose the PQR Evaluation Framework, which jointly measures Personalization Alignment, Content Quality, and Factual Reliability. Our experiments on a range of systems highlight current capabilities and limitations in handling personalized deep research. This work establishes a rigorous foundation for developing and evaluating the next generation of truly personalized AI research assistants.

1 INTRODUCTION

Deep research agents can conduct end-to-end investigations, but current evaluations overlook whether reports adapt to users. The paper introduces a benchmark and evaluation framework for assessing personalized deep research, then tests a range of systems.

1 INTRODUCTION

  • Existing deep research benchmarks assess accuracy and comprehensiveness, while personalization benchmarks address narrower tasks rather than complex research.The paper frames personalization as adapting retrieval, reasoning, and reporting to user personas.Source: §1 · p. 2
  • PDR-Bench pairs 50 tasks across 10 domains with 25 real-world user profiles to create 250 user-task pairs.The benchmark is intended to evaluate task complexity and persona-driven adaptation.Source: §1 · p. 2
  • The PQR Evaluation Framework assesses reports on Personalization Alignment, Content Quality, and Factual Reliability.These are presented as three orthogonal dimensions for evaluating report utility.Source: §1 · p. 2
  • Experiments across open-source agents, commercial systems, search-equipped LLMs, and memory systems reveal strengths and limitations in personalization.Source: §1 · p. 2

2 RELATED WORK

Prior work separately evaluates deep research capabilities and personalization, but the related-work review identifies a gap at their intersection: benchmarks for personalized deep research agents.

2.1 EVALUATING DEEP RESEARCH CAPABILITIES

  • Deep research benchmarks assess retrieval, tool use, and report quality, but the reviewed benchmarks lack metrics for aligning research with user goals and preferences.The review distinguishes controlled close-ended suites from open-ended research benchmarks.Source: §2.1 · p. 2

2.2 BENCHMARKING PERSONALIZATION PERFORMANCE

  • Personalization benchmarks cover tasks such as persona adherence, evolving profiles, and response tailoring, yet do not capture complex deep research scenarios.The review concludes that current benchmarks either neglect personalization or fail to capture deep research complexity.Source: §2.2 · p. 3

3 BENCHMARK CONSTRUCTION

PDR-Bench combines expert-designed research tasks with authentic user profiles and relevance-based task-profile pairing to create a standardized testbed for personalized deep research.

3.1 DEEP RESEARCH TASK FORMULATION

  • The task set contains 50 balanced tasks, five per domain, developed with domain experts and reviewed for complexity, clarity, and alignment.A semantically aligned English task set was also created alongside the Chinese set.Source: §3.1 · pp. 3–4

3.2 AUTHENTIC USER PROFILE CONSTRUCTION.

  • Profiles combine structured personas from 25 volunteers with dynamic contexts derived from simulated app interactions and recorded memory snippets.The context data captures evolving interests, habits, and implicit preferences; the complete profile set pairs both profile types.Source: §3.2 · p. 4

3.3 PERSONALIZED DEEP RESEARCH QUERY FORMULATION

  • Volunteers selected personally relevant tasks, then a committee curated pairings for user diversity and alignment, assigning five profiles to each task.This process yielded 250 personalized queries, each joining a research task with an assigned user profile.Source: §3.3 · p. 4

3 BENCHMARK CONSTRUCTION

  • The resulting benchmark is presented as a standardized, reproducible, and scalable testbed for assessing profile integration and personalized research outputs.Its design jointly models task complexity, authentic profile diversity, and motivational alignment.Source: §3.3 · p. 4

4 EVALUATION METHODOLOGY

The PQR Evaluation Framework assesses deep-research reports across personalization alignment, content quality, and factual reliability. Its components use dynamically generated criteria to evaluate user fit, report quality, and factual grounding, then aggregate the three scores.

4 EVALUATION METHODOLOGY

  • PQR jointly assesses personalization alignment, content quality, and factual reliability, addressing evaluations that typically emphasize quality or correctness while neglecting personalization.Figure 2 presents the three dimensions as a multi-dimensional assessment system.Source: §4 · pp. 4–5, Figure 2 · p. 5

4.1 P - PERSONALIZATION ALIGNMENT

  • P-Score dynamically generates task- and persona-specific criteria, weights four personalization dimensions, and scores the report against the resulting sub-criteria.The dimensions are Goal Alignment, Content Alignment, Presentation Fit, and Actionability & Practicality; scores are combined using dynamically generated weights.Source: §4.1 · p. 5
  • P-Score first computes weighted sub-criterion averages within dimensions, then combines dimension scores using the dynamically generated dimension weights.The sub-criterion scores are assigned by a separate LLM scorer.Source: §4.1 · pp. 5–6

4.2 Q – QUALITY OF CONTENT

  • Content Quality evaluates reports independently of user profiles on depth and insight, logical coherence, and clarity and readability.An LLM meta-evaluator weights the three dimensions and generates task-specific sub-criteria; a separate scorer rates the report, and the Q-Score is a hierarchical weighted average.Source: §4.2 · p. 6

4.3 R – FACTUAL RELIABILITY.

  • Factual Reliability extracts and deduplicates verifiable claims, retrieves cited sources for automated support checks, and reports factual accuracy and citation coverage.Factual accuracy measures verified claims supported by their sources, while citation coverage measures the share of claims with explicit citations.Source: §4.3 · p. 6
  • Factual Reliability averages factual accuracy and citation coverage into a single score.Factual accuracy concerns support from cited sources; citation coverage concerns whether factual claims have explicit citations.Source: §4.3 · p. 6

4.4 FINAL SCORE AGGREGATION

  • The final overall score is the arithmetic mean of the personalization, content quality, and factual reliability scores.Source: §4.4 · p. 7

5 EXPERIMENTS

Across the experiments, open-source agents led personalization, while commercial agents showed stronger reliability and consistent quality; adding user information improved personalization, but explicit personas remained more effective than context. Memory systems showed promise but fell short of ideal persona-provided performance, and GPT-5 aligned most closely with human ratings among the judges tested.

5.1 EXPERIMENTAL SETTINGS

  • Evaluation covered 150 representative queries; GPT-5 judged Personalization and Quality, while GPT-5-Mini judged Reliability.The tested systems included commercial and open-source research agents and LLMs with search tools.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • 6.64: OAgents led personalization, including GOAL (6.68), PRES (7.13), and LOGC (7.44), but had factual accuracy of 3.77; MiroFlow and DeerFlow had poor citation coverage.Table 1 reports the Task w/Persona results.Source: §5.2 · p. 7, Table 1 · p. 7
  • Gemini-2.5-Pro Deep Research led commercial systems at 6.58, with FA 8.40 and CC 9.26; commercial agents were reliable and robust in quality but moderately behind open-source agents in personalization.O3 Deep Research followed at 6.11 and led personalization within the commercial category.Source: §5.2 · pp. 7–8
  • Gemini-2.5-Pro w/Search led search-equipped LLMs at 5.53, while GPT-4.1 w/Search had CC 0.10; adding search alone did not match dedicated agents' personalization and quality.Perplexity-Sonar-Reasoning-Pro had FA 8.44 but poor CC and weak personalization.Source: §5.2 · p. 8

5.3 IMPACT OF INFORMATION AVAILABILITY ON PERSONALIZATION

  • Across systems, more user information raised P-Score; explicit personas outperformed context, consistent with agents struggling to extract preferences from implicit context (OAgents GOAL: 6.32 to 6.68).The compared conditions were Task Only, Task w/Context, and Task w/Persona.Source: §5.3 · pp. 8–9

5.4 BOOSTING PERSONALIZATION VIA CONTEXT-AWARE MEMORY SYSTEMS

  • On 50 queries, Mem0, Memory OS, and O-Mem were tested; O-Mem beat the No Memory baseline and peers but remained short of ideal Task w/Persona performance.Table 3 reports memory-system results under Task w/Context using Perplexity Deep Research, prioritizing GOAL and CONT.Source: §5.4 · p. 9, Table 3 · p. 9

5.5 ALIGNMENT WITH HUMAN CONSISTENCY

  • On 15 queries scored by human experts, GPT-5 achieved the highest PCA and lowest MARD at $0.68 per query, so it was selected as the primary judge.PCA measures pairwise agreement and MARD measures mean absolute rating deviation.Source: §5.5 · p. 9, Table 4 · p. 9

6 CONCLUSION

The conclusion presents PDR-Bench as a benchmark for personalized deep research and pairs it with PQR to evaluate personalization, content quality, and factual reliability.

6 CONCLUSION

  • PDR-Bench introduces 250 realistic queries pairing 50 research tasks across 10 domains with 25 authentic user profiles.The authors describe it as the first benchmark of its kind.Source: §6 · p. 10
  • PQR jointly measures personalization, content quality, and factual reliability.Source: §6 · p. 10
  • The study reports both potential and current limitations in personalized deep research, establishing a foundation for developing and benchmarking personalized AI research assistants.Source: §6 · p. 10

7 ETHICS STATEMENT

The study reports ethical review and informed consent for volunteer profiles and evaluations, with anonymization and measures intended to support fairness and confidentiality.

7 ETHICS STATEMENT

  • The study received institutional ethical approval after compliance, privacy, and safety review, and followed applicable research ethics regulations.Source: §7 · p. 10
  • Volunteers provided informed consent, could withdraw, and had their data anonymized before release; the public benchmark contains only de-identified, abstracted profile derivatives without PII.Source: §7 · p. 10
  • Annotators and evaluators participated voluntarily under informed consent and were fairly compensated, while task and profile design sought broad demographic coverage and bias mitigation.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

The reproducibility statement identifies where benchmark, evaluation, and experiment details are documented; the limitations section notes linguistic and experimental scope constraints.

8 REPRODUCIBILITY STATEMENT

  • Benchmark construction, evaluation procedures, systems, configurations, prompts, persona schema, and evaluation dimensions are documented across the paper and appendices.Evaluation data were submitted as supplementary material, with full benchmark release planned after double-blind review.Source: §8 · p. 10

A LIMITATIONS

  • Persona and context annotations were collected in Chinese, constraining the linguistic and cultural scope despite a parallel English version.Source: A LIMITATIONS · p. 15
  • Computational constraints limited the main experiments to a selected subset of queries rather than the full benchmark.The authors plan broader persona construction and context annotation and experiments using all benchmark queries.Source: A LIMITATIONS · p. 15

B EXPERIMENT DETAIL

The experiment details standardize agent configurations and select a subset of queries for memory-system evaluation; the appendix also defines human-consistency metrics and identifies the persona-schema table.

B.1 CONFIGURATION OF METHODS

  • Open-source agents used GPT-5-Mini, with execution limits and search configurations standardized across systems.The setup used SerperAPI and Jina for web search and retrieval; built-in search systems used medium reasoning effort.Source: §B.1 · p. 15

B.2 DATA SELECTION

  • Evaluation narrowed 250 queries to 150 by selecting three personas per task, then chose 50 queries to span personalization demands.Memory-system evaluation emphasizes Goal Alignment and Content Alignment because current systems primarily align content with profiles rather than adapt at the task level.Source: §B.2 · p. 15

C FORMULATION OF HUMAN CONSISTENCY METRICS

D PERSONA SCHEMA

  • Table 5 is identified as the predefined schema for persona collection, with private data anonymized for release.The schema was used to structure explicit ground-truth personas collected from 25 volunteers.Source: Table 5 · p. 16

E DEFINITIONS OF EVALUATION DIMENSIONS

Tables 6 and 7 present dimensions and definitions for personalization and quality evaluation, while Table 8 compares running costs per query.

E DEFINITIONS OF EVALUATION DIMENSIONS

  • Table 6 presents dimensions and definitions for personalization evaluation.Source: Table 6 · p. 17
  • Table 7 presents dimensions and definitions for quality evaluation.Source: Table 7 · p. 17

F RUNNING COST

  • Table 8 compares running costs across models and personalization settings on a per-query basis.Source: Table 8 · p. 17

G A MORE DETAILED EXPERIMENTS RESULTS

The detailed results table covers three personalization settings and marks the best and second-best scores; Figure 4 is a case study of personalized deep research.

G A MORE DETAILED EXPERIMENTS RESULTS

  • Table 9 reports personalization results for Task Only, Task w/Context, and Task w/Persona configurations.Source: Table 9 · p. 18
  • Table 9 highlights the best score in each column in bold and underlines the second-best.Source: Table 9 · p. 18

H CASE STUDY

  • Figure 4 presents a case study of personalized deep research.Source: Figure 4 · p. 19

I DATA ANNOTATION

This section describes annotation prompts and procedures for evaluating personalized research reports, covering task-specific quality criteria and the extraction of verifiable factual claims with citations.

I DATA ANNOTATION

  • Six trained annotators labeled 5,939 instances in 85 person-days at an approximate $6,000 budget.Each persona receives a separate aimemory database, with each item titled and associated with its persona.Source: I DATA ANNOTATION · p. 20
  • Context annotation prioritizes persona-reversible, diverse, high-quality data, including at least five items per preference and 20–50% conversational content.Collected content should plausibly reflect the persona without contradicting its attributes; quality takes precedence over quantity.Source: I DATA ANNOTATION · p. 20

J.1 PROMPTS IN PERSONALIZATION EVALUATION

  • Personalization scores combine task- and persona-specific weights across Goal Alignment, Content Alignment, Actionability & Practicality, and Presentation Fit, with weights summing to 1.0.Evaluators are instructed to explain how the task and persona justify each weight.Source: §J.1 · p. 21
  • Goal Alignment criteria should target a persona’s explicit and implicit needs and assess adaptation in style, depth, perspective, and practicality.Criteria require explanations and weights tied to the particular task and persona, with higher weights for closer alignment.Source: §J.1 · pp. 22–23
  • Content Alignment criteria assess persona-linked content selection and depth, prioritize distinctive preferences, and avoid overlap with other evaluation dimensions.The prompts call for task- and persona-based analysis, criterion rationales, and weights summing to exactly 1.0.Source: §J.1 · pp. 24–25

J.2 PROMPTS IN QUALITY EVALUATION

  • Quality evaluation dynamically weights Depth & Insight, Logical Coherence, and Clarity & Readability according to the task, with weights totaling 1.0.The prompt requires a task-linked rationale for the allocation.Source: §J.2 · pp. 27–28
  • Depth & Insight criteria must be task-specific, explain their rationale, assign weights summing to 1.0, and avoid overlap with coherence or readability.The criteria should assess analytical depth, rigor, originality, and value rather than merely listing information.Source: §J.2 · p. 29
  • Articles are evaluated across Depth & Insight, Logical Coherence, and Clarity & Readability, with each criterion analyzed for strengths and weaknesses and assigned an integer score from 0 to 10.Source: §J.2 · p. 30

J.3 PROMPTS IN RELIABILITY EVALUATION

  • Claim extraction includes verifiable statements about the external world, such as data, past events, classifications, and research findings, while excluding intentions, plans, self-referential content, predictions, opinions, and methods.The prompt requires complete, understandable factual statements and advises caution when classification is uncertain.Source: §J.3 · pp. 31–32
  • Each extracted fact is returned with a reference index and URL, handling multiple citations as separate triples and leaving both fields empty when a fact has no citation.For linked sources embedded in text, the index is set to 0; for location-qualified numeric citations, only the first numeric part is used.Source: §J.3 · pp. 32–33
Loading 2509.25106v3…