PaperlayerView exact revision on arXiv

Source-linked AI summary

Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

Serin Kim, Sangam Lee, Dongha Lee

arXiv:2602.17003v3cs.CLcs.AI

TL;DR

Practical web agents must resolve underspecified requests from users’ implicit preferences, but existing benchmarks do not provide realistic context or ambiguity. Persona2Web addresses this gap with a real-open-web benchmark built around implicit histories, ambiguous queries, and reasoning-aware evaluation; experiments show that history access alone reaches only 13% success at best.

  • Problem

    Existing benchmarks lack realistic user context and ambiguous queries needed to evaluate whether web agents infer and apply implicit preferences.

  • Method

    Persona2Web combines long-term implicit user histories, deliberately ambiguous queries, and reasoning-aware rubric-based evaluation for personalized agents on the real open web.

  • Results

    13% success rate at best is achieved when user history is provided, while all agents achieve 0% without it on ambiguous queries.

  • Takeaways & Limitations

    The results show that user-history access alone is insufficient and that personalized web agents require effective use of context alongside evaluation beyond task completion.

  • Takeaways & Limitations

    The benchmark uses synthetic user profiles and browsing histories rather than real user data, while deployed personalization would raise privacy and safety concerns.

Abstract

from arXiv · show

Large language models have advanced web agents, yet current agents lack personalization capabilities. Since users rarely specify every detail of their intent, practical web agents must be able to interpret ambiguous queries by inferring user preferences and contexts. To address this challenge, we present Persona2Web, the first benchmark for evaluating personalized web agents on the real open web, built upon the clarify-to-personalize principle, which requires agents to resolve ambiguity based on user history rather than relying on explicit instructions. Persona2Web consists of: (1) user histories that reveal preferences implicitly over long time spans, (2) ambiguous queries that require agents to infer implicit user preferences, and (3) a reasoning-aware evaluation framework that enables fine-grained assessment of personalization. We conduct extensive experiments across various agent architectures, backbone models, history access schemes, and queries with varying ambiguity levels, revealing key challenges in personalized web agent behavior. For reproducibility, our codes and datasets are publicly available at https://serin-kimm.github.io/Persona2Web/.

1. Introduction

PERSONA2WEB addresses the lack of realistic personalization benchmarks by combining implicit user histories, ambiguous queries, and reasoning-aware evaluation. Experiments show that agents cannot resolve ambiguity without user history, while history access alone yields only marginal success.

  • Motivation: Existing web-agent benchmarks omit realistic user context and rely on fully specified queries, preventing evaluation of preference-based ambiguity resolution.They therefore cannot distinguish agents that genuinely apply user preferences from agents that merely follow explicit instructions.
  • Benchmark: Its clarify-to-personalize design requires agents to infer missing query details from preferences revealed implicitly across long-term user histories.Queries intentionally conceal website or preference constraints at multiple ambiguity levels, with Level 2 as the target personalized setting.
  • Evaluation: The benchmark combines ambiguous queries with reasoning-aware evaluation to assess personalization beyond simple task completion.The framework examines reasoning traces through structured rubrics, enabling fine-grained assessment.
  • Results: 0% success rate occurs without user history, while performance reaches just 13% at best when history is provided.These results show that supplying history alone is insufficient for effective personalization.
  • Benchmark: PERSONA2WEB is the first benchmark for evaluating personalized web agents on the real open web.It evaluates personalization through user histories, personalization query sets, and a dedicated evaluation framework.

2. Related work

Prior web-agent benchmarks use simulated, cached, or open-web environments but generally evaluate actions or outcomes without modeling personalization. Existing personalization benchmarks likewise do not test preference inference within web navigation and execution.

  • Web-agent benchmarks: Prior web-agent benchmarks span simplified, simulated, cached, and real open-web environments with corresponding evaluation methods.Examples include MiniWOB++, WebArena, Mind2Web, WebCanvas, and WebVoyager.
  • Evaluation gaps: Action-wise and outcome-based evaluation cannot distinguish navigation errors from personalization deficiencies or other reasoning failures.Open-web tasks can admit multiple valid trajectories, making single-path action assessment especially limited.
  • Personalization benchmarks: Personalization benchmarks for memory-augmented, dialogue, and function-call agents evaluate user-relevant information but not web-interface navigation.PersonalWAB uses user history, yet focuses on abstract function calls rather than web interactions.
  • Research gap: PERSONA2WEB targets the unresolved setting where agents infer preferences from interaction histories and apply them during web action planning and execution.This bridges personalization evaluation with open-web navigation scenarios.

3. PERSONA2WEB

PERSONA2WEB is a benchmark for personalized web agents in realistic open-web settings, combining implicitly revealing user histories, ambiguity-controlled queries, and reasoning-aware evaluation. Its design distributes preference evidence across noisy, long-span browsing behavior and distinguishes personalization from navigation and task-completion failures.

  • PERSONA2WEB evaluates personalized web agents through user histories, ambiguous queries, and reasoning-aware evaluation in realistic open-web environments.The benchmark is organized around these three components.
  • 3.1. Realistic User History: User histories encode preferences through browsing actions rather than explicit linguistic cues, requiring agents to infer patterns from behavior.Histories include timestamps, action types, objects, and websites across searches, visits, purchases, bookings, and reviews.
  • 3.1. Realistic User History: 50 distinct user profiles are generated from demographic information and domain preferences across 21 web domains.The generation process selects relevant domains and produces domain-specific preferences and rationales.
  • 3.1. Realistic User History: Event seeds generate recurring high- and low-frequency activities, while actions are dispersed across a year to require integration of remote history.Approximately 10% of records receive cancellations or modifications to reflect real-world noise and inconsistencies.
  • 3.2. Ambiguous Query Sets: Query sets vary ambiguity by masking website and preference cues, with Level 2 masking both and serving as the target for personalization.Level 0 is fully specified, Level 1 masks the website, and Level 2 masks both website and preference constraints.
  • 3.2. Ambiguous Query Sets: Only 3.56% of history entries contain preference values as exact string matches, supporting evaluation of implicitly encoded preferences.Human evaluation also reports that the multi-stage pipeline produces more realistic histories than naive generation.
  • 3.3. Reasoning-aware Evaluation: The evaluation framework separates personalization from navigation using personalization scores, intent satisfaction, and success rate over full reasoning trajectories.Pweb measures website alignment, Ppref measures preference alignment, and each score separates retrieval accuracy from utilization accuracy.

4. Personalized Web Agent

The benchmark adapts generic web-agent architectures for personalized navigation by adding history retrieval and reasoning components. It compares dynamic on-demand access with pre-execution retrieval to test different forms of personalization planning.

  • AgentOccam and Browser-Use2 are adapted because their original designs do not account for user history.The adapted process introduces components for planning, retrieval, and generation over relevant history.
  • The planner decides whether history is needed, the retriever searches relevant entries, and the generator uses them to resolve query ambiguities.The resulting trajectory records actions, reasoning, accessed histories, disambiguation, and final option selection.
  • History Access Scheme: On-demand access retrieves history during execution when needed, testing real-time recognition of personalization requirements.This scheme evaluates situational awareness about when and what information to retrieve.
  • History Access Scheme: Pre-execution access retrieves relevant histories before execution, testing long-horizon planning about information needed later.The agent generates multiple queries before execution to resolve ambiguities in advance.

5. Experiments

Experiments show that personalized web agents struggle to resolve and use implicit preferences, especially as ambiguity increases. Reasoning-aware evaluation separates personalization from navigation and aligns better with human judgment than action-wise or outcome-based evaluation.

  • Evaluation: The reasoning-aware method achieves the highest correlation with human judgments across all metrics, with reasoning traces producing a gap over action-wise evaluation.The meta-evaluation compares reasoning-aware, action-wise, and outcome-based schemes against human judgments.
  • Main Results: Browser-Use consistently achieves higher Pweb and Ppref than AgentOccam when history access is enabled.The paper attributes this difference to Browser-Use preserving richer information through a full enhanced DOM tree.
  • Main Results: Proprietary models achieve 6–13% success rates under history access, while most open-source models remain below 5%.Qwen3-80B-Instruct is the only open-source model reported to approach proprietary-level performance in specific settings.
  • Evaluation: Task completion alone cannot distinguish personalization failures from navigation failures because agents with the same SR = 0.02 fail for opposite reasons.The reasoning-aware framework evaluates personalization and navigation separately.
  • Ambiguity: Success rates decline from 23.8% on clear queries to 16.3% at level 1 and 7.8% at level 2 ambiguity.This decline persists despite access to a personalization module and user history.
  • Utilization: Even clear queries yield average preference and website scores of only 0.92 and 0.97 because agents omit constraints or apply information at the wrong step.The findings indicate that personalization requires improving utilization as well as retrieval.
  • Robustness: Repeated executions have a maximum standard deviation of 0.025, and GPT-4.1 remains ahead of Qwen-80B-Instruct across measurements three months apart.These results support stable scoring despite open-web dynamics.

6. Error Analysis

The error analysis identifies failures in retrieving, interpreting, and applying user history. These errors range from vague retrieval queries and fabricated preferences to missed evidence and failure to use correctly retrieved information.

  • Redundant history access: Underspecified personalization queries cause repeated history retrieval attempts because they fail to identify relevant evidence.Adding domain-aware attributes such as activity type, activity level, and price range enables relevant retrieval.
  • Personalization hallucination: Personalization hallucination occurs when agents fabricate user information instead of grounding preferences in retrieved history.The analysis highlights fabricated rating or pricing thresholds as common examples.
  • History retrieval failure: History retrieval failure occurs when relevant but implicit information, such as a favorite team, is not connected to the user’s preference.The Baltimore Orioles example appears in multiple histories without explicitly using the phrase “favorite team.”
  • History utilization failure: History utilization failure occurs when the correct information is retrieved but omitted from the agent’s action.The agent retrieves the user’s Los Angeles location but fails to incorporate it into search.

7. Conclusion

Persona2Web establishes a benchmark for evaluating personalized web agents on the real open web. It combines implicit user histories, ambiguous queries, and reasoning-aware scoring to assess how agents infer and apply preferences.

  • Conclusion: Persona2Web combines clarify-to-personalize queries, rigorously constructed user histories, and rubric-based reasoning-aware evaluation.The benchmark evaluates personalization across architectures, backbone models, history access, ambiguity, and user context formats.

Impact Statement

The benchmark uses synthetic profiles and browsing histories rather than real user data. The paper identifies privacy, consent, data protection, and harmful open-web content as practical considerations for deployed personalized agents.

  • Impact Statement: Synthetic user profiles and browsing histories avoid collecting real user data, but deployment would require consent and data-protection safeguards.The paper also notes that real open-web execution may expose agents to harmful content.

A.2. Dataset Validation

PERSONA2WEB validates its generated user histories through automated checks, human review, diversity analysis, and pairwise realism judgments. The histories show broad topical coverage, substantial per-user lexical diversity, and greater perceived realism than histories generated without the pipeline.

  • GPT-5 verification and human review checked generated profiles for instruction adherence, formatting, temporal consistency, preference alignment, plausibility, and behavioral coherence.
  • The dataset measures history diversity globally, across users, and within individual users using lexical metrics and semantic topic clustering.
  • 4.503 topic entropy and 0.978 Shannon evenness indicate broad, balanced global topical coverage across user histories.
  • 9.932 per-user Ent-4 exceeds the prior leading score of 8.72, indicating high lexical diversity within individual histories.
  • Pairwise human evaluation compares PERSONA2WEB histories with histories generated without the multi-stage pipeline on sequence and behavioral realism.

B.1. Human Inter-annotator Agreement

The evaluation framework checks both agreement among human annotators and consistency across repeated or different LLM judges. These analyses support the reliability of the reported evaluation scores.

  • Human meta-evaluation uses independent annotations, average agreement, and Gwet’s AC1 to assess agreement among annotators.
  • GPT-5-mini produced stable repeated scores, with standard deviations no greater than 0.015 across metrics and history access schemes.
  • Agreement across GPT-5-mini, Qwen-80B-Instruct, and GPT-OSS-120B tests robustness to the choice of LLM judge.

C. Implementation Details

The implementation evaluates personalized web agents through different observation pipelines, history-retrieval mechanisms, access schemes, and rubric-based metrics. It separates personalization from task completion and requires perfect scores across all three metrics for overall task success.

  • Observation and navigation: AgentOccam condenses observations by retaining pivotal accessibility-tree nodes, whereas Browser-Use constructs a filtered full DOM representation from CDP data.
  • History retrieval: History entries are indexed with type and object attributes, and Stella V5 1.5B with FAISS retrieves up to 20 nearest items using cosine similarity.
  • History access: On-demand access tests whether agents recognize when personalization is needed, while pre-execution tests anticipation and integration of history before navigation.
  • Evaluation metrics: The evaluation scores website personalization, preference personalization, and intent satisfaction as separate dimensions.
  • Evaluation metrics: A task is successful if and only if website, preference, and intent metrics all achieve perfect scores.
  • Retriever ablation: The retriever ablation compares condensed type-and-object keys with keys incorporating timestamp, type, object, and website attributes.

I. Error Statistics

Error analysis shows that personalization failures remain substantial even when agents navigate successfully. For Gemini 2.5 Flash, retrieval failures are the most common personalization error, followed by utilization failures and redundant history access.

  • Error analysis separates personalization-related errors from non-personalization-related errors using preference, website, and intent scores.
  • Across AgentOccam backbone models, personalization-related errors form a substantial share of failures, showing that navigation improvements alone are insufficient.
  • For Gemini 2.5 Flash, history retrieval failure is the most prevalent personalization error, accounting for nearly half of personalization errors.
  • Representative execution logs illustrate redundant access, hallucination, retrieval failure, and utilization failure during agent execution.
Loading 2602.17003v3…