Source-linked AI summary
AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping
Sunghwan Kim, Ryang Heo, Yongsik Seo, Jinyoung Yeo, Dongha Lee
TL;DR
Web shopping creates fragmented information and cognitive burden, while existing benchmarks largely omit exploratory scenarios and personalization. AgenticShop addresses this gap with realistic shopping intents, user profiles, and checklist-driven evaluation, finding that current agentic systems remain insufficient for personalized product curation.
Problem
Existing benchmarks focus on simplified single-platform lookups and do not explicitly evaluate whether agents satisfy diverse, fine-grained shopping preferences.
Method
AgenticShop benchmarks open-web product curation using realistic shopping intents, purchase-history-based user profiles, tailored queries, and verifiable personalization checklists.
Results
Experiments show that current agentic systems remain insufficient for personalized product curation across the open web.
Takeaways & Limitations
Effective personalized curation requires exploratory information seeking across diverse sources and careful comparison of alternatives.
Takeaways & Limitations
The benchmark addresses open-web shopping but identifies current limitations in evaluating realistic exploratory scenarios and personalization.
Abstract
from arXiv · showhide
The proliferation of e-commerce has made web shopping platforms key gateways for customers navigating the vast digital marketplace. Yet this rapid expansion has led to a noisy and fragmented information environment, increasing cognitive burden as shoppers explore and purchase products online. With promising potential to alleviate this challenge, agentic systems have garnered growing attention for automating user-side tasks in web shopping. Despite significant advancements, existing benchmarks fail to comprehensively evaluate how well agentic systems can curate products in open-web settings. Specifically, they have limited coverage of shopping scenarios, focusing only on simplified single-platform lookups rather than exploratory search. Moreover, they overlook personalization in evaluation, leaving unclear whether agents can adapt to diverse user preferences in realistic shopping contexts. To address this gap, we present AgenticShop, the first benchmark for evaluating agentic systems on personalized product curation in open-web environment. Crucially, our approach features realistic shopping scenarios, diverse user profiles, and a verifiable, checklist-driven personalization evaluation framework. Through extensive experiments, we demonstrate that current agentic systems remain largely insufficient, emphasizing the need for user-side systems that effectively curate tailored products across the modern web.
1 Introduction
AgenticShop addresses the gap between fragmented, exploratory web shopping and benchmarks limited to single-platform search without personalization. It introduces realistic intents, diverse user profiles, and checklist-based evaluation, finding that current agents remain insufficient for personalized product curation.
- Motivation: Fragmented information across platforms makes high-involvement shopping cognitively burdensome and motivates user-side agents that integrate open-web evidence.Users compare alternatives, consult external sources, and synthesize information before purchasing.
- Benchmark gap: Existing benchmarks mainly test predetermined single-platform lookups and omit fine-grained personalization across diverse shopping preferences.They do not adequately evaluate exploratory discovery or whether agents satisfy contextual factors such as delivery speed and reviews.
- Benchmark design: AgenticShop evaluates personalized open-web curation using Target Finding, Alternative Selection, and Open Exploration intents derived from realistic shopping behavior.These intents cover specific-product requests, meaningful comparisons, and casual discovery.
- Benchmark design: The benchmark combines purchase-history-based user profiles, tailored queries, and verifiable checklists covering requirements such as brands, price, reviews, functions, aesthetics, and purchasing conditions.The checklist provides a fine-grained basis for evaluating personalization quality.
- Findings: Experiments find that current agentic systems remain insufficient for personalized curation across the open web.Performance varies by domain, while agents struggle with price, reviews, dynamic content, and discovering adequately reviewed products.
- Findings: Effective personalized curation depends on exploratory information seeking across diverse sources and careful comparison of alternatives.
2 Related Work
Related work spans search-augmented LLMs, autonomous web agents, and increasingly realistic e-commerce benchmarks. Evaluation has progressed toward live environments and automated assessment, but scalability and reliance on human annotation remain concerns.
- Web agents: Search-augmented LLMs retrieve and synthesize web evidence, while autonomous web agents browse live sites, synthesize information, and execute actions.
- Benchmark evolution: Web-agent benchmarks have progressed from static snapshots to reproducible sandboxes and live websites, with e-commerce becoming a prominent evaluation domain.
- Evaluation: Automated evaluation improves scalability through rule-based functions and LLM-as-a-judge, but prior methods still rely on human annotation.
3 AgenticShop
AgenticShop is a benchmark for personalized product curation in open-web settings, built around realistic shopping intents, diverse user profiles, and evaluation aligned with user preferences.
- AgenticShop evaluates agentic systems on personalized product curation in open-web settings.
- Shopping intents: The benchmark covers Target Finding, Alternative Selection, and Open Exploration shopping intents.These represent specific-product search, comparison across options, and proactive browsing for appealing items.
- Benchmark construction: AgenticShop uses Amazon Review data and samples domains suited to the characteristics of each shopping intent.Target Finding uses Grocery & Gourmet Food; Alternative Selection uses Home & Kitchen, Electronics, and Clothing, Shoes & Jewelry; Open Exploration aggregates domains.
- User profiles: User profiles are built from purchase histories and reviews to represent category-specific preferences and decision-making factors.Users are filtered by interaction history and dominant product subcategories before narrative profiles are generated.
- Shopping intents: Queries are tailored to scenarios using seed purchases and varying specificity, including Explicit Title, Attribute-Specific, and Brand-Categorical forms.
- Personalization criteria: The personalization checklist spans six dimensions, including brand, price, reviews, functional requirements, and aesthetics.
3.4 Personalized Evaluation
AgenticShop evaluates curated products by extracting product-page evidence across web platforms and checking each item against personalized user criteria. This produces a fine-grained, scalable measure of preference satisfaction.
- Agentic systems explore multiple shopping platforms to curate products matching each user’s preferences and constraints.
- Information extraction: Product information is extracted from linked product pages to provide objective evidence for personalized evaluation.The extraction collects information such as brand, price, descriptions, delivery options, and reviews, while multimodal inputs support heterogeneous layouts.
- Checklist evaluation: LLM-as-a-judge performs binary verification of whether each curated product satisfies each personalized checklist criterion.
- Curation score: The overall curation score averages satisfied criteria across users, products, and checklists.This integrated evaluation measures how effectively systems satisfy individual preferences and requirements across web shopping scenarios.
4 Experiments
AgenticShop evaluates search-augmented LLMs and autonomous web agents using checklist-based personalization criteria, with meta-evaluations testing checklist validity and judge alignment.
- Baselines: The experiments compare search-augmented LLMs with autonomous web agents for personalized product curation.Search-augmented systems use web retrieval, whereas autonomous agents browse and navigate live interfaces.
- Meta-Evaluation: The evaluation includes 250 samples for alignment analysis of checklists and alternative user representations across domains.
- Meta-Evaluation: The protocol also evaluates checklist reliability and agreement between LLM-as-a-judge and human judges.
- Preference Alignment: The checklist achieves 0.81 average alignment accuracy, outperforming queries, purchase histories, and user context across domains.User ratings provide the ground-truth preference signal for this comparison.
- Judge Reliability: A human study compares LLM-as-a-judge evaluations with two independent human annotations on 50 sampled instances.Each instance includes a product URL, extracted product information, and its corresponding checklist.
5 Results and Discussion
Across shopping intents, domains, queries, and checklist dimensions, current agentic systems show modest personalization performance and several recurring failure modes. Search success alone does not ensure personalization, while attribution, exploration, dynamic prices, and reviews remain major challenges.
- 5.1 RQ1: ChatGPT Search and Claude Sonnet 4 Search achieve only around 30-35% across scenarios, while Open Exploration is difficult for all models.Open Exploration requires balancing relevance and novelty without explicit user queries.
- 5.1 RQ1: Query specificity produces only marginal performance differences in Target Finding, despite slight gains from more detailed specifications.Repeated query reformulation can generate overly complex or inconsistent searches.
- 5.2 RQ2: Electronics generally challenge systems more than Home & Kitchen, while Fashion causes significant performance drops for search-augmented LLMs and web agents.Fashion exposes limitations when subjective visual or aesthetic preferences matter.
- 5.2 RQ2: Accurate product search does not guarantee personalization: Claude Sonnet 4 Search reaches 86% hit accuracy with Explicit Title queries.Search capability is measured independently by matching target brand and type, regardless of checklist alignment.
- 5.2 RQ2: Attribution errors affect all systems, with ChatGPT Search hallucinating approximately 20% of product references.Invalid URLs and irrelevant pages dominate failures; excluding these cases, autonomous web agents outperform search-augmented LLMs.
- 5.2 RQ2: Systems handle brand preferences relatively well but struggle with review and price sensitivity, with approximately 70% of price failures exceeding users’ budgets.Dynamic prices and scattered shipping, tax, discount, and variant costs complicate final-cost calculation.
- 5.2 RQ2: Exploratory information seeking appears to support personalization, while balancing exploration with alternative comparison remains a promising direction.Web agents perform more information-seeking behaviors than search-augmented LLMs.
- Error Analysis: Hallucination remains a critical bottleneck, and autonomous web agents are frequently distracted by promotional content.
6 Conclusion
AgenticShop introduces a benchmark for personalized product curation in open-web environments and finds that current agentic systems struggle to align curation with individual user needs.
- AgenticShop combines realistic shopping intents, diverse user profiles, and checklist-driven evaluation for rigorous, verifiable personalization assessment.
- Extensive experiments show that current agentic systems struggle with personalized product curation.
A Benchmark Details
AgenticShop constructs personalized tasks across shopping scenarios, with tailored checklists representing user-specific preferences.
- AgenticShop includes 50 user profiles for each shopping scenario, totaling 350 personalized tasks with tailored checklists.Each task specifies a product type, producing 50 unique products per scenario; Open Exploration is the exception because it lacks predetermined products.
B Dimension-wise Evaluation Reliability Analysis
The checklist-driven evaluation framework aligns substantially with human judgments across all six personalization dimensions. Both Cohen’s κ and Spearman’s ρ are approximately 0.7 throughout.
- Human evaluators show substantial agreement with the checklist-driven evaluation framework.
- Agreement is assessed across six checklist dimensions.
- Cohen’s κ and Spearman’s ρ are approximately 0.7 across all dimensions.
C Analysis of Successful Cases
When all curated product URLs are valid and accessible, agentic systems receive higher personalization scores. The improvement is particularly large for autonomous web agents, which surpass search-augmented LLMs.
- The successful-case analysis includes only responses whose curated product URLs are valid and accessible.This ensures that each response is grounded in a real product page.
- Both search-augmented LLMs and autonomous web agents achieve higher personalization scores when failure cases are excluded.
- Autonomous web agents show the particularly large improvement and surpass search-augmented LLMs.
D Error Analysis
Manual inspection of logs, responses, and navigation traces identifies recurring failure patterns in search-augmented LLMs and autonomous web agents. These include invalid URLs, query-rewriting problems, and failures involving product attributes and personalization.
- Error Analysis: Manual examination of system logs, generated responses, and navigation traces is used to identify recurring failure patterns and underlying causes.
- Search-augmented LLMs: Search-augmented LLMs frequently hallucinate malformed product links, including fabricated Amazon ASINs and missing URL path segments.
- Search-augmented LLMs: Search-augmented LLMs may prioritize brand-focused rewritten queries even when users emphasize functional attributes.The analysis reports this pattern in Figure 7(c).
- Autonomous web agents: Autonomous web agents generate invalid links and overly complicated rewritten queries, with URL errors attributed partly to lengthy response-generation contexts.