PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalization Matters: Long-Horizon Conversation Agent with User-Centric Information in Online Shopping Interactions

Rena Gao, Yue Dai, Hao Guan, Shengxiang Gao, Wangyang Wu, Yixin Shen, Jey Han Lau

arXiv:2610.11375v1cs.MA

TL;DR

Personalized shopping agents must track preferences disclosed across turns, while existing approaches and evaluation methods provide limited coverage of long-horizon alignment. The paper combines multi-agent multimodal retrieval with trajectory-level evaluation; retrieval-enabled variants outperform no-RAG on benchmark trajectory scores, though the evidence remains directional.

  • Problem

    Conversational shopping reveals user preferences gradually, and local fluency or accuracy measures can miss preference adherence and cross-turn usefulness.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    A multi-agent multimodal RAG framework separates dialogue tracking, retrieval, preference-aware reasoning, and response generation, and evaluates full trajectories across four dimensions.

    Source: §1 · pp. 1–2

  • Results

    Retrieval-enabled variants achieve average trajectory scores of 4.72–4.82 versus 3.74 for the no-RAG baseline; Full receives a 4.60 mean overall rating versus TextRAG’s 3.40 in the exploratory study.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend reporting human feedback alongside automatic scores because text-only LLM judges may understate perceived benefits from visual grounding.

    Source: §7 · p. 8

    Evaluation focuses on controlled shopping dialogues generated from product and review data, with future validation needed on naturally occurring conversations and larger user studies.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Personalized conversational shopping requires maintaining preference consistency over multi-turn interactions, where users reveal constraints gradually. Existing approaches often rely on static profiles and do not explicitly control long-horizon interaction behavior. We propose a multi-agent, multimodal Retrieval-Augmented Generation (RAG) framework that decomposes dialogue state tracking, recommendation retrieval, preference-aware reasoning, and response generation, while integrating product metadata, product reviews, image-derived descriptions, and user historical reviews. To evaluate interaction-level quality, we adopt a trajectory-level protocol with four dimensions: Global Preference Consistency, Cumulative Information Synthesis, Interaction Trajectory, and Tone Consistency. On an Amazon Reviews 2023 benchmark, retrieval-enabled variants outperform a no-RAG baseline on automatic trajectory metrics (average 4.82 vs. 3.74). In a small real-user study ($n{=}5$), the Full variant achieves the highest mean overall rating (4.60 vs. 2.20 for Baseline), providing exploratory evidence that role decomposition plus user-centric retrieval improves perceived personalization.\footnote{Code and dataset are available at: https://github.com/RenaGao/Multimodel_RAG_Indexing

1 Introduction

Conversational shopping requires tracking preferences that emerge across turns, not just producing locally relevant replies. The paper proposes a multi-agent multimodal RAG framework and evaluates its long-horizon interaction quality.

1 Introduction

  • Preferences emerge gradually across turns, so conversational shopping quality depends on sustained alignment rather than turn-level correctness alone.Figure 1 illustrates a locally reasonable response that ignores previously stated style preferences.Source: §1 · p. 1, Figure 1 · p. 1
  • The framework separates dialogue tracking, recommendation retrieval, preference-aware reasoning, and response generation while integrating product evidence, image-derived descriptions, and user-review history.The contribution also describes a trajectory-level protocol covering preference consistency, information synthesis, dialogue progression, and tone.Source: §1 · pp. 1–2
  • The evaluation uses four trajectory dimensions—Global Preference Consistency, Cumulative Information Synthesis, Interaction Trajectory, and Tone Consistency—on an Amazon Reviews 2023 benchmark.The user study is exploratory and includes five participants.Source: §1 · p. 2
  • Retrieval-enabled variants achieve average trajectory scores of 4.72–4.82 versus 3.74 for the no-RAG baseline; Full rates 4.60 versus TextRAG’s 3.40 in the exploratory user study.The authors note that Full and TextRAG are nearly tied under text-based automatic evaluation.Source: §1 · p. 2

2 Related Work

Prior work spans personalized agents and conversational recommendation, while interaction evaluation has increasingly focused on quality across turns. This paper adapts trajectory-level assessment to personalized shopping and sustained preference tracking.

2.1 Personalized Agents and Conversational Recommendation

  • Personalized agents have shifted from static persona-conditioned generation toward retrieval and memory that inject user history at inference time.Examples include approaches using external knowledge and goal planning, review evidence, specialized agents, and dialogue state tracking to retrieve reviews.Source: §2.1 · p. 2
  • The framework combines user-history retrieval and role decomposition without task-specific training, comparing retrieval inputs and agent configurations with trajectory metrics and a small user study.The comparisons use the same frozen model and user-query trajectories.Source: §2.1 · p. 2

2.2 Long-Horizon Interaction Evaluation

  • Fluency, coherence, and local accuracy metrics can miss user-specific goals such as preference adherence, mixed initiative, and cross-turn usefulness.Prior trajectory-level assessment evaluates how quality evolves across turns rather than at isolated response points.Source: §2.2 · p. 2
  • The paper adapts trajectory-level assessment to personalized shopping, emphasizing full-trajectory preference consistency and recommendation grounding over isolated turn quality.The setting centers on sustained preference tracking under retrieval-augmented reasoning.Source: §2.2 · p. 2

3 Methodology

The methodology simulates multi-turn personalized shopping and separates assistant work into specialized agents that use dialogue, product, and user-review evidence. It constructs query trajectories from user histories and retrieves complementary evidence to ground responses.

3.1 Multi-Agent Pipeline

  • Five agents simulate the interaction: a User agent generates queries, while Dialogue, Recommend, Reason, and Reply agents produce assistant responses.Role decomposition is intended to improve controllability, modularity, and interpretability, and to support ablations that isolate sources of improvement.Source: §3.1 · p. 3

3.2 Problem Definition

  • Each turn comprises a user query, internal context, ranked candidates with rationale, a preference-aware explanation, and a response; only queries and responses appear in the transcript.The benchmark generates user-side trajectories first, then runs assistant-side generation under controlled variants.Source: §3.2 · p. 3

3.3 User Query Generation

  • User profiles summarize historical reviews, and an LLM generates preference-oriented multi-turn queries around a positively rated target product.The target review is excluded from profile construction, and the same query trajectories are reused across assistant variants.Source: §3.3 · p. 3

3.4 Assistant-Side Multi-Agent Response Generation

  • The assistant pipeline tracks context, retrieves and reranks candidates, explains their fit to preferences and intent, then generates the visible reply.The Dialogue Agent combines an accumulated summary with a relevant prior turn; separating Reply from retrieval and reasoning provides control over wording and style.Source: §3.4 · pp. 3–4

3.5 Embedding Representations

  • Dialogue, product, and user-review evidence use Qwen3-Embedding-8B; product vectors represent metadata, review means, and image-derived descriptions.Historical user-review embeddings exclude the target-product review, and the current query is embedded for retrieval.Source: §3.5 · p. 4

3.6 Retrieval-Augmented Generation

  • Three retrieval functions use the same query embedding to retrieve dialogue, product, and user-review evidence from different corpora.Per query, the system retrieves one prior dialogue turn and up to five products and five user reviews.Source: §3.6 · pp. 4–5

4 Dataset

The dataset section describes constructing a synthetic benchmark from Amazon Reviews 2023, whose metadata, images, and review histories support personalized shopping dialogues.

4 Dataset

  • Amazon Reviews 2023 supplies product metadata, images, and user review histories for the simulation data and retrieval corpora.The section describes the source and preprocessing for the synthetic benchmark.Source: §4 · p. 5

4.1 Dataset Processing

  • The dataset processing removes the Unknown category and selects the 100 most active users in each of the remaining 33 categories.For each selected user, it collects purchased items and their purchase reviews, then gathers all reviews for those items, including reviews by users outside the selected pool.Source: §4.1 · p. 5
  • Synthetic dialogue instances are constructed from the processed user–item–review subset using the user-query generation procedure.Per-category statistics are reported in Appendix Table 6.Source: §4.1 · p. 5

5 Experiments

The experiments compare assistant-side retrieval and agent configurations, then evaluate personalization across full dialogue trajectories rather than isolated turns. The evaluation operationalizes trajectory quality through preference consistency, information synthesis, interaction progress, and tone.

5.1 Multi-Agent System

  • The five assistant-side variants differ in retrieval configuration, product-description embeddings, and whether four assistant roles remain separate or collapse into one.Full uses all components; NoRev removes user-review retrieval, TextRAG drops description embeddings, Single collapses the four roles, and Baseline disables retrieval. Table 1 summarizes processed-dialogue dataset scale.Source: §5.1 · p. 5, Table 1 · p. 5

5.2 Evaluation Metrics Design

  • Evaluation scores full-dialogue quality across four trajectory dimensions on 1–5 scales, using GPT-4o as judge and paired comparisons for significance.The framework is adapted for personalized shopping because turn-level metrics can miss violations of earlier constraints; the judge receives a user profile and full transcript.Source: §5.2 · pp. 5–6

Global Personalization Consistency (GPC)

  • Global Personalization Consistency measures fidelity to the user persona and constraints across the interaction; examples contrast Baseline’s GPC 1 with a constraint-synthesizing response scoring 5.High GPC indicates stable long-term context use and reduced attribute hallucination.Source: Global Personalization Consistency (GPC) · p. 6
  • Cumulative Information Synthesis evaluates whether early and late cues are integrated, contrasting repeated requests for already supplied details at CIS 1 with preference-linked refinement at CIS 5.It assesses aggregation of constraints across turns rather than isolated retrieval.Source: Global Personalization Consistency (GPC) · p. 6
  • Interaction Trajectory rewards progress from clarification toward recommendation, refinement, and closure, while Tone Consistency tracks stability of friendly, professional, or empathetic style.Repeated loops without progress receive lower interaction-trajectory scores; abrupt register shifts can reduce perceived naturalness.Source: Global Personalization Consistency (GPC) · p. 6
  • Three PhD volunteers independently annotated 20 randomly sampled dialogues, with pairwise agreement above 85% across all four dimensions.The sample covered five product categories.Source: Global Personalization Consistency (GPC) · p. 6

6 Results

Automatic evaluations favor retrieval-enabled systems and show benefits from role decomposition on cross-turn synthesis and dialogue progression. In the five-person user study, Full received the strongest feedback, though the authors treat the evidence as exploratory.

6.1 Automatic Evaluation

  • 4.93 GPC and ∼4.89 CIS were the highest scores for Full and TextRAG, above Baseline at 3.77 / 3.17.On IT, TextRAG scored 4.47 and Full 4.45; Full’s improvements over Single in information synthesis, dialogue progression, and overall quality were statistically significant.Source: Table 3 · p. 6, §6.1 · p. 6
  • GPC reflects long-horizon memory, CIS cross-turn integration, and IT is most architecture-sensitive; TC scores are uniformly 5.00 and non-discriminative.The paper interprets the Full-to-Single IT drop as consistent with weaker dialogue-state progression when reasoning is compressed into one prompt.Source: §6.1 · pp. 6–7
  • +0.11 CIS and +0.26 IT for Full over Single suggest role decomposition mainly helps cross-turn synthesis and interaction flow, at higher per-turn call cost.The comparison holds retrieval branches constant; retrieval-enabled variants also outperform the no-RAG baseline on average trajectory scores.Source: §6.1 · p. 7
  • 10.31% retrieval accuracy for Baseline contrasted with more than 40% for every retrieval-enabled variant.This token-overlap measure tracks target-product mention, not long-term preference alignment, so the paper reports it alongside trajectory scores and user ratings.Source: Table 4 · p. 7, §6.1 · p. 7

6.2 User Study: Real Interaction

  • Four of five users preferred Full overall, and all five judged it the most personalized variant; the authors characterize this feedback as directional.The study collected ratings from five users and was not a statistically powered hypothesis test.Source: §6.2 · pp. 7–8, Table 5 · p. 8
  • Users described Full carrying earlier constraints and tendencies forward, grounding recommendations in prior budget and style preferences, and connecting current requests with earlier product attributes.Users also reported that weaker variants could be polite but less reliable for sustained personalized recommendation.Source: §6.2 · p. 8
  • Across automatic and human evaluation, Baseline was weakest, while Full showed stronger human-perceived personalization than Single and TextRAG despite close automatic averages.The authors describe the five-participant feedback as an initial basis for a larger user study.Source: §6.2 · p. 8

7 Conclusion

The conclusion reports strong performance for the multi-agent multimodal RAG setup against no-RAG baselines across automatic and human-facing evaluations. The authors characterize the findings as directional given the synthetic benchmark and small user study.

7 Conclusion

  • The multi-agent multimodal RAG setup clearly exceeds no-RAG baselines on automatic and human-facing evaluations.The conclusion describes its performance relative to simpler variants as strong.Source: §7 · p. 8
  • Text-only LLM judges may understate user-perceived gains from visual grounding, supporting joint reporting of automatic scores and human feedback.Source: §7 · p. 8
  • Synthetic benchmark data and a five-person user study make these results directional; future work should test larger studies and naturally collected multimodal shopping interactions.Source: §7 · p. 8

Limitations

The evaluation uses controlled shopping dialogues generated from product and review data. The authors identify natural conversations and larger user studies as future extensions.

Limitations

  • The evaluation focuses on controlled shopping dialogues generated from product and review data.Source: Limitations · p. 8
  • Future work could extend the setting to naturally occurring conversations and larger user studies.Source: Limitations · p. 8
  • The current system uses text queries and image-derived descriptions; direct image queries and richer retrieval methods remain future directions.Source: Limitations · pp. 8–9
  • Related models for generation and judging may influence the automatic scores.Source: Limitations · p. 8

Ethical considerations

The paper uses the public Amazon Reviews 2023 dataset without identifying reviewers and reports that study participants consented. It also notes recommendation risks and privacy safeguards for practical use.

Ethical considerations

A.1 Dataset Statistics

Table 6 reports product and review statistics alongside released-query-set trajectory counts. The supplied text gives a total of 327 trajectories and counts for two categories.

A.1 Dataset Statistics

  • Table 6 organizes statistics by category and lists item, review, and average-dialogue columns.Source: §A.1 · p. 12, Table 6 · p. 12
  • 327 trajectories appear in the released query set, including 8 for Movies and TV and 9 for Subscription Boxes.Source: Table 6 · p. 12
  • The caption reports 10 trajectories for each other category.Source: Table 6 · p. 12

A.2 Cost–Performance Trade-off

Full trades a more involved, four-call-per-turn pipeline for higher user ratings, while Single uses one call and offers simpler inference.

A.2 Cost–Performance Trade-off

  • Full uses four assistant-role calls per dialogue turn and receives higher user ratings, while Single uses one call and offers a simpler inference pipeline.Table 7 compares call counts with automatic-evaluation average scores and user-study Overall ratings.Source: §A.2 · p. 12, Table 7 · p. 12
  • Full assigns Dialogue, Recommend, Reason, and Reply to separate stages; Single combines these roles in one call.Source: Table 7 · p. 12
  • The Baseline call count in Table 7 is retained from the original report and awaits configuration verification.Source: Table 7 · p. 12

A.3 Supplementary Experiments

The supplementary material documents experiments, evaluation criteria, resource and prompt designs, and the user-study questionnaire and data/code licensing. It also illustrates multi-turn shopping dialogue and analyzes how user history and visual metadata inform personalized recommendations and explanations.

A.3 Supplementary Experiments

  • Full has the highest target-item coverage, while TextRAG reaches successful target ASINs earlier; first-hit turns are averaged only over trajectories with a hit.These measures assess target retrieval rather than user acceptance.Source: §A.3 · p. 12, Table 9 · p. 12
  • A second judge produces the same system ranking on 99 re-scored trajectories, comprising three per category and 495 system outputs.The reported GPT-4o score correlations omit tone consistency (TC) because of its ceiling effect.Source: §A.3 · pp. 12–13, Table 10 · p. 13
  • Summary-only memory performs similarly to full memory, while removing memory or keeping only one prior turn lowers scores.The comparison uses 99 trajectories and contrasts full memory (summary and retrieved history), no memory, summary only, and the top-ranked prior turn only.Source: §A.3 · p. 13, Table 11 · p. 13

A.4 Resources in Related Work

User Profile Generation Prompt

  • The profile-generation prompt extracts likes from 4–5-star reviews and dislikes from 1–3-star reviews, including preferences, style, budget, size, quality priorities, and pain points.It also asks the model to identify recurring rating patterns.Source: User Profile Generation Prompt · p. 13

Question Generation Prompt

  • The question-generation prompt uses latent user preferences and target metadata to create generic recommendation questions with preference-sensitive decision points, without stating hidden preferences.Questions must address only attributes present or strongly implied in the metadata, and the profile is the primary source of latent preferences.Source: Question Generation Prompt · p. 14

Dialogue Agent Prompt

  • The dialogue agent first retrieves similar conversation history, then produces a round-marked summary using prior summaries and retrieved conversations to capture user preferences and context.The prompt specifies a 2–4 sentence summary and prohibits other tool calls.Source: Dialogue Agent Prompt · p. 16

Reason Agent Prompt

  • The reasoning agent explains how actual past reviews influenced recommendations, connecting extracted preference patterns to specific product choices within the current question round.It extracts the current query, dialogue history and summary, recommendation-tool result, and final recommendation while ignoring messages before the round boundary.Source: Reason Agent Prompt · pp. 16–17

Reply Agent Prompt

  • The Reply Agent prompt requires friendly greetings in round one, contextual greetings in later rounds, and responses personalized using the user’s actual past reviews.It directs the agent to reference specific review preferences and connect recommendations to past experiences.Source: Reply Agent Prompt · p. 17

1. Global Personalization Consistency (GPC)

  • The 1–5 evaluation rubric assesses whether persona constraints persist throughout dialogue and whether combined responses build a complete, factual picture without repetition or contradiction.The criteria distinguish consistent constraint retention from forgetting explicit or implicit preferences; the table caption describes four dimensions of cumulative dialogue quality.Source: §1 · p. 18, Table 13 · p. 18

A.7 Four Interactivity Aspects Definitions and Descriptions

  • Table 14 defines macro-level interactivity features, with higher scores indicating more natural, authentic interaction and active engagement.The definitions are adapted from Gao et al. (2025b).Source: Table 14 · p. 18, §A.7 · p. 18

Recommender Agent Response

  • The Recommender Agent example elicits skin concerns and skin type, while accompanying tables present its output and a Reasoner evaluation.The table captions describe structured, evidence-based recommendations and evaluation of personalization, structure, credibility, and educational value.Source: Recommender Agent Response · p. 19, Table 15 · p. 19, Table 16 · p. 19

A.9 Qualitative Case Analysis

  • Combining historical preferences with visual metadata helps anchor goals, surface latent attributes in sparse categories, and produce diverse, context-aware, explainable recommendations.The qualitative analysis contrasts RAG variants that connect recommendations to past affinities with generic NoRev suggestions that fail to make that connection.Source: §A.9 · p. 20

Multimodal RAG Dialogue Example

  • The example dialogue connects past reviews of pinecones, driftwood, and geodes to nature-inspired accessory recommendations, then carries those preferences into later turns.Later responses refer to preferences for handcrafted textures, hypoallergenic materials, natural textures, and polished finishes.Source: Multimodal RAG Dialogue Example · p. 20
  • Packaging advice combines protective cushioning with recyclable boxes and biodegradable or reusable fillers, reflecting the user’s stated sustainability preferences.The suggestions also include recycled-paper thank-you cards and carbon-neutral shipping options.Source: Multimodal RAG Dialogue Example · p. 24
  • For a craft beer lover, the gift suggestions include a hops pin and an upcycled bottle-cap turtle, while the bear brooch is excluded as a weaker fit.The recommendations connect the first two options to craft beer, recycled materials, and distinctive design.Source: Multimodal RAG Dialogue Example · p. 24
  • Across follow-ups, recommendations connect prior preferences for nature-inspired motifs and sustainable materials to product suggestions, including earrings and brooches as well as enamel pins.The examples frame these suggestions around qualities such as hypoallergenic materials, artisanal detail, and natural symbolism.Source: Multimodal RAG Dialogue Example · p. 25
  • For craftsmanship-focused requests, the dialogue highlights textured, handmade designs and sustainable materials, while including adjacent products such as earrings alongside enamel pins.Examples include a hops pin, handmade apple-blossom earrings, and hand-painted hummingbird earrings.Source: Multimodal RAG Dialogue Example · p. 25
  • The size discussion contrasts a larger 1" × 1.4" hops pin for visible botanical detail with smaller options for subtle aesthetics and material efficiency.It also considers a 3.5" rose clothes clip and 12mm resin stud earrings, extending beyond pins.Source: Multimodal RAG Dialogue Example · p. 25
  • For durability and a polished look, suggestions pair gold-plated hard enamel with a metallic hop design or stainless steel and glitter resin with a sleek, colorful finish.The examples also associate these materials with longevity, skin-safe wear, and sustainability.Source: Multimodal RAG Dialogue Example · pp. 25–26
  • The shopping dialogue recommends pins around eco-friendly materials and beer-themed motifs, while also discussing durability, polished looks, packaging, and gift styles.Source: Multimodal RAG Dialogue Example · pp. 26–27

A.10 Interactive Demo

A.11 User Study Questionnaire

  • Five unpaid volunteers recruited through the authors’ contacts used the systems via a web interface and answered the same questionnaire for each system.Source: §A.11 · p. 45
  • The questionnaire’s first five questions used five-point scales to assess satisfaction, preference memory, information connection, conversational flow, and tone; a sixth invited written comments.Source: §A.11 · p. 45

A.12 Data and Code Licensing

  • The Amazon Reviews 2023 maintainers do not assign a dataset license or determine its terms of use; the authors’ MIT-licensed code excludes third-party code and data.Source: §A.12 · p. 45

Related papers

Loading 2610.11375v1…