PaperlayerView exact revision on arXiv

Source-linked AI summary

A Behavioral Trait Leaks into Preferences: Diagnosing Trait Interference in LLM User Simulators

Chaehyun Kim, Sein Kim, Hongseok Kang, Chanyoung Park

arXiv:2609.25572v1cs.AI

TL;DR

Whether activity preserves its intended role in LLM user simulators has been largely overlooked, despite its relevance to satisfaction-based recommender evaluation. The paper proposes PQA, which anchors browsing decisions in personalized page quality; experiments report reduced activity-driven over-browsing, restored trait independence, and improved evaluation reliability under activity shifts.

  • Problem

    Whether activity preserves its intended browsing-depth role in LLM user simulators has been largely overlooked, despite its relevance to satisfaction-based recommender evaluation.

    Source: §1 · p. 1

  • Method

    PQA injects a personalized page-quality anchor derived from a user’s interaction history so the simulator can assess preference alignment before continuing to browse.

    Source: §1 · p. 2

  • Results

    PQA reduces activity-driven over-browsing, restores trait independence, and improves the reliability of simulator-generated interactions under activity shifts.

    Source: §6 · pp. 5–6

  • Takeaways & Limitations

    Grounding exit decisions in personalized page-level preference alignment supports more reliable simulator evaluation under activity distribution shifts.

    Source: §1 · p. 2, §6 · p. 6

    PQA is based on category-level signals; extending it to richer preference indicators is identified as a future direction.

    Source: §6 · p. 6

Abstract

from arXiv · show

LLM-based user simulators aim to bridge the offline-online gap in recommender evaluation by emulating users through injected traits, where preference attributes determine what a user engages with and a behavioral activity trait governs how long they browse. However, we show this intended trait independence collapses during simulation, causing two failures: (i) Trait Interference, where amplified activity distorts preference boundaries and forces interactions with mismatched items to sustain browsing, and (ii) Evaluation Invalidity, where satisfaction scores inflate with activity-driven page counts despite taste mismatches, biasing evaluation toward trait distributions rather than recommender performance. To resolve this, we propose PQA, a page-level quality anchoring method that guides simulators using a personalized anchor reflecting each user's intrinsic preference standard. By assessing whether a page meets this standard before further browsing, PQA enables proactive exits from low-quality pages, letting the activity trait retain its intended role of modulating browsing depth within preference-conforming pages. Experiments show PQA mitigates trait interference and improves the reliability of LLM-based simulator evaluation under activity shifts. Our code is available at https://github.com/chaehyun1/PQA

1 Introduction

LLM user simulators assign preferences to shape what users engage with and activity to shape browsing duration, but whether activity stays within that role has been largely overlooked. The paper identifies trait interference and evaluation invalidity, then proposes PQA to anchor browsing decisions in personalized page quality.

1 Introduction

  • Existing simulators inject preference attributes to define what users like and activity to determine how long they browse recommendation pages.The simulator presents recommendation lists page by page and decides when to stop.Source: §1 · p. 1
  • Amplified activity distorts preference judgments, leading simulators to select mismatched items and deviate from users’ intrinsic category preferences.The paper contrasts this with real-world data, where more active users tend to give lower average ratings, indicating stricter rather than more permissive behavior.Source: §1 · p. 1
  • Activity amplification sharply increases explored pages and satisfaction scores, even when recommendation quality declines and users continue through preference-mismatched pages.This makes satisfaction confounded by injected activity rather than reflective of recommender performance.Source: §1 · p. 1
  • The authors identify the missing page-level quality anchor as the root cause, allowing activity constraints to override preference alignment.Activity should govern browsing depth without relaxing preference boundaries; prior simulator designs also assign it a browsing or interaction-frequency role.Source: §1 · pp. 1–2
  • PQA injects a personalized anchor based on historical category consumption so simulators can assess page quality before continuing and exit low-quality pages.The authors report that PQA preserves trait independence and restores evaluation validity under activity distribution shifts.Source: §1 · p. 2

2 Preliminary Analysis

The preliminary analysis tests whether activity affects only browsing depth or also changes preference alignment, and whether activity-driven browsing distorts satisfaction-based evaluation. It finds that higher activity sustains browsing through mismatched items and inflates evaluation metrics.

2 Preliminary Analysis

  • The simulation framework presents a recommender’s personalized ranked list page by page, with the simulator choosing interactions and whether to continue or exit.The framework uses a backbone recommender and injected personas and user traits.Source: §2 · p. 2

2.1 Trait Interference

  • At the same page depth, high-activity simulators have lower persona–item cosine similarity and maintain interactions as recommendation relevance declines.The comparison holds presented items constant across activity levels.Source: Figure 1 · p. 2, §2.1.1 · p. 2
  • The controlled protocol uses Agent4Rec and SimUSER with SASRec on MovieLens and Amazon CDs, presenting four items per page from a 20-item ranked list.Lists use a 1:k ratio of positive top-ranked to negative bottom-ranked items, so relevance declines with page depth.Source: §2.1.2 · p. 2
  • High activity lowers the category-overlap-to-baseline ratio across both datasets, while low activity’s higher ratio partly reflects early exits that miss later preference-matching items.The low-activity ratio is a structural blind spot, not evidence of precise filtering.Source: Figure 2 · p. 3, §2.1.2 · p. 3
  • The category-overlap diagnostic compares each browsed page with a historical baseline built from users’ top five categories and session-level category overlaps.A page-overlap-to-baseline ratio near 1.0 indicates adherence to the user’s preference boundaries.Source: §2.1.2 · pp. 2–3
  • Unlike the simulator’s relaxed criteria under high activity, real-world activity correlates negatively with average historical ratings in MovieLens and CDs.The reported correlations are Kendall’s τ = −0.366 (MovieLens) and τ = −0.069 (CDs).Source: §2.1.3 · p. 3

2.2 Evaluation Invalidity

  • Amplifying activity sharply increases explored pages and satisfaction, whereas amplifying conformity or diversity produces only marginal changes in those metrics.Table 1 summarizes changes when each user trait is amplified from low to high.Source: Table 1 · p. 3, §2.2.1 · p. 3
  • Across positive-to-negative page ratios, amplifying activity increases explored pages and satisfaction, even when the 1:9 list contains items mismatched with users’ tastes.These inflated scores therefore reflect injected activity rather than recommended-item relevance.Source: §2.2.2 · pp. 3–4, Table 2 · p. 4

3 Methods

PQA gives the simulator a personalized page-quality standard so activity affects browsing depth only after preference alignment is assessed. Its page labels and continuation rules are designed to avoid both premature exits and continued browsing through low-quality pages.

3 Methods

  • PQA gates activity-driven browsing with a personalized assessment of whether each page meets the user’s preference standard.The paper attributes trait interference to the lack of an explicit page-level quality anchor.Source: §3 · p. 4
  • The anchor μ_u is the user’s historical category-overlap baseline, used to label pages ABOVE, NORMAL, or BELOW.The prompt compares each page’s category-match score with μ_u; dataset-specific ratio thresholds are (1.0, 0.7) for MovieLens and (0.5, 0.3) for CDs.Source: §3 · p. 4
  • Page quality informs the simulator’s overall feeling, while continuation decisions jointly consider the quality label, activity, and accumulated fatigue.Source: §3 · p. 4
  • ABOVE pages delay immediate exit by one page; NORMAL pages get one sticky exploration, while two consecutive BELOW pages force exit regardless of activity.A first-page BELOW label gets one grace page; otherwise, activity and fatigue inform exit decisions, with high activity allowing limited patience.Source: §3 · p. 4

4 Experiments

Experiments use Agent4Rec and SimUSER with MovieLens and Amazon CDs, generally running simulations with GPT-4o-mini. The human-likeness evaluation uses GPT-4o across recommendation domains.

4 Experiments

  • Experiments apply PQA to Agent4Rec and SimUSER on MovieLens and Amazon CDs.Source: §4 · p. 4
  • GPT-4o-mini is the underlying LLM for simulations unless otherwise stated.Source: §4 · p. 4
  • Human-likeness scores are evaluated by GPT-4o across recommendation domains.Source: Table 4 · p. 5

1. Restoring Trait Independence and Evaluation Validity un-

Under activity shifts, baseline simulators browse too little or too far relative to page quality, and their satisfaction scores can reflect activity rather than recommendation quality. PQA grounds exit decisions in personalized page alignment, mitigating these failures.

1. Restoring Trait Independence and Evaluation Validity un-

  • PQA mitigates low-activity premature exits and high-activity over-browsing across MovieLens and CDs, improving evaluation reliability under activity shifts.For Agent4Rec, PQA helps explore remaining relevant items under low activity and limits exploration after quality falls below the user’s standard under high activity.Source: §1 · p. 5
  • Baseline satisfaction scores rise with high activity even when browsing reaches low-quality pages, confounding scores with activity rather than recommendation quality.Source: §1 · p. 5
  • SimUSER already browses more deeply under low activity through its exit-refinement procedure, but still over-browses under high activity without PQA.Source: §1 · p. 5

2. Downstream Validation via Simulated Interaction Augmen-

PQA-generated interactions improve downstream recommendation performance and human-likeness evaluations across the reported datasets. The results are consistent with PQA filtering low-quality pages and producing interaction logs that better reflect user preferences.

2. Downstream Validation via Simulated Interaction Augmen-

  • PQA-Agent4Rec reaches NDCG@5 of 0.3046 on MovieLens and 0.3479 on CDs, with the largest gains; PQA-generated interactions yield the best performance across both datasets.The interactions are used as positive labels to fine-tune SASRec.Source: §2 · p. 5
  • PQA terminates browsing before low-quality interactions accumulate, yielding cleaner data that better reflects users’ genuine preferences.Source: §2 · p. 5
  • PQA-Agent4Rec human-likeness scores rise from 3.57 to 4.15 on MovieLens and from 3.64 to 4.22 on CDs; PQA-SimUSER scores highest at 4.35 on MovieLens.Scores are evaluated by GPT-4o on a 5-point Likert scale.Source: §2 · p. 5

5 Related Work

Prior work developed realistic user simulators and later used them to provide recommender feedback, but whether simulators preserve injected traits’ intended roles received little attention.

5 Related Work

  • Agent4Rec, profile-aware simulators, PUB, SimUSER, and AgentCF model user personas, traits, or interaction patterns in different ways.RecoWorld shifts focus toward proactive simulator feedback for recommender adaptation.Source: §5 · p. 5
  • Existing evaluations often report aggregate satisfaction, which can conceal whether injected traits retain their intended roles.Source: §5 · p. 5
  • The paper examines the assumption that preferences govern engagement while activity governs browsing duration, finding that this independence collapses and affects satisfaction-based evaluation.Source: §5 · p. 5

6 Conclusion

The paper identifies trait interference as a failure mode in LLM user simulators and proposes PQA to anchor exit decisions in personalized page-level preference alignment. Experiments report reduced activity-driven over-browsing and more reliable simulator-generated interactions, while richer preference indicators remain a future direction.

6 Conclusion

  • PQA grounds exit decisions in personalized page-level preference alignment to address trait interference in LLM-based user simulators.Source: §6 · p. 5
  • Experiments show PQA reduces activity-driven over-browsing and restores trait independence.Source: §6 · p. 5
  • The authors identify richer preference indicators, including item-item relationships and collaborative-filtering signals, as a promising direction for capturing intrinsic preferences across domains.Source: §6 · p. 6

GenAI Usage Disclosure

The authors disclose limited use of generative AI tools for language editing and minor plotting or visualization code work; they state that all changes were reviewed and validated.

GenAI Usage Disclosure

  • Gemini and Claude were used for grammar checks, readability, expression, and length adjustments.Source: GenAI Usage Disclosure · p. 7
  • Generative AI tools were also used for minor refactoring and debugging of plotting and visualization code.Source: GenAI Usage Disclosure · p. 7
  • The authors state that all AI-assisted revisions and code modifications were reviewed and validated, and that the core ideas, methods, experiments, and interpretations are their original contributions.Source: GenAI Usage Disclosure · p. 7
Loading 2609.25572v1…