Source-linked AI summary
VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences
Yiwen Jiang, Yang Deng, Stephanie Fong, Zimu Wang, Yaling Shen, Wei Feng, Hongxi Yang, Xiangyu Zhao, Zhongxing Xu, Deval Mehta, Xuelian Cheng, Zongyuan Ge
TL;DR
Existing PLLM personalization often assumes that query-relevant preferences can be retrieved from semantically related history, but PRCM places profile cues and target preferences in different concept spaces. VIBE-BENCH isolates this regime with psychology-grounded tasks and shows that current methods struggle to learn robust cross-concept mappings, while explicit concept-aware rationales improve performance.
Problem
Existing personalization methods assume query-relevant preferences are retrievable from semantically related history, leaving cross-concept preference reasoning under-evaluated.
Method
VIBE-BENCH isolates PRCM through two psychology-grounded tasks and evaluates personalization beyond semantic retrieval.
Results
Current personalization methods largely exploit semantic correlations but struggle to induce robust profile-preference mappings under PRCM.
Takeaways & Limitations
Concept-aware reasoning improves accuracy by 44% on Task 1 and 8% on Task 2, indicating that explicit cross-concept guidance helps under PRCM.
Takeaways & Limitations
VIBE-BENCH covers two psychology-grounded tasks with synthetic histories and English-only evaluation, so transfer to natural, multilingual, cross-cultural, or additional-domain settings remains open.
Abstract
from arXiv · showhide
Personalized Large Language Models (PLLMs) aim to tailor responses to individual users, where a central challenge is preference reasoning: inferring query-relevant preferences from user-related history. Existing benchmarks, however, largely assume that such preference can be retrieved from semantically related history. We study an underexplored but practically important regime, profile-preference conceptual misalignment (PRCM), where observable profile cues and query-specific preferences lie in different concept spaces, making semantic retrieval inconsistent for personalization. We introduce VIBE-Bench, a benchmark with two psychology-grounded tasks, 3,504 personas and 12,239 dialogues, including a manually verified gold test set, and requires cross-concept preference reasoning beyond surface semantic overlap. Experiments with several personalization methods show that current PLLMs largely rely on shallow semantic correlations and fail to acquire robust cross-concept mappings. These findings establish PRCM as a distinct failure regime in PLLMs and position VIBE-Bench as a focused testbed for advancing preference reasoning beyond semantic matching.
1 Introduction
PLLMs personalize responses by reasoning over user history, but existing approaches assume query-relevant preferences are semantically retrievable. VIBE-BENCH isolates PRCM, where profile cues and target preferences occupy different concept spaces and require cross-concept mappings.
- PLLMs use user-specific data to move beyond one-size-fits-all responses.
- Existing personalization methods typically retrieve semantically related history before generating personalized responses.
- Indirect or semantically mismatched evidence makes better retrieval insufficient for inferring query-relevant preferences.
- PRCM occurs when profile cues and target preferences lie in different concept spaces, requiring cross-concept mappings rather than semantic matching.The paper illustrates this with personality cues from sports-related topics and emotion-regulation strategies.
- VIBE-BENCH provides two psychology-grounded tasks with 3,504 personas and 12,239 dialogues to isolate PRCM.
- Current personalization methods favor semantic correlations over robust profile-preference mappings under PRCM.
2 Related Work
Prior PLLM benchmarks evaluate preference alignment through historical behavior, personas, dialogue, and long-context memory, but largely leave cross-concept reasoning unexplored. VIBE-BENCH contrasts with these benchmarks through low semantic similarity between histories and target queries or responses.
- Existing benchmarks evaluate preference induction and persona-consistent responses using classification, generation, multi-turn dialogue, and open-ended prompts.
- VIBE-BENCH measures semantic similarity distributions against prior benchmarks using query-history and answer-history comparisons.
- Prior implicit preference benchmarks generally emphasize semantically related evidence rather than cross-concept preference reasoning.
3 Task Formulation
The task formulation distinguishes reusable user profiles from query-specific preferences and defines personalization as conditioning responses on both. A PLLM therefore maps a query-profile pair to a response aligned with the user's current preference.
- A user profile is query-independent prior information shared across that user’s queries as a reusable long-term representation.
- A target preference consists of query-specific constraints and objectives that may vary across queries for the same user.
- A non-personalized LLM maps queries to responses, whereas a PLLM maps query-profile pairs to responses.
- The resulting response is expected to align with the user’s preferences given their profile.
4 Proposed Taxonomy
The taxonomy defines three preference-reasoning paradigms, culminating in PRCM. Cross-concept implicit reasoning requires mapping profile-level concepts to query-relevant preference concepts before optional refinement.
- Cross-Concept Implicit Preference Reasoning: Cross-concept reasoning applies a mapping ϕ from profile-level concepts to preference-level concepts, optionally followed by preprocessing and refinement.The mapping may use external knowledge, population-level statistics, or aligned ontologies.
- The taxonomy contains explicit extraction, within-concept implicit reasoning, and cross-concept implicit reasoning.
- Explicit Preference Extraction: Explicit preference extraction retrieves a target preference directly because it is encoded in the profile.
- Within-Concept Implicit Preference Reasoning: Within-concept implicit reasoning infers unstated preferences from conceptually related profile evidence and refines them into query-appropriate preferences.
- Cross-Concept Implicit Preference Reasoning: Under PRCM, semantic matching alone cannot retrieve the target preference because profile evidence and preference concepts occupy different spaces.
5 VIBE-BENCH Construction
VIBE-BENCH constructs a psychology-grounded benchmark for profile-preference conceptual misalignment using Big Five personality–emotion regulation and occupation–RIASEC interest relationships. It combines structured persona metadata, multi-session dialogues, shared profiles across two tasks, and manually verified evaluation data.
- Cross-Concept Design Principles: VIBE-BENCH grounds PRCM in Big Five personality–emotion regulation and Holland RIASEC occupation–interest relationships.These theories capture natural profile-preference misalignment, while the relevant traits are rarely stated explicitly in everyday dialogue.
- Metadata Collection: The benchmark contains 3,504 samples built from 876 occupations and structured personality, occupational, interest, and emotional-distress metadata.The metadata includes 696 high/low Big Five trait descriptions, 180 RIASEC activity-preference items, and sampled dialogue events.
- Profile Construction: Each profile contains 2–4 dialogue sessions, with at least one target-personality session and one target-interest session; 54.91% contain four sessions.Profiles contain 2, 3, and 4 sessions in 5.62%, 39.47%, and 54.91% of cases, respectively.
- Task Design: Task 1 infers a salient personality dimension and maps it to an emotion-regulation strategy for a tailored distress response.Task 2 infers a job title, derives its RIASEC interest type, and predicts whether it matches the user’s leisure-activity interest type.
- Task Design: Shared profiles support two distinct queries, while turns relevant to one task act as distractors for the other.This design increases the challenge of identifying query-relevant preferences from multi-session histories.
- Data Splits and Evaluation: The test set uses 32 occupations held out from training and validation and includes 128 manually verified and corrected gold samples.The full benchmark contains 12,239 dialogues and approximately 130K utterances, averaging 10.6 utterances per dialogue.
6 VIBE-BENCH Quality Assessment
The quality assessment tests whether dialogue histories recover target profile attributes and whether generated emotion-regulation responses meet five quality dimensions. Profile attributes are generally recoverable, while unseen-occupation classification is harder; expert prompting improves response quality.
- Information Recoverability: Personality and RIASEC recovery exceed 96% accuracy, while candidate-free Job-CLS reaches 65% on unseen occupations.The occupation result is attributed to the need for cross-occupation generalization under distribution shift.
- Response Quality Assessment: Figure 3 evaluates Task 1 responses on Helpfulness, Psychological Appropriateness, Conversational Naturalness, Event Grounding, and Strategy Realization.The baseline is vanilla GPT-5-mini without the expert-designed prompt template.
- Response Quality Assessment: The expert-designed prompt template consistently improves response quality over vanilla GPT-5-mini.The largest gains occur in Naturalness and Strategy Realization, with additional improvements in Event Grounding and Psychological Appropriateness.
- Response Quality Assessment: The generated responses average above 4 across all five dimensions, resembling realistic emotional-support scenarios.Scores are produced for 256 samples after agreement between expert and Claude.
- Main Results: Table 2 compares personalization methods against Base-LLM, with green indicating improvement and red indicating degradation.Darker colors represent larger differences, and full results appear in Tables 10 and 11.
7 Experiments
VIBE-Bench experiments show that standard personalization methods struggle with cross-concept preference reasoning, while explicit concept-aware guidance substantially improves performance.
- Main Results: Across evaluated methods, personalization gains over Base-LLM remain limited when profile cues and query preferences occupy different conceptual spaces.PAP-LLM performs strongest among non-parametric methods, whereas more history does not necessarily improve personalization.
- Main Results: Accuracy dropped from 21.48 to 18.36 and Macro-F1 from 14.15 to 10.36 when FHP-LLM concatenated full histories instead of using Base-LLM on Task 1 strategy metrics.The results associate long-history prompting with noise and redundancy that obscure query-relevant cues.
- Main Results: On Task 2, RAP-BERT achieved F1 61.59 versus 56.39 for RAP-BM25, but both retrieval methods remained weaker than PAP-LLM and P-SFT.Semantic retrieval outperformed lexical matching modestly but did not resolve the cross-concept reasoning challenge.
- Ablation Study on Benchmark Design: Under PRCM, P-SFT accuracy dropped to 58% on Task 1 and 10% on Task 2 when cross-concept mapping was required.P-SFT achieved perfect accuracy under the explicit-preference and within-concept paradigms, but not under PRCM.
- Ablation Study on Benchmark Design: Ablating profile inference caused only 7% and 4% degradation, indicating that cross-concept mapping, rather than semantic anchoring, is the primary PRCM bottleneck.The smaller degradation from removing profile inference suggests semantic mismatch contributes only marginal additional error.
8 Discussion
The discussion argues that PRCM requires models to discover latent concepts and mappings from weakly supervised interaction data. Explicit concept-aware reasoning helps, but automatically induced mappings can diverge from dataset regularities.
- Discussion: The weakly supervised setting gives models user histories, current queries, and end-task supervision, requiring them to identify concepts, infer attributes, and induce mappings.Concept and mapping annotations are costly, domain-specific, and often unavailable, motivating reliance on scalable interaction data.
- Discussion: Concept-Aware Reasoning improved accuracy by 44% on Task 1 and 8% on Task 2 when ontology and cross-concept knowledge were explicitly available.The result suggests the main difficulty is discovering and inducing mappings rather than applying mappings once provided.
- Discussion: Figure 5 compares dataset ground-truth Big Five-emotion-regulation distributions with GPT-5-mini’s zero-shot induced distributions.Blue denotes the ground-truth distribution and green denotes the induced distribution.
- Discussion: Zero-shot mappings induced by GPT-5-mini deviated markedly from dataset-implicit mappings in the preliminary study.The study supplied full user-specific data, query-response pairs, and a manually defined concept set before aggregating concept-label frequencies.
9 Conclusion and Future Work
The paper establishes PRCM as a distinct preference-reasoning regime and introduces VIBE-BENCH to evaluate cross-concept personalization beyond semantic retrieval. Experiments identify robust profile-preference mapping as the central bottleneck and motivate training frameworks that can learn and internalize such mappings.
- PRCM is identified as a distinct, underexplored regime requiring cross-concept implicit preference reasoning beyond semantic retrieval.
- VIBE-BENCH provides a taxonomy and benchmark for evaluating cross-concept personalization beyond semantic retrieval.
- Current personalization methods exploit semantic correlations but struggle to induce robust profile-preference mappings under PRCM.
- Future training frameworks should induce profile-preference mappings from data and robustly internalize them into model parameters.
Limitations
VIBE-BENCH is a controlled diagnostic benchmark rather than a comprehensive evaluation of real-world personalization. Its coverage and mappings have important scope boundaries involving synthetic data, limited settings, uncertain population-level regularities, and incomplete method comparisons.
- VIBE-BENCH covers two psychology-grounded tasks, synthetic interaction histories, and English, so transfer to broader real-world settings remains open.The unresolved settings include naturally occurring histories, multilingual and cross-cultural use, additional domains, and real-user outcomes.
- The benchmark’s construction quality does not establish ecological validity for real-world personalization.The gold test set is manually verified, but the passage distinguishes this from demonstrating real-world validity.
- Profile-preference mappings represent probabilistic, context-dependent population regularities rather than deterministic prescriptions or individual-level ground truth.Users with similar inferred attributes may have heterogeneous preferences, and association strength or direction may vary across contexts and cultures.
- These mappings are weak priors for cold-start-like settings and should not override direct preferences or feedback.Overgeneralizing them to individuals may produce stereotyped or overly confident personalization.
- Experiments omit preference-optimization approaches, while automatic mapping discovery remains preliminary and relies on predefined concepts in Concept-Aware Reasoning.Future work is needed on automatic concept discovery, uncertain mapping induction, and robust internalization.
Ethical Considerations
The paper frames VIBE-BENCH as a research-only diagnostic rather than a deployment-ready personalization resource. Its scenarios, synthetic data, conceptual terminology, and occupational metadata support controlled study but impose ethical and interpretive boundaries.
- Intended Use: VIBE-BENCH uses Holland’s RIASEC framework and emotion-regulation strategies as test scenarios, not for real-world deployment.
- Intended Use: All training and dialogue data are synthetically generated by language models and may contain biases or unreliable inferences.
- Intended Use: The dataset is restricted to research use and prohibited for real-world emotion-regulation guidance, mental-health intervention, career planning, and other high-stakes decisions.
- Data Privacy, Licensing, and Terms: VIBE-BENCH contains no real user conversations or personal data, and external resources were used according to documented licenses and terms.
- Conceptual Framework: The paper defines concepts as human-interpretable abstractions and concept spaces as coherent domains whose concepts can be related or compared.Examples distinguish Big Five personality concepts from emotion-regulation concepts.
- Conceptual Framework: Cross-concept relationships span distinct domains and generally require mappings supported by external knowledge, theory, or population-level regularities.Personality traits and emotion-regulation strategies are given as an example.
- Concept-Mapping Distribution: The benchmark maps personality to emotion regulation and occupation to interest, with Table 5 deliberately clean but Table 6 limited to coarse occupational-group mappings.A major occupational group cannot fully capture the RIASEC profile implied by specific job responsibilities.
J Generation Artifact Diagnostic
The benchmark is synthetically generated, raising concern that models could exploit lexical, stylistic, or template artifacts rather than meaningful profile–preference relationships. A history-shuffling diagnostic tests whether performance depends on correspondence between user histories and labels.
- Synthetic generation may introduce recurring lexical, stylistic, or template-level patterns that models could exploit instead of meaningful profile–preference relationships.
- The diagnostic randomly shuffles user histories while preserving Task 2 labels, marginal distributions, and generation characteristics.
- After shuffling, Qwen3-4B and Llama3.1-8B fall to approximately chance-level accuracy, with MCC scores of 0.05 and −0.03, respectively.The reported recall remains relatively high because of degenerate prediction bias, whereas MCC accounts for all confusion-matrix entries.
K Details of Semantic Similarity Analysis
VIBE-BENCH evaluates semantic similarity between user histories and queries or answers while constructing controlled persona and dialogue data from psychology-grounded profile cues. Its generation prompts enforce profile consistency, realistic dialogue, and structured occupation- and interest-related content.
- Semantic Similarity Analysis: BERTScore uses the maximum similarity between each query or answer and any historical user utterance to test whether evidence is directly retrievable by semantic matching.The low similarity observed for VIBE-BENCH supports its intended cross-concept design.
- Dataset Distributions: The benchmark includes distributions for emotion-regulation mappings, occupational-to-RIASEC mappings, Big Five traits, RIASEC types, and occupations.These distributions are reported in Tables 5, 6, 7, 8, and 9.
- Results Reporting: Task 1 and Task 2 results are reported across personalization methods and backbone models in Tables 10 and 11.
- Persona Construction: Persona-card prompts generate profiles with names, ages, genders, jobs, responsibilities, education levels, interests, and Big Five traits while enforcing logical consistency and individualized details.Occupation-related prompts also require structured job responsibilities and interests fields.
- Occupation Experience Generation: Occupation-experience prompts ask for first-person narratives grounded in job content and Big Five traits, with realistic, detailed, and profile-consistent writing.