Source-linked AI summary
Textual User Taste: Natural-Language User Context for Foundation-Model Recommender System at Scale
Ghazal Fazelnia, Paul Gigioli, Eliza Klyce, Sharon Zheng, Katie Zelvin, Ye Myat Thein, Anurag Deshpande, Seda Davtyan, Kate Remeika, Maya Hristakeva, Erik Franco, Karen Banzon, Peng Ge, Jacqueline Wood, Nandini Singh, David Murgatroyd, Mounia Lalmas, Yves Raimond, Andreas Damianou
TL;DR
Foundation-model recommenders need user context that can be consumed and refined through natural-language interaction, while the paper frames lifecycle management of these representations as an open question. Textual User Taste provides a production framework for textual profiles; the paper concludes that these profiles complement behavioral embeddings as an interpretable, steerable interface rather than replacing them.
Problem
The lifecycle of textual user representations shared across products—how to generate, evaluate, refresh, and govern them—remains an open question, while users need to correct preference assumptions or express new context.
Source: §1 · pp. 1–2Method
The paper presents an end-to-end production framework for generating, evaluating, maintaining, and serving structured textual user representations at scale.
Source: §1 · p. 2Results
Behavioral embeddings remain stronger for compact prediction, while text provides semantic organization, inspectability, and an interface for explicit context.
Source: §6 · p. 8Takeaways & Limitations
Taste profiles are contextual interfaces that complement, rather than replace, behavioral embeddings for evolving user context and foundation-model recommendation.
Source: §6 · p. 8Requests using negation, such as “no electronic music,” can shift recommendations toward the named category rather than the negated meaning.
Source: §5.3 · p. 7
Abstract
from arXiv · showhide
Foundation model recommender systems require user context that can be consumed by large language models, reasoned over, and refined through natural-language interaction. Traditional behavioral embedding vectors remain highly effective for retrieval and ranking, but they are opaque to users and not natively expressed for language model workflows. We present Textual User Taste, a system that generates structured natural-language taste profiles from listening behavior, interaction signals, content metadata, and optional user feedback, and deploys them to millions of Spotify users. We describe the end-to-end production lifecycle required to generate, evaluate, optimize, and maintain these representations at industrial scale, including prompt development and compression, user steering, and integration with downstream personalization systems. Because no unique ground-truth taste profile exists, we introduce a multi-faceted evaluation framework to evaluate taste profiles as a production representation: they carry user-specific predictive signal independently, and when integrated with behavioral embeddings, improve MRR by 0.6% for future-track prediction and NDCG@7 by 2.2% for search ranking. Our evaluation also reveals that taste profiles support positive natural-language steering, while exposing important limitations, including challenges with negation and short-term temporal adaptation. These findings position taste profiles not as replacements for behavioral embeddings, but as an interpretable and steerable interface between evolving user context and foundation-model recommender systems.
1 Introduction
Foundation-model recommenders need user context that is usable in language-model workflows and that users can inspect and refine. The paper presents Textual User Taste as a production lifecycle for generating and evaluating such profiles, treating them as operational representations while addressing the lack of a single ground truth and other deployment challenges.
1 Introduction
- Behavioral representations are compact and predictive but opaque, while foundation-model workflows need textual context they can consume and reason over.The paper frames textual profiles as an interface alongside behavioral representations, not as a replacement for them.Source: §1 · p. 1
- Users can inspect and refine profiles, correct mistaken assumptions about their preferences, or express new contextual preferences.Profiles summarize listening patterns across content types using behavioral signals and optional user-authored feedback.Source: §1 · pp. 1–2
- A taste profile offers language models a cached, semantically organized representation of longer-term user preferences and consumption patterns.The motivation includes the expense and practical context-budget limits of passing raw interaction histories to downstream models.Source: §1 · p. 1
- Taste profiles are evaluated as operational user representations, requiring grounding, user specificity, downstream influence, and predictable response to feedback.The paper treats generation and evaluation as a closed loop in which evaluation informs generation, optimization, deployment, and monitoring.Source: §1 · p. 2
- Because no unique reference profile exists, production use also raises challenges of grounding, consistency across regenerations, downstream propagation, and sensitive or unsupported inferences.Fluent language alone may not establish grounding in observed behavior.Source: §1 · p. 2
- The paper presents an end-to-end production framework for generating, evaluating, maintaining, and serving structured textual user representations at scale.Textual User Taste is Spotify’s production system for generating and serving taste profiles.Source: §1 · p. 2
2 Related Work
Prior work studies textual profiles, scalable persona generation, and combinations of textual and behavioral representations, often within a single domain, task, or setting. This paper focuses instead on persistent, user-visible taste profiles that can be refined through natural-language feedback and shared across production systems.
Textual user profiles
- Earlier studies establish textual profiles for recommendation, transparency, control, and cold-start prediction, but often focus on one domain, task, or experimental setting.The paper contrasts that scope with persistent context shared across production systems.Source: §2 · p. 2
- The paper addresses the gap by studying taste profiles as persistent context shared across production systems.Related work includes editable profiles, incremental updates from reviews, and multi-interest profiles for recommendation agents [13], [3].Source: §2 · p. 2
Industrial scale and complementary representations
- Scalable persona-generation work demonstrates industrial feasibility, while leaving persistent, user-visible, multi-content profiles with natural-language refinement and reuse across systems unaddressed.The cited work uses distillation, asynchronous inference, and clustered input representations [18].Source: §2 · p. 2
- Other approaches combine textual and behavioral representations, but leave open how to evaluate, maintain, and operate them as production context.This paper treats the representations as complementary production components rather than proposing a new fusion architecture.Source: §2 · pp. 2–3
3 Textual User Taste System
Textual User Taste turns aggregated listening context into structured profiles for downstream systems, then evaluates and refines their generation in a production lifecycle.
3.1 Inputs and Profile Structure
- Profiles draw on multi-horizon interaction signals and content metadata, while optional filtered user notes remain distinct from observed behavior.Aggregating signals reduces prompt size and avoids passing complete interaction histories to the generator.Source: §3.1 · p. 3
- Four profile fields cover music taste, podcast taste, audiobook taste, and listening rhythms, letting downstream systems select relevant fields and avoid unsupported inferences when evidence is sparse.The schema separates enduring content preferences from consumption context.Source: §3.1 · p. 3
3.2 Generation, Guardrails, and Serving
- The prompt prioritizes recent evidence while retaining durable preferences and instructs the model to avoid unsupported entities and sensitive inferences, with safeguards applied before and after generation.Input safeguards filter unsafe or adversarial notes; output safeguards monitor risks including protected-attribute inference and personally identifiable information.Source: §3.2 · p. 3
- Stored profiles serve user-facing and backend consumers rather than being regenerated for each request, while evaluation and feedback inform later generations.The model and prompt are versioned together so candidate versions can be evaluated before deployment and monitored after rollout.Source: §3.2 · pp. 3–4
3.3 Prompt Development and Optimization
- Across seven prompt versions, factuality rose from 7.70 to 8.57 and coherence from 6.46 to 7.98, while a shorter variant achieved the best specificity.The longest prompt led in factuality and coherence, showing that prompt length alone is a poor proxy for quality.Source: §3.3 · p. 4
- Two-fold prompt compression preserved factuality and structured validity; a prompt achieved an 80.2% composite objective (average of evaluation metrics) using approximately 2,000 tokens, one-third the length of the best hand-authored prompt.Source: §3.3 · p. 4
4 Evaluation Methodology
Because taste profiles have no unique ground truth, the paper evaluates them with complementary intrinsic and functional tests. The methodology also tests robustness, downstream utility, steering, representation relationships, and short-term stability.
4.1 Intrinsic Quality and Robustness
- Seven LLM judges assess factuality, coherence, coverage, specificity, diversity, relevance, and safety, while roughly 50 stress scenarios test safety and handling of conflicts between behavior and feedback.Judge scores are diagnostic signals rather than definitive quality measures, and intrinsic assessment is complemented by functional evaluation.Source: §4.1 · p. 5
4.2 Behavioral and Downstream Tests
- Functional tests measure future-track prediction, user specificity, search-ranking utility, and steering, including negative, situational, and multi-turn requests.The steering study tests embedding-based recommendation changes, not user satisfaction.Source: §4.2 · pp. 5–6
- Future-track prediction compares behavioral embeddings, taste-profile embeddings, their concatenation, and gated fusion; search ranking tests four context configurations across 10,000 query sessions.The prediction evaluation uses 400,000 user–track examples, splits 80/20 by user, and averages five runs across MRR, ROC-AUC, and NDCG@10.Source: §4.2 · p. 5
4.3 Representation Complementarity and Temporal Stability
- Representation comparison uses RSA with Spearman correlation, nearest-neighbor overlap at k=5, and ARI, while stability tests regenerate profiles after three and seven days under largely unchanged inputs.This stability setup measures generation consistency rather than responsiveness to genuine preference change.Source: §4.3 · p. 6
5 Results
Taste profiles provide complementary predictive signal alongside behavioral embeddings and can improve downstream search ranking. The results also show positive steering, challenges with negation, modest structural agreement, and high short-term stability under largely unchanged inputs.
5 Results
- Two intrinsic-quality components explain 93.7% of variance, and factuality and relevance are only weakly negatively correlated (r=−0.18).The findings support evaluating profile quality along multiple complementary dimensions rather than with one intrinsic metric.Source: §5 · p. 6
5.1 Text Adds Complementary Predictive Signal
- MRR rises from 0.956 to 0.962 with concatenated representations, while taste profiles alone underperform behavioral embeddings across all three prediction metrics.The profile’s user-specific signal is predictive: using the user’s own profile raises MRR from 0.656 to 0.776, while another user’s profile reduces it to 0.477.Source: §5.1 · pp. 6–7
5.2 Downstream Context Works Best in Combination
- 2.2%: combining taste profiles and behavioral embeddings yields the largest SessionNDCG@7 improvement over the no-user-context baseline.Across a second temporal partition, standalone gains are inconsistent while the combined representations remain consistently positive.Source: Table 3 · p. 7, §5.2 · p. 7
5.3 Text Is Steerable, but Not by Simple Negation
- Positive genre and mood requests increase mean Top-20 precision from 56% to 74%, but negated requests can move recommendations toward the named genre rather than away from it.Situational requests produce larger shifts than genre- or mood-based requests, with a mean shift of 0.069; multi-turn interactions refine recommendations toward more specific preferences.Source: §5.3 · p. 7
5.4 Structural and Temporal Diagnostics
- Taste profiles and behavioral embeddings show modest structural agreement: RSA Spearman correlation is ρ=0.286, nearest-neighbor overlap is 17.5% at k=5, and ARI is 0.08.These metrics describe structural relationships, not the information captured by either representation; complementarity is more directly evidenced by downstream improvements when both are used.Source: §5.4 · p. 8
- Full-profile embedding cosine similarity is 0.954 after three days and 0.955 after seven days, with no significant differences across seven output measures.Approximately 95.0% and 94.4% of inputs remain unchanged over the respective intervals, so responsiveness to genuine preference change remains untested.Source: §5.4 · p. 8
6 Discussion and Conclusion
Taste profiles are best treated as contextual interfaces that complement behavioral embeddings, with their value judged through downstream consumers. The discussion highlights trade-offs in evaluation, adaptation, and safe operation at scale.
6 Discussion and Conclusion
- Taste profiles should serve as contextual interfaces rather than standalone substitutes, and evaluation should follow their consumers because intrinsic dimensions are difficult to define.The conclusion frames them as operational user representations rather than behavioral replacements.Source: §6 · p. 8
- Behavioral embeddings remain stronger for compact prediction, while text adds semantic organization and inspectability; the most consistent gains come from combining them.The representations are not interchangeable, and profile versions produce different downstream trade-offs.Source: §6 · p. 8
- Text and behavioral representations share behavioral inputs, but language-model knowledge and prompt design give text a different inductive bias that may diminish as encoders improve.The authors note that richer metadata or language-model reasoning in behavioral encoders could reduce this complementarity.Source: §6 · p. 8
- Positive requests can steer profiles, but negation and conflicts between stated and observed preferences require explicit policies and signal provenance.Refresh strategies must balance stability under unchanged inputs with responsiveness to meaningful events, and safety must be enforced at the representation layer because inferences are visible and reused.Source: §6 · p. 8
- Without gold labels, the evaluation suite supports comparing system variants better than establishing absolute profile quality.The suite measures factuality, coverage, specificity, relevance, safety, behavioral signal, and downstream utility.Source: §6 · p. 8
- Mature interaction patterns and long-term user-value measures are unavailable, proprietary data and infrastructure limit reproduction, and deployment does not ensure equal profile accuracy.The authors say each application still requires controlled evaluation and monitoring.Source: §6 · p. 8
- Textual User Taste demonstrates that natural-language user context can be generated, evaluated, refreshed, safeguarded, and served at scale.Profiles are presented as a complementary, interpretable, and steerable interface between evolving context and foundation-model recommenders.Source: §6 · p. 8