Source-linked AI summary
When LLM-Inferred User Context Adds Value in Production Streaming Recommendation
Milad Sabouri, Neeraj Sharma, Sardar Hamidian, Shaghayegh Agah
TL;DR
Predefined context variables miss factors that are neither enumerable in advance nor directly observable, motivating context inferred from behavioral records. We compare semantic profiling strategies and find that the stronger representation depends on users’ consumption regimes, with LLM-generated profiles also showing a popularity-attractor trade-off.
Problem
Predefined context variables miss factors that are neither enumerable in advance nor directly observable, motivating context inferred from behavioral records.
Source: §1 · p. 1Method
The study compares four semantic user-profiling strategies that cross representation type and contextual scope on a production streaming platform.
Source: §6 · p. 5Results
Aggregate profiles are stronger for habitual users, while LLM-generated profiles are stronger for Explorers; LLM strategies also raise within-list diversity but lower coverage and novelty.
Source: §1 · p. 2, §4.1.3 · p. 3, §4.2.2 · p. 4Takeaways & Limitations
Because the stronger representation varies by consumption regime, the findings support regime-sensitive profiling rather than one representation for every user.
Source: §1 · p. 2, §6 · p. 5The evaluation covers one platform, content domain, and generation and encoding pipeline, and is offline; the identified regimes should not be assumed to transfer unchanged.
Source: §5.3 · p. 5
Abstract
from arXiv · showhide
Contextual information in recommender systems is shifting from static, predefined variables toward latent representations inferred from behavior. Large language models support this shift by rendering an unstructured interaction history as a natural-language summary, which yields a thematic user context that can be encoded and used in place of an aggregate profile. The conditions under which such generated profiles outperform aggregate embeddings have received limited characterization mainly at the domain level. We evaluate semantic user-profiling strategies on a production streaming platform, ranking against the full catalog. The evaluation covers a 2*2 design space crossing representation type (aggregate or LLM-generated) with contextual scope (holistic history or attention-fused short-term and long-term contexts). The relative ordering of the two representation types is conditional on the user's consumption regime. Aggregate profiles are consistently stronger under habitual consumption, which characterizes approximately four-fifths of the population, while LLM-generated profiles are stronger for exploratory users whose subsequent interactions diverge semantically from their history. We also observe a popularity-attractor effect in LLM-generated profiles, which modestly raises within-list diversity while substantially lowering catalog coverage and reducing novelty. These results indicate that a context-aware system can select a profiling strategy from the inferred consumption regime rather than applying one representation to all users.
1 Introduction
Context inferred from behavior offers an alternative to predefined context variables, but prior work had mainly characterized profile differences across domains. This study compares four profiling strategies on a production streaming platform and finds that their relative performance depends on users’ consumption regimes.
1 Introduction
- Behavior can serve as a source of latent context because many consumption-shaping factors are not enumerable in advance or directly observable.An LLM can summarize an interaction history into a natural-language profile that is encoded for downstream recommendation.Source: §1 · p. 1
- Earlier public-benchmark work found domain-varying differences between LLM-generated and aggregate profiles, but did not test conditionality among users within one dataset.Those evaluations used smaller catalogs and measured accuracy alone.Source: §1 · p. 1
- The study compares four profiling strategies on a production streaming platform, ranking against its full filtered catalog and holding the item encoder, scoring head, and training protocol fixed.It also reports beyond-accuracy metrics and analyzes users by whether subsequent consumption revisits content in their history.Source: §1 · pp. 1–2
- Aggregate profiles are stronger for habitual users, while LLM-generated profiles are stronger for exploratory users whose later interactions diverge semantically from their history.The findings also report that the effect of contextual scope depends on representation type and consumption regime.Source: §1 · p. 2
- LLM-generated profiles modestly raise within-list diversity but lower catalog coverage and novelty, concentrating recommendations on a smaller, more popular part of the catalog.Contextual adaptation redistributes this popularity bias rather than removing it.Source: §1 · p. 2
2 Related Work
Prior work covers predefined context variables, aggregate and LLM-generated user profiles, and temporal modeling that separates recent from historical behavior. This paper situates its approach at the intersection of behavior-inferred context and content-based ranking.
Context in recommender systems
- Context-aware recommendation has generally used predefined, directly observed variables, while language models have also enriched familiar geo-temporal variables with semantic information,,,.In contrast, this work infers context from the behavioral record without designating a variable in advance.Source: §2 · p. 2
Aggregate and LLM-generated user profiles
Temporal structure in user modeling
3 Methodology
The methodology compares four semantic profiling strategies that vary by profile representation and history scope. All share a downstream architecture and differ only in how the user profile vector is constructed.
3 Methodology
- The four strategies cross aggregate or LLM-generated profiles with full-history or attention-fused short-term and long-term contexts.Centroid, TemporalCentroid, Narrative, and TemporalNarrative instantiate these combinations.Source: §3 · p. 2
3.1 Notation and scoring
- The task scores every catalog item for a user and returns the top-K using a learned MLP over the profile vector and item embedding.The item embedding is SBERT applied to item metadata.Source: §3.1 · p. 2
3.2.1 Centroid (C).
- Centroid averages the SBERT embeddings of all history items, without temporal structure or a natural-language artifact.It serves as the reference profile for comparisons.Source: §3.2.1 · p. 3
3.2.2 TemporalCentroid (TC).
- TemporalCentroid computes short-term and full-history centroids, then combines them with learned attention.The long-term centroid uses the full history, so recent interactions can appear in both segments.Source: §3.2.2 · p. 3
3.2.3 Narrative (N).
- Narrative prompts the LLM once over the complete history and encodes its single preference summary with SBERT.This removes the temporal split used by TemporalNarrative.Source: §3.2.3 · p. 3
3.2.4 TemporalNarrative (TN).
- TemporalNarrative prompts the LLM separately for recent shifts and themes stable across the full history, then fuses the encoded summaries with attention.Its short-term and full-history segments follow the same window and segment definitions as TemporalCentroid.Source: §3.2.4 · p. 3
3.3 Shared components
- The shared scoring head is a two-layer ReLU MLP; the accuracy table reports percentage changes against Centroid at PCT = 25%.Training uses binary cross-entropy with five uniformly sampled negatives per positive and Adam; the stated significance threshold is α = 0.05.Source: §3.3 · p. 3, Table 2 · p. 3
4.1 Experimental setup
The evaluation uses a temporally split production-streaming sample and holds the content-based paradigm fixed while comparing consumption regimes and profile metrics.
4.1 Experimental setup
- The sample contains 10,000 users and approximately 2.4 × 10^5 English-described items, with each user's history split chronologically into 80% training, 10% validation, and 10% test.The chronological split is per user and prevents within-user temporal leakage.Source: §4.1.1 · p. 3
4.1.2 Scope.
- Holding content-based recommendation fixed isolates profile design; sequence-based and collaborative methods are excluded because they use complementary collaborative signals.The paper identifies SASRec and BERT4Rec as examples of the excluded methods.Source: §4.1.2 · p. 3
4.1.3 Consumption regimes.
- Explorers have no test item in their training history; the sample includes 1,761 Explorers, approximately 18%, while Non-Explorers have at least one overlap.The label is post-hoc and analytical; results are reported for All and for each regime.Source: §4.1.3 · p. 3
4.1.4 Evaluation metrics.
- Six metrics at K=10 and 100 cover accuracy (Recall, NDCG, HitRate) and beyond-accuracy (Diversity, Coverage, Novelty).Diversity measures within-list cosine dissimilarity; Coverage measures the catalog fraction recommended across users. Novelty is mean self-information, and lower Novelty indicates more popular items.Source: §4.1.4 · pp. 3–4
4.1.5 Short-term window.
- PCT is the percentage of history assigned to the short-term segment, swept from 15% to 30%; Centroid and Narrative are PCT-invariant.The sweep uses ρ values of 0.15, 0.20, 0.25, and 0.30.Source: §4.1.5 · p. 4
- Cross-model comparisons use Centroid at the same segment and cutoff as reference, with pairwise per-user Wilcoxon tests at PCT=25%.Tests are independent for each metric, cutoff, and regime; system-level Coverage is not subject to per-user testing.Source: §4.1.6 · p. 4
4.2 Results
Accuracy rankings reverse by consumption regime, while LLM profiles trade modest within-list diversity gains for lower catalog coverage and novelty. Temporal-profile scope also interacts with regime, and Explorer results are more sensitive to the short-term window.
4.2 Results
- For All, Narrative and TemporalNarrative trail Centroid on Recall@10 by 30.7% and 21.6%; for Explorers, they lead by 18.7% and 14.3%, while TemporalCentroid trails by 7.4%.Non-Explorers are approximately 82% of the sample, so All tracks them closely; Explorer direction also holds across NDCG, HitRate, and K=100, though significance is partial.Source: §4.2.1 · p. 4
4.2.1 Accuracy by regime.
- On All and Non-Explorers, TemporalNarrative leads Narrative by approximately 9 Recall@10 points and 11 to 12 NDCG@10 points; Explorer ordering reverses at K=10.At K=100, the Explorer strategies fall within about three points and their order is unstable.Source: §4.2.1 · p. 4
4.2.2 Beyond-accuracy.
- LLM strategies raise Diversity over Centroid by approximately 4% at K=10 and 3% at K=100, but lower Coverage by approximately 80% and Novelty by 18 to 22%.Diversity and Novelty comparisons are significant across both cutoffs and all three regimes; Coverage and Novelty gaps widen at K=100.Source: Table 3 · p. 4, §4.2.2 · p. 4
- For Explorers, TemporalNarrative Recall@10 falls approximately 9.5% at PCT=15% and 19% at 30%, while TemporalCentroid falls approximately 11% at 30%.Both temporal strategies degrade most at 30%; the reported results use PCT=25%.Source: §4.2.3 · p. 4
5 Discussion
Aggregate profiles fit habitual users’ item-level preferences, while LLM-generated profiles’ thematic abstraction better reaches beyond history for exploratory users. The discussion connects these regime differences to profile scope, adaptive selection, and limits on the findings’ transfer.
5.1 Interpreting the regime-conditional effect
- Thematic abstraction helps LLM-generated profiles reach unseen items for Explorers, while Centroid profiles’ proximity to consumed items suits habitual users.The authors offer this as one explanation for the regime-conditional ordering.Source: §5.1 · p. 4
- LLM-generated profiles’ popularity attractor is hypothesized to reflect denser semantic associations for popular items in both LLM training text and encoder training data.The proposed explanation is consistent with modestly higher within-list diversity alongside lower novelty and coverage, but generation and encoding contributions were not separated.Source: §5.1 · p. 5
- Temporal scope changes little for aggregate profiles, but its effect for LLM profiles depends on regime: TemporalNarrative gains roughly nine points of Recall@10 over Narrative for habitual users, while Narrative leads at K=10 for Explorers.The authors interpret the temporal split as making LLM profiles more concrete, a quality that suits habitual but not exploratory consumption.Source: §5.1 · p. 5
5.2 Context-adaptive strategy selection
- If a user’s regime can be estimated at serving time, a system could select the matching profile strategy; Explorers make up approximately 18% of the sample.The discussion treats profiling strategy itself as context-dependent, following a direction proposed from domain-level differences.Source: §5.2 · p. 5
- History size and within-history content diversity correlate with Explorer status (r = −0.30 and r = +0.11, respectively; both p < 0.001), suggesting the regime is partially inferable before the subsequent consumption.The authors qualify this as evidence of feasibility, not a predictor.Source: §5.2 · p. 5
- For exploratory users, adaptive selection concentrates the popularity-attractor effect, lowering catalog coverage and novelty for a group a system might otherwise use to broaden exposure.The authors conclude that adaptation redistributes this bias rather than removing it.Source: §5.2 · p. 5
5.3 Limitations
- The evidence comes from one platform, content domain, and generation-and-encoding pipeline; the evaluation is offline, and whether the findings transfer or hold online remains untested.The authors also note that whether earlier public-data domain differences reduce to regime composition remains untested.Source: §5.3 · p. 5
6 Conclusion
On a production streaming platform, the study compares four profile strategies and finds their relative effectiveness depends on users’ consumption regimes. The conclusion supports selecting profiles by inferred regime, while noting that LLM-generated profiles concentrate recommendations on a smaller, more popular part of the catalog.
6 Conclusion
- Aggregate profiles are stronger for habitual users, while LLM-generated context is stronger for exploratory users whose subsequent consumption diverges from history.The four strategies cross representation type with contextual scope.Source: §6 · p. 5
- Contextual scope changes little for aggregate profiles but matters for LLM profiles, with its direction also depending on regime.The evaluation is based on a sample from a production streaming platform.Source: §6 · p. 5
- LLM-generated profiles concentrate recommendations on a smaller, more popular part of the catalog, while training-period features correlate with regime and suggest it is partially inferable before the described consumption.The authors support treating profiling strategy as context-dependent and selecting the representation matched to the inferred regime.Source: §6 · p. 5