Source-linked AI summary
When LLM-Based User Profiling Adds Value in Production Streaming Recommendation
Milad Sabouri, Neeraj Sharma, Sardar Hamidian, Shaghayegh Agah
TL;DR
In production streaming recommendation, it is unclear when costly LLM-generated user profiles are worth their added complexity. The paper compares four profile strategies across representation type and temporal handling, finding that their relative accuracy depends on user behavior and that LLM profiling reduces catalog coverage and novelty.
Problem
In production streaming, it is unclear when LLM-based profiles’ accuracy benefits justify their added complexity relative to simpler aggregate profiles.
Source: §1 · p. 1Method
The paper compares four profiling strategies crossing aggregate or LLM-generated representations with full-history or temporally disentangled profiles on production streaming data.
Source: §6 · p. 8Results
Aggregate methods are stronger for habitual users, while LLM-based methods gain accuracy among exploratory users and reduce catalog coverage and recommendation novelty.
Source: §1 · pp. 1–2, §6 · p. 8Takeaways & Limitations
Aggregate methods remain a robust default, while LLM profiling’s additional cost is justified selectively on segments where it yields measurable accuracy gains.
Source: §1 · p. 2The evaluation is offline on one platform and content domain, so generalization to other settings and online dynamics remains open.
Source: §5.4 · p. 8
Abstract
from arXiv · showhide
Personalized recommendation depends critically on how user representations are constructed from historical behavior. Two paradigms have emerged for constructing semantic user profiles in content-based recommendation. First, aggregate methods derive user representations as numerical aggregates of semantic item embeddings. Second, LLM-based methods generate natural-language summaries of user preferences and encode them through a text encoder. Each paradigm can be combined with temporal disentanglement of recent versus historical behavior. LLM-based profile generation is significantly more expensive than aggregate approaches, raising the question of when this additional cost is justified. We present a systematic comparison of four semantic user-profiling strategies, factorially crossed across representation type and temporal handling, evaluated on a real-world production dataset. The comparison reveals how these strategies differ across user behavior types, across both accuracy and beyond-accuracy dimensions of recommendation quality, and across the temporal-window setting that governs the disentanglement.
1 Introduction
User-profile design shapes recommendation accuracy and production costs, making the value of expensive LLM-generated profiles an open question for production streaming. A four-strategy comparison finds that effectiveness depends on user behavior and that LLM-based profiling also reduces catalog coverage and novelty.
1 Introduction
- Production streaming raises the question of when LLM profiling’s benefits justify its additional computational, latency, and operational costs.The setting differs from academic evaluations in catalog size, history depth, and behavioral heterogeneity.Source: §1 · p. 1
- The study compares four strategies crossing aggregate versus LLM-generated profiles with full-history versus temporally disentangled profiles on production streaming data.The temporal strategies fuse short- and long-term representations through attention.Source: §1 · p. 1
- LLM-based methods gain accuracy among exploratory users, while aggregate methods dominate among habitual users.LLM profiling also concentrates recommendations on a smaller, more popular catalog subset, reducing coverage and novelty.Source: §1 · pp. 1–2
2 Related Work
Prior work spans aggregate and natural-language user profiles, temporal behavior modeling, and production-scale evaluation. This study draws on these traditions while examining segment-level and beyond-accuracy properties in a production setting.
Related work
- Aggregate profiles pool item features or semantic embeddings, while LLM-based profiles summarize preferences in natural language and encode the summary for recommendation.Prior LLM profiling work has been evaluated primarily on academic benchmarks, with production costs less characterized.Source: §2 · p. 2
- Temporal recommendation includes sequence models and approaches that separate short-term interests from long-term preferences and combine them through attention.The paper’s temporally disentangled strategies apply the latter framework within content-based ranking.Source: §2 · p. 2
- Production-scale research has raised concerns about candidate-pool size, full-catalog ranking, user heterogeneity, and beyond-accuracy metrics such as coverage and novelty.Segment-level analysis has also been advocated for identifying where recommendation methods produce value or fail.Source: §2 · p. 2
3 Methodology
The methodology isolates user-profile construction in a four-strategy, 2×2 design spanning representation type and temporal handling. All strategies share the downstream scoring and training infrastructure.
3 Methodology
- The study compares four semantic user-profiling strategies for content-based recommendation in a 2×2 design: numerical item-embedding aggregates or LLM-generated natural-language summaries, each using full-history or temporally disentangled short-term and long-term profiles fused via attention.Source: §3 · p. 2
- Each strategy constructs a user profile vector that is concatenated with an item embedding and scored by a learned MLP.The item embedding is produced from item metadata with SBERT, and the item encoder and evaluation protocol are held constant.Source: §3 · p. 2, §3.1 · p. 3
3.1.1 Centroid (C).
- Centroid represents a user by the arithmetic mean of SBERT embeddings for items in the user’s interaction history.It uses no temporal structure or natural-language artifact and serves as the reference strategy.Source: §3.1.1 · p. 3
3.1.2 TemporalCentroid (TC).
- TemporalCentroid computes mean item embeddings for recent and full-history segments, then combines the two centroids through attention.The long-term segment is the full history, so it includes the recent interactions rather than only the complement of the short-term segment.Source: §3.1.2 · p. 3
3.1.3 Narrative (N).
- Narrative prompts an LLM to summarize preferences across the user’s full history and encodes the resulting text with SBERT.The prompt asks for enduring themes and recurring patterns.Source: §3.1.3 · p. 3
3.1.4 TemporalNarrative (TN).
- TemporalNarrative prompts an LLM separately for recent interests and persistent preferences, encodes both summaries with SBERT, and fuses them through attention.It uses the same short-term window and segment definitions as TemporalCentroid.Source: §3.1.4 · p. 3
3.2 Shared infrastructure
- The strategies differ only in user-profile construction; shared infrastructure includes a frozen SBERT item encoder, an identical MLP, and binary-crossentropy training with negative sampling.This setup supports attributing evaluation differences to profile design rather than candidate scoring or item representation.Source: §3.2 · pp. 3–4
4 Experiments
The four strategies are evaluated offline on a sample from a production streaming platform.
4 Experiments
- The experiment is an offline evaluation of four profiling strategies on a sample drawn from a production streaming platform.Source: §4 · p. 4
4.1 Experimental setup
The experimental setup fixes a production content-based evaluation protocol, segments users by future behavior, and evaluates accuracy, beyond-accuracy quality, and temporal-window sensitivity.
4.1.1 Dataset and sample.
- Evaluation uses 10,000 production-platform users and roughly 2.4×10^5 English-text catalog items, with per-user chronological 80/10/10 training, validation, and test splits.The split is performed per user to prevent within-user temporal leakage.Source: §4.1.1 · p. 4
- Comparisons hold content-based ranking fixed and exclude sequence-based and collaborative methods, isolating user-profile design within that paradigm.Those methods use collaborative signals for next-item prediction, which would confound the signal source.Source: §4.1.2 · p. 4
4.1.3 User segmentation.
- Explorers have no test-history item in training; 1,761 users meet this post-hoc definition, with results reported for them, Non-Explorers, and all users.Non-Explorers have at least one test item in training; their consumption is characterized as habitual, unlike the Explorers’ limited past-behavior predictiveness.Source: §4.1.3 · p. 4
4.1.4 Evaluation metrics.
- At K=10 and 100, accuracy uses Recall@K, rank-aware NDCG@K, and HitRate@K, the fraction of users with a test item in the top-K list.Recall@K is the fraction of test-set items appearing in the user’s top-K list.Source: §4.1.4 · p. 4
- Beyond accuracy, Diversity@K measures within-list semantic spread, Coverage@K catalog footprint, and Novelty@K item self-information; lower novelty means more popular recommendations.Coverage is the fraction of the catalog appearing in any user’s top-K recommendations.Source: §4.1.4 · p. 4
4.1.5 Short-term window sweep.
- PCT is the percentage of chronological history assigned to the short-term segment; ρ is swept at 15%, 20%, 25%, and 30%, while Centroid and Narrative remain invariant.Centroid and Narrative provide reference values across the PCT settings.Source: §4.1.5 · p. 4
4.1.6 Reporting and statistical testing.
- Cross-model comparisons use Centroid at the same segment and cutoff as reference; temporal-window trajectories instead normalize each model to its own ρ=25% value.Wilcoxon signed-rank tests compare per-user metrics at PCT=25% by metric, cutoff, and segment; Coverage is excluded because it is system-level.Source: §4.1.6 · p. 4
4.2 Results
At PCT = 25%, accuracy rankings depend on user segment: aggregate profiling performs better for habitual users, while LLM-based profiling performs better for Explorers. LLM-based profiles also trade modestly higher list diversity for lower catalog coverage and novelty, and temporal strategies are most stable with PCT from 15% to 25%.
4.2.1 Segment-conditional accuracy.
- On Non-Explorers, Narrative trails Centroid by approximately 32% and TemporalNarrative by approximately 22% on Recall@10; on Explorers, they lead by approximately 19% and 14%, respectively.TemporalCentroid is within a few percent of Centroid on Non-Explorers; on Explorers it is approximately 7% below Centroid, a difference that is not statistically significant.Source: §4.2.1 · p. 5, Table 2 · p. 5, Figure 1 · p. 6
- The segment-conditional flip is most robust at K = 100, where both LLM-based strategies significantly outperform Centroid on Explorer Recall and NDCG; Explorer K = 10 results are less conclusive.Explorer HitRate differences are not significant at either cutoff, and the K = 10 Recall differences are also not significant; the passage attributes this to the smaller Explorer sample.Source: §4.2.1 · p. 5
4.2.2 Beyond-accuracy trade-offs.
- LLM-based strategies raise within-list Diversity by approximately 4% but reduce Coverage by around 80% and Novelty by approximately 18–20% versus Centroid.These Coverage and Novelty deficits deepen at K = 100; TemporalCentroid remains within a few percent of Centroid on the three metrics.Source: Table 3 · p. 5, §4.2.2 · p. 6
4.2.3 Sensitivity to the short-term window.
- Both temporal strategies are stable at PCT values of 15%, 20%, and 25%, then degrade at 30%, with a larger drop on Explorers.On Explorers, TemporalNarrative drops to approximately −19% at PCT = 30%, while TemporalCentroid drops to approximately −11%; PCT = 25% is adopted as the default.Source: §4.2.3 · p. 6
5 Discussion
Profiling effectiveness depends on behavioral regime: aggregate methods suit habitual users, while thematic LLM profiles help on exploratory users. The discussion also weighs accuracy against catalog effects, deployment feasibility, cost, and evaluation scope.
5.1 Interpreting the segment-conditional effect
- Aggregate profiles win for habitual users, while thematic Narrative profiles outperform Centroid for Explorers whose future consumption diverges from past behavior.Centroid specificity helps when future items resemble the user's history; Narrative abstraction is useful when item-level affinity has limited predictive power.Source: §5.1 · p. 6
- LLM-based profiles modestly raise Diversity but lower Coverage and Novelty; a proposed popularity-attractor account remains unisolated across generation and encoding.The account links popular-item-heavy training distributions to profile vectors near popular items; the reported pattern is consistent across PCT settings and segments.Source: Figure 2 · p. 7, §5.1 · p. 7
- TemporalNarrative shows its largest Recall@10 degradation at PCT=30%, particularly for Explorers; Centroid and Narrative are PCT-invariant.Temporal disentanglement may help when recent and historical behavior form coherent complementary signals, but Explorer behavior departs from historical patterns.Source: Figure 3 · p. 7, §5.1 · p. 7
5.2 Deployment implications
- If inference-time Explorer detection is reliable, routing aggregate profiles to habitual users and LLM profiles to Explorers could target accuracy gains; training-history size and diversity correlate with Explorer status.This is preliminary rather than deployable: detection construction, calibration, and online validation remain; aggregate profiling is a fallback for an unavailable or noisy signal.Source: §5.2 · pp. 7–8
5.3 Cost considerations
- Selective routing would concentrate LLM generation on approximately 18% of the sample, paying its greater compute, latency, and operational costs where accuracy gains occur.Quantitative cost-benefit estimates depend on platform-specific scale, refresh cadence, compute pricing, and segment engagement value.Source: §5.3 · p. 8
5.4 Limitations
- The offline evaluation uses a sample from one production streaming platform and content domain, so generalization to other platforms, domains, pipelines, and online settings remains open.The authors recommend cautious extrapolation to platforms with substantially different user-base characteristics or interaction patterns.Source: §5.4 · p. 8
6 Conclusion
In a full filtered-catalog evaluation on production data, profiling effectiveness depended on behavioral regime: aggregate methods favored habitual users, while LLM methods gained accuracy for Explorers. LLM profiling also reduced coverage and novelty, supporting aggregate profiles as a robust default and selective LLM use.
6 Conclusion
- The comparison factorially varied representation type and temporal handling across four strategies, using real-world production data and ranking against the full filtered catalog.The strategies were compared for content-based recommendation.Source: §6 · p. 8
- Aggregate methods were stronger for habitual users, while LLM-based methods gained accuracy for Explorers whose future consumption diverges from past behavior.Source: §6 · p. 8
- LLM-based profiling concentrated recommendations on a smaller, more popular catalog slice, reducing catalog coverage and recommendation novelty.Source: §6 · p. 8
- Aggregate methods remain a robust default; the added cost of LLM-based profiling is justified selectively where it delivers measurable accuracy gains.Source: §6 · p. 8