Source-linked AI summary
Large Language Model Simulator for Cold-Start Recommendation
Feiran Huang, Yuanchen Bei, Zhenghang Yang, Junyi Jiang, Hao Chen, Qijie Shen, Senzhang Wang, Fakhri Karray, Philip S. Yu
TL;DR
Cold items lack historical behaviors for learning embeddings, and current content-based synthetic embeddings do not address that underlying gap. ColdLLM simulates user interactions with an LLM and uses a coupled filter to narrow the candidate users. The paper reports a 21.69% improvement in cold recommendation performance over existing solutions and a two-week A/B test validating its superiority.
Problem
Cold items lack historical behavior data for learning embeddings, while existing content-derived synthetic embeddings leave this cold-start gap unresolved.
Source: §1 · pp. 1–2Method
ColdLLM simulates user interactions for cold items with an LLM and uses a trained coupled filter to reduce candidate users from billions to hundreds.
Source: §1 · p. 2, §4.1.1 · p. 3Results
21.69% improvement in cold recommendation performance over existing solutions was reported, and a two-week A/B test validated ColdLLM’s superiority.
Source: §1 · p. 2Takeaways & Limitations
Online and offline experiments verified ColdLLM’s strength, and the paper says it opens possibilities for using LLMs in large-scale online recommendations.
Source: §6 · p. 8The industrial scenario may pose computational and storage challenges for academic researchers.
Source: Ethical Considerations · p. 9
Abstract
from arXiv · showhide
Recommending cold items remains a significant challenge in billion-scale online recommendation systems. While warm items benefit from historical user behaviors, cold items rely solely on content features, limiting their recommendation performance and impacting user experience and revenue. Current models generate synthetic behavioral embeddings from content features but fail to address the core issue: the absence of historical behavior data. To tackle this, we introduce the LLM Simulator framework, which leverages large language models to simulate user interactions for cold items, fundamentally addressing the cold-start problem. However, simply using LLM to traverse all users can introduce significant complexity in billion-scale systems. To manage the computational complexity, we propose a coupled funnel ColdLLM framework for online recommendation. ColdLLM efficiently reduces the number of candidate users from billions to hundreds using a trained coupled filter, allowing the LLM to operate efficiently and effectively on the filtered set. Extensive experiments show that ColdLLM significantly surpasses baselines in cold-start recommendations, including Recall and NDCG metrics. A two-week A/B test also validates that ColdLLM can effectively increase the cold-start period GMV.
1 Introduction
Cold items lack the historical behaviors used to learn embeddings, while existing content-derived embeddings do not resolve that absence. ColdLLM simulates user interactions with an LLM and uses a coupled filter to scale the process; the paper reports improved cold recommendation performance.
1 Introduction
- Cold items lack historical user-item behaviors for learning embeddings, hindering their recommendation and affecting the recommender ecosystem and revenue.Source: §1 · p. 1
- Existing cold-start models rely on content-derived synthetic embeddings, leaving a content-behavior gap and often mixing content-based and behavior-based recommendations.Source: §1 · pp. 1–2
- Applying LLM simulation raises challenges of predicting intentions without interaction data, inference efficiency, and scalability to large user bases.Source: §1 · p. 2
- ColdLLM uses an LLM simulator to model user interactions for cold items and a trained coupled filter to reduce candidate users from billions to hundreds.Source: §1 · p. 2
- 21.69% improvement in cold recommendation performance over existing solutions is reported, alongside a two-week A/B test validating ColdLLM’s superiority.Source: §1 · p. 2
2 Related Works
Prior cold-start recommendation work includes embedding simulation and interaction simulation, while LLM recommendation studies apply language models to recommendation tasks and cold-start signals.
2.1 Cold-Start Item Recommendation
- Cold-start item recommendation concerns newly occurring items without behavioral interactions; approaches include aligning content-generated embeddings with warm-item behavior embeddings or simulating potential interactions.Source: §2.1 · p. 2
2.2 Recommendation with LLMs
- LLM recommendation work applies language models to recommendation capabilities, including tuning, strengthening interaction links, enriching item features, profiling users, and generating cold-item training signals.Source: §2.2 · pp. 2–3
- This paper focuses on combining LLM world knowledge with recommendation models’ collaborative-filtering capabilities for cold-start items.Source: §2.2 · p. 3
3 Preliminaries
The paper distinguishes warm items with historical interactions from strict cold-start items without them. Cold-item recommendations therefore use a different formulation from recommendations based on historical sequences.
3 Preliminaries
- Warm items have historical user sequences and learned behavioral embeddings, whereas cold items have no historical interactions and an empty user sequence.Source: §3 · p. 3
- Warm-item recommendation uses historical user sequences, while cold-item recommendation is formulated with the cold item's user sequence set empty.Source: §3 · p. 3
- The restricted cold-start task is to recommend both warm and cold items well.Source: §3 · p. 3
4 Methodology
The methodology section covers ColdLLM’s framework, simulation strategy, training, industrial implementation, and complexity.
4 Methodology
- The methodology proceeds from the general framework to coupled-funnel user-sequence simulation, LLM and filter training, and industrial implementation and complexity analysis.Source: §4 · p. 3
4.1 Overall Framework
ColdLLM replaces synthetic embedding construction with simulated user behavior for cold items, then uses the simulated interactions to optimize their embeddings. The simulation can enrich offline training data, while online platforms use it only to update cold-item embeddings.
4.1.1 Behavior Simulation.
- ColdLLM simulates user sequences for cold items and uses behavior-embedding optimization, rather than a mapping function, to obtain their embeddings.The simulator analyzes positive historical behaviors; the resulting interactions let the recommender use trained user and warm-item embeddings.Source: §4.1 · p. 3, §4.1.1 · p. 3, §4.1.2 · p. 3
- The ideal simulator would access a cold item's content, the complete user set and interaction histories, and content details for all items.Source: §4.1.1 · p. 3
4.1.2 Embedding Optimization.
- For online billion-scale platforms, simulated interactions update only cold-item embeddings; offline datasets can also use them to enrich training data.Source: §4.1.2 · p. 3
4.2 Coupled Funnel ColdLLM
ColdLLM couples user filtering with LLM-based simulation to manage the computational cost of examining users for cold items. It filters candidates, refines them with item and user context, and passes simulated interactions to embedding optimization.
4.2 Coupled Funnel ColdLLM
- ColdLLM couples filtering and refining to address the computational cost of applying an LLM to cold-item recommendations at scale.The original LLM simulator's computational complexity requires more expensive GPUs and makes it unsuitable for billion-scale recommendations.Source: §4.2 · pp. 3–4
4.2.1 Filtering Simulation.
- LLM-processed item embeddings support filtering that narrows a billion-scale user pool to tens or hundreds of candidates.The filtering process maps the item's LLM content embedding and uses user-item embedding similarity to select top-scoring candidates.Source: §4.2.1 · p. 4, §4.2.2 · p. 4
- The item embedding is read from the LLM's last-layer token embeddings using mean pooling, then mapped for filtering.Source: §4.2.1 · p. 4
4.2.2 Refining Simulation.
- The refining stage supplies user context and item content to the LLM, which predicts interaction as “yes” or “no.”The prompt combines fixed prompts, user context, and item content; item embeddings help select related historical items for context.Source: §4.2.2 · pp. 4–5
- The simulator produces binary predictions; the authors identify a more intricate framework for continuous values as future work.Source: §4.2.2 · p. 5
- The refining process retains users predicted as “yes” and feeds the simulated sequence into behavior-embedding optimization for the cold item.Source: §4.2.2 · p. 5
4.3 Simulator Training
ColdLLM training adapts the LLM to recommendation scenarios and trains a coupled filter to reflect user-item behavior while aligning with the LLM. Its data preparation samples positive, negative, and unobserved behavior pairs.
4.3.1 LLM Training.
- Low-rank fine-tuning adapts the LLM to each recommendation scenario and supports online updating to capture trends.The fine-tuning structure adds low-rank modifications to transformer attention and feed-forward layers while maintaining most pretrained weights.Source: §4.3.1 · p. 5
- Training data includes positive clicks, negative ignores, and unobserved pairs where the system did not recommend the item.Source: §4.3.1 · p. 5
- For online data, the LLM uses 1:1 positive-negative sampling and additional 1:1 positive-unobserved sampling; offline datasets use positive-unobserved sampling only.The paper reports negative behaviors comprise approximately 70%-90% of online behaviors.Source: §4.3.1 · p. 5
4.3.2 Coupled Filter Model Training.
- The coupled filter combines two embedding pairs to reflect user-item behavior and couple the filter with the LLM.Source: §4.3.2 · p. 5
- Behavior filtering uses BPR loss to rank positive items above sampled negatives and an ALDI aligning loss; coupled ColdLLM filtering adds a coupled ColdLLM loss alongside BPR.Source: §4.3.2 · p. 5
4.4 Implementation Strategy
ColdLLM deployment comprises online serving, online embedding updates, and offline simulation. Simulated interactions update only cold-item embeddings before those embeddings are sent to the recommendation service.
4.4 Implementation Strategy
- Deployment includes online serving, online embedding updating, and offline simulation.Simulated user-item pairs update only cold-item embeddings, which are then sent to the online recommendation service.Source: Figure 3 · p. 6, §4.4 · p. 6
- Coupled filtering reduces billions of users to hundreds in about 60 ms; LLM refining identifies 20 qualified users in less than 8 seconds.The refining step takes about 200-400 ms per user-item pair, and embedding optimization takes within 120 ms.Source: §4.4 · p. 6
- With three 8×A100 GPU machines, ColdLLM handles 8,640 cold items per hour, and capacity can scale by adding GPUs.Source: §4.4 · p. 6
5 Experiments
Experiments compare ColdLLM with cold-start recommendation baselines across datasets and backbones, then examine component contributions, parameter choices, and industrial performance.
5.1.1 Datasets.
- The experiments use CiteULike and MovieLens, designate 20% of items as cold-start items, and split their interactions equally between validation and testing.The remaining 80% of items use an 8:1:1 training, validation, and testing split.Source: §5.1.1 · p. 6
5.1.2 Compared Baselines.
- ColdLLM is evaluated against ten cold-start recommendation models using MF, NGCF, and LightGCN backbones.The baselines span dropout-based embedding simulation, generative embedding simulation, and user behavior simulation.Source: §5.1.2 · p. 6
5.1.3 Hyperparameter Setting.
- The filtering phase uses AdamW at 1×10−5 with batch size 128 and top-k 20; the refining phase uses a learning rate of 5×10−5.Embedding dimension is 200, and the Adam optimizer uses learning rate 1 × 10−3 with early stopping on validation NDCG.Source: §5.1.3 · p. 6
5.1.4 Evaluation Metrics.
- The evaluation uses full-ranking Recall@K and NDCG@K with K=20, and randomly selects 2,000 users for testing.The same evaluation covers overall, warm, and cold recommendations.Source: §5.1.4 · p. 6
5.2 Main Results (RQ1)
- 10.79% average NDCG improvement overall and 37.10% for cold-item recommendations over LightGCN; ColdLLM is superior across datasets and backbones.Table 1 compares overall, cold, and warm recommendations over MF, NGCF, and LightGCN.Source: §5.2 · p. 6, Table 1 · p. 7
5.3 Ablation Study (RQ2)
- Removing the refining stage reduces performance, and removing it alongside either filter causes a larger decline than removing that filter alone.The ablation compares ColdLLM with variants that omit filtering and refining components.Source: §5.3 · pp. 7–8, Figure 4 · p. 8
5.4 Parameter Study (RQ3)
- CiteULike achieves its best overall and warm results at K=10, but its best cold results at K=50; excessively large K can add noise from irrelevant interactions.The learning-rate study finds similar optimal rates across overall, warm, and cold recommendation tasks.Source: §5.4 · p. 8, Figure 5 · p. 8
5.5 Online Evaluation (RQ4)
- Against ALDI, ColdLLM improves Cold-PV by 7.25%, Cold-PCTR by 3.70%, and Cold-GMV by 16.90% in the online evaluation.The two-week A/B test compared ColdLLM with Random, MetaEmb, and ALDI.Source: §5.5 · p. 8, Table 3 · p. 8
6 Conclusion
ColdLLM addresses cold-start recommendation, and the paper reports that online and offline experiments verified its strength. The authors say it significantly improves performance and economic impact.
6 Conclusion
- ColdLLM addresses the cold-start problem in large-scale recommendation systems.The paper reports that both online and offline experiments verified the strength of the approach.Source: §6 · p. 8
- The authors report significant improvement in recommendation performance and economic impact.Source: §6 · p. 8
- The paper presents ColdLLM as opening possibilities for using large language models in large-scale online recommendations.Source: §6 · p. 8
Ethical Considerations
The ethical considerations discuss leveraging LLMs to simulate cold-item behaviors and identify possible benefits and resource burdens. They note that industrial applications may pose computational and storage challenges for academic researchers.
Ethical Considerations
- The paper focuses on using LLMs to simulate cold-item behaviors, a direction it considers relevant to newly occurring items in recommender systems.The authors also note potential ethical considerations.Source: Ethical Considerations · p. 9
- Industrial applications may benefit industrial users and encourage communication between industrial and academic researchers, while posing computational and storage challenges for academic researchers.Source: Ethical Considerations · p. 9
- LLM training and deployment require significant computational power and may increase energy consumption and environmental impact.Source: Ethical Considerations · p. 9