Source-linked AI summary
PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations
Ruining He, Lukasz Heldt, Lichan Hong, Raghunandan Keshavan, Shifan Mao, Nikhil Mehta, Zhengyang Su, Alicia Tsai, Yueqi Wang, Shao-Chuan Wang, Xinyang Yi, Lexi Baugher, Baykal Cakici, Ed Chi, Cristos Goodrow, Ningren Han, He Ma, Romer Rosales, Abby Van Soest, Devansh Tandon, Su-Lin Wu, Weilong Yang, Yilin Zheng
TL;DR
Industrial recommenders rely on large embedding tables, and pre-trained LLMs lack target-domain behavior and item data needed for recommendation. PLUM adapts LLMs with Semantic IDs, continued pre-training, and task-specific fine-tuning; its generative retrieval is reported as more sample-efficient than production Transformer retrieval with large embedding tables.
Problem
Industrial recommenders rely on massive embedding tables, while off-the-shelf LLMs lack target-domain user behavior and item data and have shown a persistent recommendation performance gap.
Source: §1 · pp. 1–2Method
PLUM tokenizes items as Semantic IDs, continues pre-training on domain-specific data and user sequences, then fine-tunes a decoder-only model to generate item IDs for retrieval.
Source: §1 · p. 2Results
PLUM generative retrieval is significantly more sample-efficient than production Transformer retrieval with large embedding tables, with comparable training cost despite 100x dense parameters.
Source: §1 · p. 2Takeaways & Limitations
PLUM is in production for YouTube video retrieval and is described as the platform’s first neural model built without large embedding tables.
Source: §4 · p. 8The scaling study is compute-constrained, and MoE-3B did not outperform MoE-900M for the compute budgets considered.
Source: §3.4.3 · p. 8
Abstract
from arXiv · showhide
Large Language Models (LLMs) pose a new paradigm of modeling and computation for information tasks. Recommendation systems are a critical application domain poised to benefit significantly from the sequence modeling capabilities and world knowledge inherent in these large models. In this paper, we introduce PLUM, a framework designed to adapt pre-trained LLMs for industry-scale recommendation tasks. PLUM consists of item tokenization using Semantic IDs, continued pre-training (CPT) on domain-specific data, and task-specific fine-tuning for recommendation objectives. For fine-tuning, we focus particularly on generative retrieval, where the model is directly trained to generate Semantic IDs of recommended items based on user context. We conduct comprehensive experiments on large-scale internal video recommendation datasets. Our results demonstrate that PLUM achieves substantial improvements for retrieval compared to a heavily-optimized production model built with large embedding tables. We also present a scaling study for the model's retrieval performance, our learnings about CPT, a few enhancements to Semantic IDs, along with an overview of the training and inference methods that enable launching this framework to billions of users in YouTube.
1 Introduction
Industrial recommenders rely on large embedding tables, while adapting LLMs to recommendation is difficult because they lack target-domain user behavior and item data. PLUM combines Semantic IDs, continued pre-training, and task-specific fine-tuning, and reports more sample-efficient retrieval than a production embedding-table model.
1 Introduction
- PLUM adapts pre-trained LLMs through item tokenization with Semantic IDs, continued pre-training on domain and general data, and task-specific fine-tuning for generative retrieval.The decoder-only retrieval model autoregressively generates Semantic IDs for next items a user is likely to engage with.Source: §1 · p. 2
- PLUM generative retrieval is significantly more sample-efficient than production Transformer retrieval with large embedding tables.The retrieval Transformer has 100x the dense parameters of LEM, yet comparable retrieval-task training cost due to faster convergence.Source: §1 · p. 2
- Although PLUM retrieval uses a Transformer with 100x the dense parameters of LEM, comparable overall retrieval-training cost is achieved through faster convergence, and retrieval performance improves up to an MoE with over 900M activated parameters (approximately 4.2B parameters total).Source: §1 · p. 2
- PLUM retrieval serves long-form and short-form video recommendations on multiple YouTube surfaces through online and offline inference.The framework is described as in production for YouTube recommendations.Source: §1 · p. 2
1.1 Related Work
- Prior recommendation advances use Transformers and LLM-like pre-training, but still rely on large categorical embedding tables; PLUM instead uses compact input tokens.PLUM also processes heterogeneous features by tokenizing numerical features or using dense embeddings as soft tokens.Source: §1.1.1 · p. 2
1.1.1 Sequential Recommendations.
- Earlier Semantic ID work showed retrieval models can predict SIDs and that SIDs can replace random item-ID embeddings in ranking models.These approaches are cited as [23], [24] [39].Source: §1.1.2 · p. 2
1.1.2 Semantic IDs and Quantization.
- PLUM’s SID-v2 combines content embeddings and behavioral signals, using contrastive training to capture useful information without dynamic CF embeddings.Unlike MMQ [30], PLUM concatenates multiple content embeddings before MLP encoding; dynamic CF embeddings can require frequent quantizer and downstream-model retraining.Source: §1.1.2 · pp. 2–3
- Most generative retrieval work trains models from scratch; PLUM investigates LLM pre-training and continued pre-training for SID-based retrieval.The cited related-work discussion notes limited study of SIDs as a new modality for aligning LLMs with recommendation.Source: §1.1.3 · p. 3
2 PLUM Framework
PLUM adapts pre-trained language models for recommendation through Semantic IDs, continued pre-training, and generative-retrieval fine-tuning. Its framework builds hierarchical item codes from multimodal content and co-occurrence signals, then trains the model to generate clicked-video codes from user context.
2.1 Semantic IDs
- Semantic IDs represent each item as hierarchical codewords quantized from a dense embedding of its content features.The paper builds on RQ-VAE and aims for codes that are comprehensive, behavior-aligned, and hierarchically coherent.Source: §2.1 · p. 3
2.1.1 Fused Multi-Modal Content Representation.
- Separate encoders map heterogeneous content embeddings into latent vectors, which are concatenated and projected into one unified representation for quantization.This design accepts varying numbers and types of input sources.Source: §2.1.1 · p. 3
2.1.2 Hierarchical Refinements in Quantization.
- Multi-resolution codebooks use discriminative high-resolution early levels and lower-resolution later levels, while progressive masking selects a random prefix of codebook levels during training.The level-dependent codebook cardinality is 2048/2^(level−1).Source: §2.1.2 · p. 3
2.1.3 RQ-VAE with Co-occurrence Contrastive Regularization.
- A co-occurrence contrastive loss shapes the embedding space to bring videos watched together closer and push apart videos that do not co-occur.The loss compares each video representation with its co-occurring partner and other videos using dot-product similarity.Source: §2.1.3 · p. 3, Figure 1 · p. 4
2.1.4 SID Training Loss.
- SID training combines reconstruction, residual-quantization, and co-occurrence-contrastive losses, with the quantized vector formed from progressively selected codes and decoded to reconstruct input embeddings.Residuals are computed recursively across quantization levels.Source: §2.1.4 · p. 4
2.2 Continued Pre-training
- Continued pre-training predicts next tokens on a mixture of user watch histories and video metadata, with each source contributing half of the training examples.Training runs for 1 million steps and approximately 260 billion tokens; CPT models retain free-form text generation and in-context few-shot learning.Source: §2.2 · p. 4
2.3 Generative Retrieval
- Supervised fine-tuning predicts clicked-video SIDs from user context using SID, text, and numerical-feature tokens; beam search generates candidate SID sequences that map to corpus videos.Reward-based sampling selects training examples, and real-time context is among the features incorporated during SFT.Source: §2.3 · pp. 4–5, Figure 2 · p. 5
3 Experiments
The experiments compare PLUM generative retrieval with production LEM retrieval and assess recommendation quality, live engagement, and training efficiency. PLUM has larger effective vocabulary with competitive user-reaction metrics, adds value in live tests, and uses fewer training FLOPs than LEM.
3.1 Performance of Generative Retrieval
- The experiments evaluate generative retrieval in a YouTube production setup against traditional large-embedding retrieval.Recommendation quality is assessed using effective vocabulary size and user-reaction metrics.Source: §3.1 · p. 5, §3.1.2 · p. 5
3.1.1 Experiment Setup.
- The 900M-activated PLUM model uses beam search, which outperformed random decoding at some cost to diversity.Its baseline is a highly optimized Transformer LEM whose neural network accounts for 0.4% of total parameters.Source: §3.1.1 · p. 5
3.1.2 Experiment Results.
- PLUM achieves much larger effective vocabulary size while remaining competitive with LEM on user-reaction metrics.The reported reaction metrics include CTR, WT/View, and WF/View.Source: §3.1.2 · p. 5
- Live experiments report that PLUM adds unique value over quota-matched LEM+ for both Long Form Video and Shorts.The comparison reports metric movements on four key metrics.Source: §3.1.2 · p. 6, Table 3 · p. 6
- < 0.55x FLOPs are used by the 900M MoE PLUM model to train compared with LEM.PLUM trains on roughly 250M examples daily, versus several billion for LEM.Source: §3.1.2 · p. 6
3.2 Semantic IDs Ablation Study
The SIDv2 ablations assess index uniqueness and downstream video recall. All proposed changes improve generative item-retrieval recall, with co-occurrence alignment notably improving both measures.
3.2 Semantic IDs Ablation Study
- The ablations report index uniqueness and Video Recall@10; recall measures retrieving a specific video from PLUM’s top K SID predictions.When an SID maps to multiple videos, recall evaluation randomly samples one video.Source: §3.2 · p. 6
- All SIDv2 changes improve generative item-retrieval recall, and co-occurrence alignment notably improves both SID uniqueness and recall.The ablations use a 900M MoE retrieval model without CPT or watch history in the prompt.Source: §3.2 · p. 6
3.3 Impact of Continued Pre-training
A controlled four-configuration study tests continued pre-training and pre-trained LLM initialization for generative retrieval. CPT improves retrieval performance and convergence speed, while LLM-initialized models outperform random-initialized counterparts.
3.3.1 Experiment Setup.
- The controlled 2x2 ablation uses four configurations combining retrieval SFT or CPT plus retrieval SFT with random or pre-trained LLM initialization.All configurations share a decoder-only Transformer architecture and use the MoE-900M model.Source: §3.3.1 · p. 6
3.3 Impact of Continued Pre-training
- CPT can benefit task-specific fine-tuning efficiency, especially when multiple downstream tasks share the same CPT model.The training methodology consists of continued pre-training followed by generative-retrieval fine-tuning.Source: §3.3.2 · p. 6, §3.3.1 · p. 6
3.3.2 Experiment Results.
- CPT models show a large retrieval-performance gap over their non-CPT counterparts and converge faster during retrieval fine-tuning.The study evaluates recall@10 on a large held-out next-day dataset; Figure 3 tracks eighth-day recall@10 and training loss against steps.Source: Table 5 · p. 6, §3.3.2 · p. 6, Figure 3 · p. 7
- Pre-trained LLM initialization consistently outperforms random initialization, both with and without CPT.The authors hypothesize that natural-language understanding or general sequence-processing ability may account for the advantage; further study is out of scope.Source: §3.3.2 · p. 7
3.4 Scaling Study
The scaling study tests how model size and compute relate to loss and retrieval on YouTube production data. Evaluation Recall@10 continues improving with fewer signs of saturation than evaluation loss, while the optimal model size shifts upward as compute budgets grow.
3.4 Scaling Study
- The study compares four Gemini-1.5 MoE sizes, from MoE-110M to MoE-3B, to examine loss and retrieval scaling and identify model sizes for compute budgets.The sizes indicate activated parameters per token; total parameters span less than 1B to more than 10B.Source: §3.4 · p. 7
3.4.1 Experiment Setup.
- The production candidate-generation study predicts next-watch Semantic IDs from watch history, user features, and context video features, then evaluates retrieval against billions of videos.Training uses seven shuffled days and evaluation uses the subsequent day; inputs are fixed at 1,536 tokens, covering roughly 100 recent watches and other features.Source: §3.4.1 · p. 7
- Models are warm-started from Continued Pre-training checkpoints, and constant learning rates allow direct comparison at intermediate Iso-FLOPS budgets from single runs.The study excludes Continued Pre-training FLOPs; batch sizes are set to approximately saturate HBM, with learning rates selected relative to MoE-900M's 5×10−5 anchor.Source: §3.4.1 · p. 7
3.4.2 Experiment Results.
- Evaluation-loss frontier shifts toward larger models sooner than the training-loss frontier, while evaluation loss eventually begins to saturate.The training-loss frontier shifts from MoE-110M to MoE-370M to MoE-900M, with MoE-3B hypothesized as a later frontier model.Source: §3.4.2 · pp. 7–8, Figure 4 · p. 8
- Evaluation Recall@10 shows fewer signs of slowing as Iso-FLOPS increase, and MoE-110M reaches 4.24 epochs without signs of overfitting.Figure 5 reports training and evaluation Recall@10 across Iso-FLOPS budgets.Source: §3.4.2 · p. 8, Figure 5 · p. 8
- The optimal model size gradually shifts larger as Iso-FLOPS budgets increase, with a peak for most budgets considered.Figure 6 evaluates training loss and evaluation Recall@10 at fixed Iso-FLOPS budgets, selecting steps for each model size.Source: §3.4.2 · p. 8, Figure 6 · p. 8
- MoE-3B did not outperform MoE-900M within the tested budgets; compute constraints and a suboptimal hyperparameter setup may have contributed.MoE-3B processed 0.57 epochs, about 5B examples, more than a 2× gap from the second-largest model; the authors conclude compute-optimal training requires scaling examples and model size together.Source: §3.4.3 · p. 8
4 Conclusion and Future Work
The conclusion presents PLUM as an LLM adaptation framework for large-scale recommendation, combining SIDv2, continued pre-training, and supervised fine-tuning for generative retrieval. It reports YouTube deployment and ablation support, while identifying further research directions.
4 Conclusion and Future Work
- PLUM adapts pre-trained LLMs for large-scale recommendation through SIDv2 item tokenization, continued pre-training, and supervised fine-tuning for generative retrieval.Continued pre-training aligns the LLM with user behavior and grounds Semantic IDs in text.Source: §4 · p. 8
- Ablations validate CPT, pre-trained LLM initialization, and the proposed SID techniques; future directions include ranking, personalized search, candidate diversity, and joint SID and natural-language generation.PLUM is described as launched in YouTube and as the paper's first neural model built without large embedding tables.Source: §4 · p. 8
A.1 Scaling Pilot Study
The pilot study tests MoE-900M sensitivity to global batch size and learning rate, with its findings informing hyperparameter choices for the later scaling analysis.
A.1 Scaling Pilot Study
- The pilot examines MoE-900M sensitivity to global batch size and learning rates to inform hyperparameter selection for the subsequent scaling analysis.The configuration table is titled “Pilot Study Configurations with MoE-900M.”Source: §A.1 · p. 10, Table 7 · p. 10
- Four trainers ran in parallel for approximately two weeks, each using 256 TPUs and the same pre-trained checkpoint.The duration was considered sufficient to establish clear performance trends.Source: §A.1.1 · p. 10
- The base setup used a constant learning rate of 1×10−4 and a batch size saturating available HBM; other configurations varied one hyperparameter at a time.Source: §A.1.1 · p. 10
A.1.2 Results.
The pilot found tolerance to a lower learning rate within its duration, poorer performance at a doubled rate, and reduced training efficiency with a smaller batch.
A.1.2 Results.
- 1 × 10−4 and 5 × 10−5 learning rates showed no statistically significant performance difference within the pilot duration.Source: §A.1.2 · p. 10
- Trainer C resulted in substantially worse performance, indicating that the tested rate is likely outside the optimal range for this model and dataset.Source: §A.1.2 · p. 10
- Halving batch size requires more than double the training steps to reach comparable performance, reducing training efficiency.The passage highlights maximizing batch size within hardware-memory limits for compute-optimal training.Source: §A.1.2 · p. 10
A.2 In-context Learning Examples
After CPT with Semantic IDs, the pre-trained-LLM model completed few-shot prompts with semantically appropriate phrases, while a randomly initialized model struggled with coherent phrasing and distinguishing SID from text tokens.
A.2 In-context Learning Examples
- The CPT model based on a pre-trained LLM completed few-shot prompts with semantically appropriate phrases by following the task description.Table 8 provides qualitative examples of the in-context learning capability.Source: §A.2 · p. 10
- With the same recommendation pre-training, the randomly initialized model struggled to form coherent phrases and distinguish SID from text tokens.Source: §A.2 · p. 10
- Table 8 pairs text prompts and outputs from two initializations, with thumbnails showing the visual content associated with each prompt Semantic ID.Source: Table 8 · p. 11