Source-linked AI summary
Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations
Anima Singh, Trung Vu, Nikhil Mehta, Raghunandan Keshavan, Maheswaran Sathiamoorthy, Yilin Zheng, Lichan Hong, Lukasz Heldt, Li Wei, Devansh Tandon, Ed H. Chi, Xinyang Yi
TL;DR
Randomly hashed item IDs provide memorization but offer no semantic basis for generalizing across similar items, while direct content embeddings can reduce ranking quality. The paper adapts compact Semantic IDs derived from frozen content embeddings for ranking models. The reported results show Semantic IDs can replace video IDs to improve generalization through meaningful collisions.
Problem
Randomly hashed item IDs lack semantic structure for generalizing across similar items, and direct content embeddings can reduce quality when replacing video IDs.
Source: §1 · pp. 1–2Method
The paper derives hierarchical Semantic IDs from frozen content embeddings with RQ-VAE and adapts them for ranking through n-gram or SentencePiece sub-piece embeddings.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Randomly-hashed item ids are used ubiquitously in recommendation models. However, the learned representations from random hashing prevents generalization across similar items, causing problems of learning unseen and long-tail items, especially when item corpus is large, power-law distributed, and evolving dynamically. In this paper, we propose using content-derived features as a replacement for random ids. We show that simply replacing ID features with content-based embeddings can cause a drop in quality due to reduced memorization capability. To strike a good balance of memorization and generalization, we propose to use Semantic IDs -- a compact discrete item representation learned from frozen content embeddings using RQ-VAE that captures the hierarchy of concepts in items -- as a replacement for random item ids. Similar to content embeddings, the compactness of Semantic IDs poses a problem of easy adaption in recommendation models. We propose novel methods for adapting Semantic IDs in industry-scale ranking models, through hashing sub-pieces of of the Semantic-ID sequences. In particular, we find that the SentencePiece model that is commonly used in LLM tokenization outperforms manually crafted pieces such as N-grams. To the end, we evaluate our approaches in a real-world ranking model for YouTube recommendations. Our experiments demonstrate that Semantic IDs can replace the direct use of video IDs by improving the generalization ability on new and long-tail item slices without sacrificing overall model quality.
1 INTRODUCTION
Random item IDs support memorization but can leave sparse and semantically unrelated items poorly represented. The paper proposes adapting content-derived Semantic IDs for ranking models to balance generalization and memorization.
1 INTRODUCTION
- One-hot item embeddings are resource-intensive at billion-item scale and vulnerable to torso- and tail-item sparsity; hashing reduces embedding needs but can create random collisions.When item IDs lack semantic meaning, collisions do not group related items.Source: §1 · p. 1
- Directly replacing video IDs with content embeddings reduced quality in YouTube experiments, while end-to-end video encoders have been reported to cost 10–50× more than the ID baseline.The paper notes prior frozen-embedding results varied across model types and datasets, with corpus sizes up to 150k.Source: §1 · p. 2
- The proposed framework uses compact, hierarchical Semantic IDs and adapts them through sub-piece hashing for resource-constrained, latency-sensitive ranking models.The authors distinguish this ranking use from TIGER’s Semantic IDs for generative retrieval.Source: §1 · p. 2
- Semantic IDs replace random IDs with content-based hashing over sub-pieces, and SentencePiece (SPM) outperforms n-grams at the same number of embedding lookups per item.The paper also reports that SID-based adaptation outperforms directly using content embeddings and demonstrates replacement of video IDs in production.Source: §1 · p. 2
2 RELATED WORK
Related work spans content-based representations, item-ID embedding methods, and discrete representations. The paper emphasizes the production constraints and hierarchical semantics that distinguish its approach.
2 RELATED WORK
- Random hashing and multiple hashing can produce random collisions; Semantic IDs instead enable collisions between semantically related items, while Deep Hash Embedding increases hidden-layer computation.The paper describes Semantic IDs as a compute-efficient way to avoid random collisions during embedding learning.Source: §2 · p. 2
- Content-based recommenders address cold-start or transferable recommendations, but end-to-end video encoders and other approaches can raise training costs that challenge production deployment.The paper focuses on real-time ranking, unlike PinnerFormer’s offline inference setting.Source: §2 · pp. 2–3
- Unlike conventional product quantization approaches cited here, Semantic IDs leverage hierarchical semantics.The related-work discussion also situates the method among VQ-VAE, VQ-GAN, and TIGER’s use of RQ-VAE.Source: §2 · p. 3
3 PROPOSED APPROACHES
The approach compresses item content embeddings into Semantic IDs, then learns ranking-model embeddings for those discrete codes. It adapts the codes with fixed-length n-grams or distribution-learned variable-length SentencePiece subwords.
3.1 OVERVIEW
- The two-stage approach freezes an RQ-VAE-compressed representation, then maps items to Semantic IDs and trains their embeddings with the downstream ranking model.Compression lets user histories represent each item with a few integers rather than high-dimensional embeddings.Source: §3.1 · p. 3
3.2 RQ-VAE FOR SEMANTIC IDS (SIDS)
- RQ-VAE encodes a content embedding into a latent vector, recursively quantizes residuals using nearest codebook vectors, and decodes the quantized latent back to embedding space.The caption’s example Semantic ID is (1, 4, 6, 2).Source: §3.2 · p. 3, Figure 1 · p. 4
- The RQ-VAE loss combines embedding reconstruction with residual-quantization terms that encourage residuals and their selected codebook vectors to move toward each other.The reconstruction term aims to reproduce the content embedding.Source: §3.2 · p. 4
3.3 SEMANTIC ID REPRESENTATION IN RANKING
- N-gram adaptation groups consecutive Semantic-ID codes into subwords, sums their learned embeddings for an item, and uses separate embedding tables for each subgroup.Each N-gram subgroup’s table has K^N rows.Source: §3.3 · p. 4
- SPM learns variable-length subwords from item distributions, grouping frequent co-occurring codes while retaining unigrams for rarer combinations in one fixed-size embedding table.This adapts subword lengths instead of relying on fixed N-gram groupings, whose table sizes grow exponentially with N.Source: §3.3 · p. 4
4 EXPERIMENTS
The experiments compare Semantic ID representations with random hashing and dense content embeddings in a production YouTube ranking model. Bigram-SID and SPM-SID outperform random hashing in large-scale experiments, while SPM’s advantage over N-grams depends on table size.
4.1 EXPERIMENTAL SETUP
- The production ranking model predicts the next video from the current video and past user activity, and evaluation uses data unseen during training.The model uses O(10) million random-hashing buckets for O(100) millions of videos; embedding-table size and fast lookups are design constraints.Source: §4.1 · p. 5
- The comparison includes random hashing, raw content embeddings, and Semantic ID representations, with deeper Dense Input models added as baselines.Semantic IDs use eight codes per video, generated with an RQ-VAE codebook of size 2048.Source: §4.1 · p. 5
- CTR AUC measures performance on the next day’s data, while CTR/1D AUC evaluates items introduced that day as a cold-start slice.The paper considers a 0.1% change in CTR AUC significant for this ranking model.Source: §4.1 · p. 5
4.2 PERFORMANCE OF SEMANTIC ID
- Directly replacing random-hashed IDs with dense content embeddings reduced quality, but deeper Dense Input models improved overall and cold-start quality at increased serving cost.The reported quality improvement grew from 1.5x to 2x model depth.Source: §4.2 · p. 6
- Bigram-SID and SPM-SID significantly outperformed random hashing in the large-scale ranking experiments, while SPM-SID outperformed N-grams with larger embedding tables.With smaller tables, N-gram methods had a slight advantage over SPM-SID; when user history was omitted, Unigram-SID and Bigram-SID had worse overall CTR than random hashing.Source: §4.2 · p. 6, Figure 2 · p. 6, Figure 3 · p. 6
- SPM-SID adapts subwords to the embedding-table size and reduces lookups for common videos, while keeping average lookups comparable to fixed N-gram representations.The paper describes this adaptive lookup pattern as more efficient and scalable for large-scale ranking models.Source: §4.2 · p. 7
5 CONCLUSION AND FUTURE WORK
The conclusion presents Semantic IDs as a way to replace video IDs in recommendation ranking. It attributes improved generalization to meaningful collisions.
5 CONCLUSION AND FUTURE WORK
- The paper develops Semantic IDs from frozen content embeddings using RQ-VAE to capture hierarchical semantic structure across billions of YouTube videos.The conclusion frames this as a way to reduce reliance on video ID features in recommendation models.Source: §5 · p. 7
- Semantic IDs are presented as an effective replacement for video IDs in the ranking model.The conclusion focuses on the YouTube ranking model as its case study.Source: §5 · pp. 7–8
- The authors report improved generalization from replacing video IDs through meaningful collisions.Source: §5 · p. 8
A.1 RQ-VAE TRAINING AND SERVING SETUP
The appendix describes RQ-VAE training and inference for generating Semantic IDs. The trained model is frozen, and its outputs are stored and served as ranking features.
A.1 RQ-VAE TRAINING AND SERVING SETUP
- RQ-VAE uses a one-layer encoder-decoder with dimension 256, eight quantization levels, and codebook size 2048 at each level.Source: §A.1 · p. 11
- Training resets unused codebook vectors to randomly sampled batch content embeddings to improve codebook utilization, then freezes the model to encode videos.The model is trained on a random sample of impressed videos until reconstruction loss stabilizes.Source: §A.1 · p. 11
- New videos receive Semantic IDs from the frozen RQ-VAE model, and those IDs are stored and served like other ranking features.Source: §A.1 · p. 11
A.2 STABILITY OF SEMANTIC IDS OVER TIME
The stability analysis compares Semantic IDs from RQ-VAE models trained on data six months apart. Ranking performance using the two versions is reported as comparable.
A.2 STABILITY OF SEMANTIC IDS OVER TIME
- The study compares Semantic IDs from RQ-VAEv0 and RQ-VAEv1 trained on data six months apart.Source: §A.2 · p. 11
- The production ranking model’s performance with SID-3Bigram-sum was comparable for Semantic IDs from the two RQ-VAE versions.The ranking model was trained on recent engagement data.Source: §A.2 · p. 11, Figure 5 · p. 11
- The authors interpret the comparable results as evidence that the RQ-VAE-learned semantic token space remains stable for downstream ranking over time.Source: §A.2 · p. 11
A.3 SEMANTIC IDS AS HIERARCHY OF CONCEPTS
Semantic IDs form a trie over videos, with prefix levels representing concepts from coarse to fine; example subtries illustrate this hierarchy in sports and food vlogging.
A.3 SEMANTIC IDS AS HIERARCHY OF CONCEPTS
- Higher trie levels represent coarser concepts, while lower levels represent finer-grained concepts.The paper describes Semantic IDs as forming a trie over videos.Source: §A.3 · p. 11
- The example sub-tries come from a trained RQ-VAE model using four-token Semantic IDs.Source: §A.3 · p. 11
- Figures 6 and 7 illustrate concept hierarchies within sports and food-vlogging videos.Source: §A.3 · p. 11
A.4 SIMILARITY ANALYSIS WITH SEMANTIC ID
As Semantic-ID prefixes lengthen, videos sharing them have greater average pairwise cosine similarity and smaller sub-tries, consistent with increasingly granular concepts.
A.4 SIMILARITY ANALYSIS WITH SEMANTIC ID
- The analysis compares average pairwise cosine similarity in content-embedding space and corresponding sub-trie sizes for videos sharing prefixes of length n.For example, IDs (1, 2, 3, 4) and (1, 2, 6, 7) share a prefix of length 2.Source: §A.4 · p. 11, Table 1 · p. 12
- Average pairwise cosine similarity increases while sub-trie size decreases as shared Semantic-ID prefix length increases.Table 1 reports aggregate metrics for videos sharing a prefix length; typical sub-trie size is the rounded 25th–75th percentile range.Source: §A.4 · p. 12, Table 1 · p. 12
- Figures 6 and 7 show example Semantic-ID sub-tries for sports and food-vlogging videos.Source: Figure 6 · p. 12, Figure 7 · p. 12