Source-linked AI summary
Recommender Systems with Generative Retrieval
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H. Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, Maheswaran Sathiamoorthy
TL;DR
Existing recommender retrieval matches query and item embeddings, while generative retrieval for recommendation with semantic identifiers remains underexplored. TIGER assigns each item a content-derived Semantic ID and trains a Transformer to autoregressively predict the next item’s ID. It outperforms state-of-the-art recommenders across benchmarks and supports recommendations for new or infrequent items, although invalid generated IDs remain a design issue.
Problem
Conventional recommender retrieval matches query and candidate embeddings, motivating a generative retrieval paradigm that predicts candidate identifiers directly.
Method
TIGER quantizes content embeddings into ordered Semantic ID codewords and trains a Transformer sequence model to predict the next item’s Semantic ID.
Results
TIGER consistently outperforms existing state-of-the-art recommenders across three benchmarks on recall and NDCG metrics.
Takeaways & Limitations
Semantic IDs support cold-start recommendations for new and infrequent items and enable diverse recommendations through a tunable parameter.
Takeaways & Limitations
Autoregressive decoding can produce invalid Semantic IDs that do not map to recommendation-corpus items, although the observed probability is low.
Abstract
from arXiv · showhide
Modern recommender systems perform large-scale retrieval by first embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search to select top candidates given a query embedding. In this paper, we propose a novel generative retrieval approach, where the retrieval model autoregressively decodes the identifiers of the target candidates. To that end, we create semantically meaningful tuple of codewords to serve as a Semantic ID for each item. Given Semantic IDs for items in a user session, a Transformer-based sequence-to-sequence model is trained to predict the Semantic ID of the next item that the user will interact with. To the best of our knowledge, this is the first Semantic ID-based generative model for recommendation tasks. We show that recommender systems trained with the proposed paradigm significantly outperform the current SOTA models on various datasets. In addition, we show that incorporating Semantic IDs into the sequence-to-sequence model enhances its ability to generalize, as evidenced by the improved retrieval performance observed for items with no prior interaction history.
1 Introduction
TIGER reframes sequential recommendation as generative retrieval: a Transformer predicts semantically meaningful item identifiers rather than matching query and candidate embeddings. Semantic IDs support knowledge sharing, large item spaces, cold-start generalization, and diverse recommendations.
- Proposed framework: TIGER predicts candidate Semantic IDs directly with an end-to-end generative model, using Transformer memory as a retrieval index.Items are represented by ordered codewords derived from content embeddings and used to train the sequential recommendation model.
- Semantic representation: Semantic token representations enable knowledge sharing across similar items and reduce reliance on atomic, random item IDs.The paper links semantic representations to reduced exposure to recommendation feedback loops and improved generalization to newly added items.
- Contributions: TIGER outperforms existing state-of-the-art recommenders across multiple datasets on recall and NDCG metrics.The reported capabilities include recommending new and infrequent items and generating diverse recommendations through a tunable parameter.
2 Related Work
Prior work spans sequential recommendation, generative retrieval, and content-based Semantic IDs. TIGER combines these directions by using semantic item codes for generative retrieval in recommendation, which the paper identifies as a first.
- Sequential Recommenders: Sequential recommendation research includes GRU, attention, self-attention, masking, and pretrained-language-model approaches.Examples include GRU4REC, NARM, AttRec, SASRec, BERT4Rec, Transformers4Rec, and P5.
- Sequential Recommenders: P5 uses a language-model tokenizer on randomly assigned item IDs, whereas TIGER learns Semantic IDs from item content information.The paper reports that Semantic ID representations yield better recommendation results than random codes in its experiments.
- Semantic IDs: VQ-Rec generates content-based codes for transferable recommendation but does not use them for generative retrieval.TIGER instead uses RQ-VAE to produce hierarchical Semantic IDs.
- Generative Retrieval: Generative retrieval returns relevant documents by learning search indices, with prior systems including GENRE, DSI, NCI, and CGR.The paper positions TIGER as the first recommendation system to use generative retrieval with Semantic ID representations.
3 Proposed Framework
TIGER builds generative retrieval around Semantic IDs: content embeddings are quantized into ordered codewords, and a Transformer predicts the next item's Semantic ID from user interaction sequences. The framework uses RQ-VAE for semantic code generation, addresses collisions, and considers invalid generated identifiers.
- Semantic ID generation encodes item content, quantizes its embedding into an ordered tuple of codewords, and uses that tuple as the item identifier.The framework assumes useful item content features and access to a pretrained content encoder.
- 3.1 Semantic ID Generation: A Semantic ID contains one codeword from each of m codebooks, with similar items expected to share prefixes or other overlapping codewords.The number of uniquely representable items equals the product of the codebook sizes.
- 3.1 Semantic ID Generation: RQ-VAE generates Semantic IDs by repeatedly quantizing residuals, while jointly training the encoder, decoder, and codebooks with reconstruction and residual-quantization losses.At each level, the nearest codebook embedding is selected and subtracted to form the next residual.
- 3.2 Generative Retrieval with Semantic IDs: Semantic ID collisions are resolved by appending extra tokens, while generated identifiers that match no corpus item are possible but reported to occur with low probability.The collision procedure makes otherwise identical Semantic IDs unique.
- 3.2 Generative Retrieval with Semantic IDs: The recommender sorts each user’s interacted items chronologically, converts their Semantic IDs into token sequences, and trains a Transformer to predict the next item’s Semantic ID.This is the second stage of the proposed framework.
4 Experiments
Experiments evaluate TIGER on Amazon sequential-recommendation benchmarks, comparing its Semantic ID design, retrieval performance, and additional capabilities. TIGER outperforms baselines and supports cold-start retrieval and controllable recommendation diversity.
- Experimental Setup: TIGER is evaluated on Beauty, Sports and Outdoors, and Toys and Games using Recall@K and NDCG@K at K = 5, 10.The benchmarks come from the Amazon Product Reviews dataset, using item metadata and user reviews.
- Sequential Recommendation: 29% improvement in NDCG@5 over SASRec and 17.3% improvement in Recall@5 over S3-Rec are reported on Beauty.On Toys and Games, TIGER is reported as 21% better in NDCG@5 and 15% better in NDCG@10.
- Item Representation: RQ-VAE Semantic IDs organize items hierarchically: c1 captures coarse categories, while c2 and c3 refine them into fine-grained categories.The qualitative analysis uses Amazon Beauty items and visualizes category distributions across semantic tokens.
- Item Representation: RQ-VAE Semantic IDs outperform LSH and Random IDs, indicating that learned content-based quantization performs better than random projections and random codes.The comparisons use the same content-based semantic embedding for LSH and RQ-VAE, while Random IDs match RQ-VAE cardinality approximately.
- New Capabilities: For cold-start retrieval, TIGER with ϵ = 0.1 consistently outperforms Semantic_KNN across Recall@K metrics, and it outperforms the baseline for all ϵ ≥0.1.Unseen items sharing the first three semantic tokens with a predicted ID are included among retrieved candidates.
- New Capabilities: Temperature sampling during decoding can increase diversity in predicted ground-truth item categories, measured with Entropy@K.Table 3 reports entropy for different temperature values on the Beauty dataset.
5 Conclusion
TIGER uses semantic item identifiers and generative retrieval to avoid explicit item indexing while achieving strong retrieval performance and generalizing to unseen items.
- TIGER uses a hierarchical RQ-VAE quantizer on content embeddings to generate Semantic IDs for items.These tokens form a semantic representation used by the generative retrieval model.
- The Transformer memory acts as a semantic index, enabling training and serving without creating an explicit item index.The embedding table’s cardinality also does not grow linearly with the item space.
- Experiments on three datasets show state-of-the-art retrieval performance while generalizing to new and unseen items.
A Related Work (cont.)
Related retrieval methods use shared query-document embedding spaces and approximate search, whereas generative retrieval autoregressively produces document identifiers; vector quantization supplies compact codeword representations.
- Generative Retrieval: Traditional document retrieval maps queries and documents into a shared vector space, then uses ANN or MIPS to retrieve nearest documents.
- Generative Retrieval: Generative retrieval produces document titles, names, or identifiers token by token instead of searching an embedding index.
- Generative Retrieval: DSI assigns structured semantic DocIDs and autoregressively decodes them, marking a shift toward end-to-end Transformer retrieval.
- Vector Quantization: Vector quantization converts a high-dimensional vector into a low-dimensional tuple of codewords, with hierarchical clustering as one approach.
- Vector Quantization: RQ-VAE applies residual quantization to VQ-VAE encoder outputs to achieve lower reconstruction error.
B Baselines
The baselines comprise recurrent, convolutional, gated, self-attentive, self-supervised, feature-aware, and pretrained-language-model approaches for sequential recommendation.
- GRU4Rec uses a customized GRU, while Caser uses horizontal and vertical convolutions to model sequential recommendation patterns.
- HGN captures long-term and short-term user interests through a hierarchical gating architecture.
- SASRec uses a causally masked Transformer for sequential recommendation.
- FDSA incorporates item features alongside item embeddings in Transformer input sequences.
- S3-Rec pre-trains a bidirectional Transformer on self-supervised tasks, whereas P5 uses a pretrained LLM to unify recommendation tasks.
C Dataset Statistics
The evaluation uses three public Amazon Product Reviews benchmarks—Beauty, Sports and Outdoors, and Toys and Games—with review histories converted into timestamp-ordered item sequences.
- Table 6 summarizes the dataset statistics for the three real-world benchmarks.
- The benchmarks come from Amazon Product Reviews data containing user reviews and item metadata collected from May 1996 to July 2014.
- The sequential recommendation task uses the Beauty, Sports and Outdoors, and Toys and Games categories.
- User review histories are converted into timestamp-sorted item sequences, excluding users with fewer than five reviews.
D Modifications to the P5 data preprocessing
The P5 Amazon preprocessing creates chronologically ordered user sessions, then remaps items to sequential integer IDs before train/test splitting, which can produce information leakage. The authors instead assign random integer IDs before splitting; the supplied materials also report invalid Semantic IDs for beam-search retrieval.
- D Modifications to the P5 data preprocessing: P5 preprocessing remaps items to sequential integers before creating training and testing splits.This can create sequences such as a, a + 1, a + 2, . . . .
- D Modifications to the P5 data preprocessing: Sequential item IDs can make train and test items share SentencePiece subwords, leading to information leakage during inference.The leakage arises because the sequential remapping occurs before the dataset split.
- D Modifications to the P5 data preprocessing: The authors resolve the leakage issue by assigning random integer IDs before creating training and evaluation splits.The remaining P5 code was kept identical to the source implementation, according to the supplied passage.
- D Modifications to the P5 data preprocessing: Figure 6 reports approximately 0.3%–6% invalid IDs when retrieving the top-20 items with beam search.The percentage varies with the beam-search value of K.
E Discussion
The discussion examines Semantic ID design, scalability, inference cost, lookup-table memory, and embedding-table memory. Semantic IDs make embedding tables much smaller, but longer IDs increase computation and autoregressive beam search can make inference more expensive than ANN-based models.
- Effects of Semantic ID length and codebook size: Recommendation metrics remain robust when varying Semantic ID length and codebook size.Longer Semantic IDs increase input sequence length and make the Transformer sequence-to-sequence computation more expensive.
- Scalability: Generating Semantic IDs from the combined three-dataset item set causes only a small performance decrease on Beauty.The comparison is between combined-dataset generation and generation using only Beauty items.
- Inference cost: TIGER inference can be more computationally expensive than ANN-based models because autoregressive decoding uses beam search.Improving inference efficiency was not the main objective of this work.
- Memory cost of lookup tables: Each lookup table is O(64N) bits, with two tables mapping Item IDs to Semantic IDs and Semantic IDs to Item IDs.A Semantic ID contains four 8-bit integers, while each Item ID is a 32-bit integer.