Source-linked AI summary
Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders
Yupeng Hou, Zhankui He, Julian McAuley, Wayne Xin Zhao
TL;DR
Existing text-based transferable recommenders can bind item representations too tightly to text, while cross-domain transfer is difficult when domains share few users or items. VQ-Rec inserts discrete codes between text and representations, enhances contrastive pre-training, and uses permutation-based fine-tuning; experiments on six benchmarks demonstrate strong transferability, including new-item recommendation without retraining.
Problem
Existing transfer methods can overemphasize item-text features, while limited overlap across domains and platforms makes sequential recommender reuse difficult.
Method
VQ-Rec maps item text to discrete codes, looks up code embeddings for item representations, uses hard negatives in contrastive pre-training, and aligns embeddings with permutation-based fine-tuning.
Results
Experiments on six cross-domain and cross-platform benchmarks demonstrate strong transferability, including recommending new items without retraining and improving known-item performance.
Takeaways & Limitations
Discrete code-based representations capture transferable sequential patterns while reducing direct dependence on text similarity and supporting adaptation across domains or platforms.
Takeaways & Limitations
The approach assumes multi-domain interaction data for training or pre-training and keeps a user’s interaction sequences separate by domain because of semantic gaps.
Abstract
from arXiv · showhide
Recently, the generality of natural language text has been leveraged to develop transferable recommender systems. The basic idea is to employ pre-trained language models~(PLM) to encode item text into item representations. Despite the promising transferability, the binding between item text and item representations might be too tight, leading to potential problems such as over-emphasizing the effect of text features and exaggerating the negative impact of domain gap. To address this issue, this paper proposes VQ-Rec, a novel approach to learning Vector-Quantized item representations for transferable sequential Recommenders. The main novelty of our approach lies in the new item representation scheme: it first maps item text into a vector of discrete indices (called item code), and then employs these indices to lookup the code embedding table for deriving item representations. Such a scheme can be denoted as "text $\Longrightarrow$ code $\Longrightarrow$ representation". Based on this representation scheme, we further propose an enhanced contrastive pre-training approach, using semi-synthetic and mixed-domain code representations as hard negatives. Furthermore, we design a new cross-domain fine-tuning method based on a differentiable permutation-based network. Extensive experiments conducted on six public benchmarks demonstrate the effectiveness of the proposed approach, in both cross-domain and cross-platform settings. Code and pre-trained model are available at: https://github.com/RUCAIBox/VQ-Rec.
1 INTRODUCTION
Transferable sequential recommenders aim to reuse knowledge across new domains and platforms, but limited overlap and tightly text-bound representations create challenges. VQ-Rec inserts discrete item codes between text and representations, adds specialized pre-training and parameter-efficient adaptation, and shows strong transferability across six benchmarks.
- Existing sequential recommenders are difficult to reuse in new scenarios, requiring costly retraining and risking cold-start problems.
- Partially shared or non-overlapping users and items make conventional cross-domain mapping difficult, especially across platforms.
- VQ-Rec maps item text to discrete codes before embedding lookup, reducing direct text influence while allowing domain- or task-specific adaptation.
- OPQ produces semantically rich, distinguishable codes, while a differentiable permutation-based network aligns code embeddings during parameter-efficient downstream fine-tuning.
- Across six cross-domain and cross-platform benchmarks, VQ-Rec demonstrates strong transferability and can recommend new items without retraining while improving known-item performance.
2 METHODOLOGY
This section presents VQ-Rec, the proposed transferable sequential recommendation approach based on vector-quantized item indices.
- The section introduces VQ-Rec as the proposed transferable sequential recommendation approach.
- VQ-Rec is based on vector-quantized item indices.
- The approach targets transferable sequential recommendation.
2.1 Approach Overview
VQ-Rec uses a Transformer backbone while learning transferable item representations from item text and discrete codes. Its framework organizes representation learning, recommender training, and transfer to new domains.
- 2.1 Approach Overview: The task uses multi-domain interaction sequences, keeping each user’s sequences separate by domain because of semantic gaps across domains.
- 2.1 Approach Overview: VQ-Rec adopts a Transformer backbone without adding components such as adaptors, instead feeding it learned transferable item representations.
- 2.1 Approach Overview: The framework addresses vector-quantized item representation, training with the new representation scheme, and transfer of the pre-trained recommender.
2.2 Vector-Quantized Item Representation
VQ-Rec represents items through a two-step pipeline: PLM-encoded item text is quantized into discrete codes, whose indexed embeddings are pooled into item representations. OPQ is used to improve code distinguishability and reduce collisions.
- 2.2.1 Vector-Quantized Code Learning: Item text is encoded with a PLM, then mapped to discrete codes through optimized product quantization.
- 2.2.1 Vector-Quantized Code Learning: PQ splits each text encoding into sub-vectors and selects the nearest centroid index for each sub-vector to form the item code.
- 2.2.1 Vector-Quantized Code Learning: OPQ learns centroid embeddings from item text encodings across training domains before independently assigning code indices.
- 2.2.2 Code Embedding Lookup as Item Representations: Each code dimension has an embedding matrix, and lookup retrieves the corresponding embeddings that are aggregated into the final item representation.
- 2.2.1 Vector-Quantized Code Learning: Uniformly distributed codes minimize collisions in the ideal quantization setting and support distinguishable item representations.
2.3 Contrastive Recommender Pre-training
VQ-Rec enhances contrastive pre-training for discrete-code item representations by combining semi-synthetic hard negatives with mixed-domain in-batch negatives.
- VQ-Rec uses contrastive learning over sequential contexts and their ground-truth next items, with negative sampling as a central training step.
- Random negatives are inadequate because discrete codes create a sparse representation space and multi-domain training requires domain-gap mitigation.
- Semi-synthetic negatives: Semi-synthetic negatives randomly replace indices in true item codes while preserving remaining indices, producing hard negatives near observed items.
- Semi-synthetic negatives: Uniform sampling makes generated code indices follow a distribution similar to true items, supporting distinguishable semi-synthetic representations.
- Mixed-domain negatives: Mixed-domain negatives use the B−1 in-batch items paired with other contexts, naturally introducing items from multiple domains during pre-training.
- The pre-training objective integrates semi-synthetic and mixed-domain negatives under a temperature hyper-parameter.
2.4 Cross-domain Recommender Fine-tuning
VQ-Rec transfers discrete-code representations across domains by aligning code-to-embedding mappings with differentiable permutation matrices, then adapting the embedding tables efficiently.
- Fine-tuning fixes the transferred Transformer sequence encoder and optimizes only item-representation parameters for parameter-efficient adaptation.
- Fine-tuning separates code-embedding alignment from subsequent code-embedding-table optimization.
- Reusing the original index set and embedding mapping directly neglects cross-domain semantic gaps and weakens downstream transfer.
- Permutation-based code-embedding alignment: VQ-Rec shares the discrete code set but rebuilds index-to-embedding mappings using new product-quantization centroids and permutation-based alignment.
- Alignment optimization: The alignment matrices are optimized with next-item prediction, using the transferred sequence encoder and downstream item representations.
- Doubly stochastic matrices simulate differentiable permutations, while later fine-tuning adapts the permuted embedding tables without item IDs.
2.5 Discussion
The discussion characterizes VQ-Rec as capacity-oriented, flexible, and efficient through discrete codes, decoupled components, and fixed modules during training and transfer.
- Capacity: VQ-Rec uses discrete codes and embeddings to capture transferable patterns while avoiding direct text-to-representation mapping and reducing sensitivity to text noise.
- Flexibility: The approach leaves the Transformer unchanged and separates text encoding and product quantization from sequence-encoder optimization.
- Efficiency: VQ-Rec improves representation-derivation complexity over UniSRec from O(dWdVD) to O(dVD).
- Efficiency: Efficiency also comes from fixing the text encoder, learning discrete codes independently, and fixing the sequence encoder during fine-tuning.
3 EXPERIMENTS
Experiments evaluate VQ-Rec across five Amazon pre-training domains and six downstream benchmarks spanning cross-domain and cross-platform transfer. VQ-Rec achieves the best or second-best performance across all evaluated datasets while supporting inductive recommendation.
- Datasets: The study pre-trains on Food, Home, CDs, Kindle, and Movies, then transfers to five Amazon domains and the cross-platform Online Retail dataset.
- Baselines: Baselines include ID-based, text-based, feature-aware, adversarial, and transferable sequential recommenders such as SASRec, ZESRec, and UniSRec.
- Evaluation: Evaluation uses Recall@K and NDCG@K at K∈{10,50} with leave-one-out splitting and ranking against all other items.
- Overall performance: Text-based models perform better on small datasets, whereas ID-based models perform better on larger datasets including Arts, Office, and Online Retail.
- Overall performance: VQ-Rec achieves the best or second-best performance on all datasets and can recommend new items without retraining in an inductive setting.
3.3 Ablation Study
The ablations show that multi-domain pre-training, semi-synthetic negatives, fine-tuning, code-embedding alignment, and informative text each contribute to VQ-Rec's performance and transferability. Additional analyses report stronger capacity with more training data, positive transfer across all six datasets, and advantages across cold-start popularity groups.
- Without multi-domain pre-training, performance is worse on all three representative datasets, indicating that pre-training transfers general sequential patterns of discrete codes.
- Removing semi-synthetic negatives causes sparsity issues and sub-optimal results, while omitting fine-tuning sharply reduces performance on domains with varied semantics.
- Reusing the pre-training PQ index set produces long-tail code distributions and reduced distinguishability under a large semantic gap, worsening performance.
- Permutation-based code-embedding alignment generally improves performance, while randomly assigned codes usually perform worse and show that vector-quantized codes preserve text characteristics.
- VQ-Rec improves with more training data and benefits from pre-training on all six datasets, with a maximum improvement exceeding 20%.UniSRec suffers negative transfer on Arts, Office, and Online Retail, whereas VQ-Rec benefits across all six experimental datasets.
- VQ-Rec outperforms SASRec across all popularity groups, especially cold-start groups, while direct text representations perform well on some cold-start groups but decline on popular items.
4 RELATED WORK
Related work covers sequential recommendation, transfer learning for recommendation, and sparse representations. VQ-Rec builds on transferable text-based representations while using discrete codes for a distinct focus on representation transfer.
- Sequential recommendation: Sequential recommendation predicts the next interacted item from historical interaction sequences using models including RNNs, CNNs, Transformers, GNNs, and MLPs.
- Transfer learning for recommendation: Transfer-learning methods address sparsity and cold-start issues by transferring knowledge across domains, markets, or platforms, usually through shared users, items, or attributes.
- Sparse representation for recommendation: Sparse representation methods use discrete or sparse codes to capture salient dimensions and have supported efficient recommendation in memory and time.
- Sparse representation for recommendation: VQ-Rec differs from prior sparse-representation recommendation studies by leveraging text semantics for transferable item representations through pre-training.
5 CONCLUSIONS
VQ-Rec learns transferable sequential-recommendation representations by separating text encoding from item representation through discrete codes. Its pre-training and permutation-based alignment support adaptation across domains and platforms.
- VQ-Rec maps PLM text encodings to discrete codes and uses embedding lookup to derive item representations.
- The method uses mixed-domain and semi-synthetic code representations as hard negatives during multi-domain pre-training.
- A permutation-based network learns domain-specific code-embedding alignment to adapt VQ-Rec to downstream domains.
- Experiments on six transferring benchmarks demonstrate VQ-Rec's effectiveness.