PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders

Yupeng Hou, Zhankui He, Julian McAuley, Wayne Xin Zhao

arXiv:2210.12316v2cs.IRcs.LG

TL;DR

Existing text-based transferable recommenders can bind item representations too tightly to text, while cross-domain transfer is difficult when domains share few users or items. VQ-Rec inserts discrete codes between text and representations, enhances contrastive pre-training, and uses permutation-based fine-tuning; experiments on six benchmarks demonstrate strong transferability, including new-item recommendation without retraining.

  • Problem

    Existing transfer methods can overemphasize item-text features, while limited overlap across domains and platforms makes sequential recommender reuse difficult.

    Source: §1 · pp. 1–2

  • Method

    VQ-Rec maps item text to discrete codes, looks up code embeddings for item representations, uses hard negatives in contrastive pre-training, and aligns embeddings with permutation-based fine-tuning.

    Source: §1 · p. 2, §2.3.2 · p. 4, §5 · p. 8

  • Results

    Experiments on six cross-domain and cross-platform benchmarks demonstrate strong transferability, including recommending new items without retraining and improving known-item performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Discrete code-based representations capture transferable sequential patterns while reducing direct dependence on text similarity and supporting adaptation across domains or platforms.

    Source: §2.5 · p. 5, §3.3 · p. 7

    The approach assumes multi-domain interaction data for training or pre-training and keeps a user’s interaction sequences separate by domain because of semantic gaps.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Recently, the generality of natural language text has been leveraged to develop transferable recommender systems. The basic idea is to employ pre-trained language models~(PLM) to encode item text into item representations. Despite the promising transferability, the binding between item text and item representations might be too tight, leading to potential problems such as over-emphasizing the effect of text features and exaggerating the negative impact of domain gap. To address this issue, this paper proposes VQ-Rec, a novel approach to learning Vector-Quantized item representations for transferable sequential Recommenders. The main novelty of our approach lies in the new item representation scheme: it first maps item text into a vector of discrete indices (called item code), and then employs these indices to lookup the code embedding table for deriving item representations. Such a scheme can be denoted as "text $\Longrightarrow$ code $\Longrightarrow$ representation". Based on this representation scheme, we further propose an enhanced contrastive pre-training approach, using semi-synthetic and mixed-domain code representations as hard negatives. Furthermore, we design a new cross-domain fine-tuning method based on a differentiable permutation-based network. Extensive experiments conducted on six public benchmarks demonstrate the effectiveness of the proposed approach, in both cross-domain and cross-platform settings. Code and pre-trained model are available at: https://github.com/RUCAIBox/VQ-Rec.

1 INTRODUCTION

Transferable sequential recommenders aim to reuse knowledge across new domains and platforms, but limited overlap and tightly text-bound representations create challenges. VQ-Rec inserts discrete item codes between text and representations, adds specialized pre-training and parameter-efficient adaptation, and shows strong transferability across six benchmarks.

  • Existing sequential recommenders are difficult to reuse in new scenarios, requiring costly retraining and risking cold-start problems.Source: §1 · p. 1
  • Partially shared or non-overlapping users and items make conventional cross-domain mapping difficult, especially across platforms.Source: §1 · p. 1
  • VQ-Rec maps item text to discrete codes before embedding lookup, reducing direct text influence while allowing domain- or task-specific adaptation.Source: §1 · p. 2
  • OPQ produces semantically rich, distinguishable codes, while a differentiable permutation-based network aligns code embeddings during parameter-efficient downstream fine-tuning.Source: §1 · p. 2
  • Across six cross-domain and cross-platform benchmarks, VQ-Rec demonstrates strong transferability and can recommend new items without retraining while improving known-item performance.Source: §1 · p. 2

2 METHODOLOGY

This section presents VQ-Rec, the proposed transferable sequential recommendation approach based on vector-quantized item indices.

  • The section introduces VQ-Rec as the proposed transferable sequential recommendation approach.Source: §2 · p. 2
  • VQ-Rec is based on vector-quantized item indices.Source: §2 · p. 2
  • The approach targets transferable sequential recommendation.Source: §2 · p. 2

2.1 Approach Overview

VQ-Rec uses a Transformer backbone while learning transferable item representations from item text and discrete codes. Its framework organizes representation learning, recommender training, and transfer to new domains.

2.1 Approach Overview

  • The task uses multi-domain interaction sequences, keeping each user’s sequences separate by domain because of semantic gaps across domains.Source: §2.1 · p. 2
  • VQ-Rec adopts a Transformer backbone without adding components such as adaptors, instead feeding it learned transferable item representations.Source: §2.1 · p. 2
  • The framework addresses vector-quantized item representation, training with the new representation scheme, and transfer of the pre-trained recommender.Source: §2.1 · pp. 2–3

2.2 Vector-Quantized Item Representation

VQ-Rec represents items through a two-step pipeline: PLM-encoded item text is quantized into discrete codes, whose indexed embeddings are pooled into item representations. OPQ is used to improve code distinguishability and reduce collisions.

2.2.1 Vector-Quantized Code Learning

  • Item text is encoded with a PLM, then mapped to discrete codes through optimized product quantization.Source: §2.2 · p. 3, §2.2.1 · p. 3
  • PQ splits each text encoding into sub-vectors and selects the nearest centroid index for each sub-vector to form the item code.Source: §2.2.1 · p. 3
  • OPQ learns centroid embeddings from item text encodings across training domains before independently assigning code indices.Source: §2.2.1 · p. 3

2.2.2 Code Embedding Lookup as Item Representations

  • Each code dimension has an embedding matrix, and lookup retrieves the corresponding embeddings that are aggregated into the final item representation.Source: §2.2.2 · p. 3

2.2.1 Vector-Quantized Code Learning

  • Uniformly distributed codes minimize collisions in the ideal quantization setting and support distinguishable item representations.Source: §2.2.3 · pp. 3–4

2.3 Contrastive Recommender Pre-training

VQ-Rec enhances contrastive pre-training for discrete-code item representations by combining semi-synthetic hard negatives with mixed-domain in-batch negatives.

  • VQ-Rec uses contrastive learning over sequential contexts and their ground-truth next items, with negative sampling as a central training step.Source: §2.3.2 · p. 4
  • Random negatives are inadequate because discrete codes create a sparse representation space and multi-domain training requires domain-gap mitigation.Source: §2.3.2 · p. 4

Semi-synthetic negatives

  • Semi-synthetic negatives randomly replace indices in true item codes while preserving remaining indices, producing hard negatives near observed items.Source: §2.3.2 · p. 4
  • Uniform sampling makes generated code indices follow a distribution similar to true items, supporting distinguishable semi-synthetic representations.Source: §2.3.2 · p. 4

Mixed-domain negatives

  • Mixed-domain negatives use the B−1 in-batch items paired with other contexts, naturally introducing items from multiple domains during pre-training.Source: §2.3.2 · p. 4
  • The pre-training objective integrates semi-synthetic and mixed-domain negatives under a temperature hyper-parameter.Source: §2.3.2 · p. 4

2.4 Cross-domain Recommender Fine-tuning

VQ-Rec transfers discrete-code representations across domains by aligning code-to-embedding mappings with differentiable permutation matrices, then adapting the embedding tables efficiently.

  • Fine-tuning fixes the transferred Transformer sequence encoder and optimizes only item-representation parameters for parameter-efficient adaptation.Source: §2.4 · p. 4
  • Fine-tuning separates code-embedding alignment from subsequent code-embedding-table optimization.Source: §2.4 · p. 4
  • Reusing the original index set and embedding mapping directly neglects cross-domain semantic gaps and weakens downstream transfer.Source: §2.4.1 · p. 5

Permutation-based code-embedding alignment

  • VQ-Rec shares the discrete code set but rebuilds index-to-embedding mappings using new product-quantization centroids and permutation-based alignment.Source: §2.4.1 · pp. 4–5

Alignment optimization

  • The alignment matrices are optimized with next-item prediction, using the transferred sequence encoder and downstream item representations.Source: §2.4.1 · p. 5
  • Doubly stochastic matrices simulate differentiable permutations, while later fine-tuning adapts the permuted embedding tables without item IDs.Source: §2.4.1 · p. 5, §2.4.2 · p. 5

2.5 Discussion

The discussion characterizes VQ-Rec as capacity-oriented, flexible, and efficient through discrete codes, decoupled components, and fixed modules during training and transfer.

Capacity

  • VQ-Rec uses discrete codes and embeddings to capture transferable patterns while avoiding direct text-to-representation mapping and reducing sensitivity to text noise.Source: §2.5 · p. 5

Flexibility

  • The approach leaves the Transformer unchanged and separates text encoding and product quantization from sequence-encoder optimization.Source: §2.5 · p. 5

Efficiency

  • VQ-Rec improves representation-derivation complexity over UniSRec from O(dWdVD) to O(dVD).Source: §2.5 · p. 5
  • Efficiency also comes from fixing the text encoder, learning discrete codes independently, and fixing the sequence encoder during fine-tuning.Source: §2.5 · p. 5

3 EXPERIMENTS

Experiments evaluate VQ-Rec across five Amazon pre-training domains and six downstream benchmarks spanning cross-domain and cross-platform transfer. VQ-Rec achieves the best or second-best performance across all evaluated datasets while supporting inductive recommendation.

Datasets

  • The study pre-trains on Food, Home, CDs, Kindle, and Movies, then transfers to five Amazon domains and the cross-platform Online Retail dataset.Source: §3.1.1 · p. 5

Baselines

  • Baselines include ID-based, text-based, feature-aware, adversarial, and transferable sequential recommenders such as SASRec, ZESRec, and UniSRec.Source: §3.1.2 · p. 6

Evaluation

  • Evaluation uses Recall@K and NDCG@K at K∈{10,50} with leave-one-out splitting and ranking against all other items.Source: §3.1.3 · p. 6

Overall performance

  • Text-based models perform better on small datasets, whereas ID-based models perform better on larger datasets including Arts, Office, and Online Retail.Source: §3.2 · pp. 6–7
  • VQ-Rec achieves the best or second-best performance on all datasets and can recommend new items without retraining in an inductive setting.Source: §3.2 · p. 7

3.3 Ablation Study

The ablations show that multi-domain pre-training, semi-synthetic negatives, fine-tuning, code-embedding alignment, and informative text each contribute to VQ-Rec's performance and transferability. Additional analyses report stronger capacity with more training data, positive transfer across all six datasets, and advantages across cold-start popularity groups.

  • Without multi-domain pre-training, performance is worse on all three representative datasets, indicating that pre-training transfers general sequential patterns of discrete codes.Source: §3.3 · p. 7
  • Removing semi-synthetic negatives causes sparsity issues and sub-optimal results, while omitting fine-tuning sharply reduces performance on domains with varied semantics.Source: §3.3 · p. 7
  • Reusing the pre-training PQ index set produces long-tail code distributions and reduced distinguishability under a large semantic gap, worsening performance.Source: §3.3 · p. 7
  • Permutation-based code-embedding alignment generally improves performance, while randomly assigned codes usually perform worse and show that vector-quantized codes preserve text characteristics.Source: §3.3 · p. 7
  • VQ-Rec improves with more training data and benefits from pre-training on all six datasets, with a maximum improvement exceeding 20%.UniSRec suffers negative transfer on Arts, Office, and Online Retail, whereas VQ-Rec benefits across all six experimental datasets.Source: §3.4.1 · p. 7, §3.4.2 · pp. 7–8
  • VQ-Rec outperforms SASRec across all popularity groups, especially cold-start groups, while direct text representations perform well on some cold-start groups but decline on popular items.Source: §3.4.3 · p. 8, Figure 4 · p. 8

4 RELATED WORK

Related work covers sequential recommendation, transfer learning for recommendation, and sparse representations. VQ-Rec builds on transferable text-based representations while using discrete codes for a distinct focus on representation transfer.

Sequential recommendation

  • Sequential recommendation predicts the next interacted item from historical interaction sequences using models including RNNs, CNNs, Transformers, GNNs, and MLPs.Source: §4 · p. 8

Transfer learning for recommendation

  • Transfer-learning methods address sparsity and cold-start issues by transferring knowledge across domains, markets, or platforms, usually through shared users, items, or attributes.Source: §4 · p. 8

Sparse representation for recommendation

  • Sparse representation methods use discrete or sparse codes to capture salient dimensions and have supported efficient recommendation in memory and time.Source: §4 · p. 8
  • VQ-Rec differs from prior sparse-representation recommendation studies by leveraging text semantics for transferable item representations through pre-training.Source: §4 · p. 8

5 CONCLUSIONS

VQ-Rec learns transferable sequential-recommendation representations by separating text encoding from item representation through discrete codes. Its pre-training and permutation-based alignment support adaptation across domains and platforms.

  • VQ-Rec maps PLM text encodings to discrete codes and uses embedding lookup to derive item representations.Source: §5 · p. 8
  • The method uses mixed-domain and semi-synthetic code representations as hard negatives during multi-domain pre-training.Source: §5 · p. 8
  • A permutation-based network learns domain-specific code-embedding alignment to adapt VQ-Rec to downstream domains.Source: §5 · p. 8
  • Experiments on six transferring benchmarks demonstrate VQ-Rec's effectiveness.Source: §5 · p. 8

Related papers

Loading 2210.12316v2…