PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-training Generative Recommender with Multi-Identifier Item Tokenization

Bowen Zheng, Enze Liu, Zhongfu Chen, Zhongrui Ma, Yue Wang, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2504.04400v3cs.IRcs.AI

TL;DR

One-to-one item identifiers can leave long-tail tokens with limited supervision and restrict token-sequence diversity. MTGRec uses related RQ-VAE tokenizers and influence-guided curriculum pre-training to create and schedule multiple sequence groups. On three public datasets, it outperforms traditional and generative recommendation baselines in effectiveness and scalability.

  • Problem

    One-to-one item identifiers can leave long-tail tokens low-frequency and weakly supervised while restricting token-sequence diversity.

    Source: §1 · p. 2

  • Method

    MTGRec uses adjacent-epoch RQ-VAE checkpoints as related tokenizers to create multiple sequence groups, then applies influence-guided curriculum sampling and single-identifier fine-tuning.

    Source: §1 · p. 2

  • Results

    Experiments on three public datasets show MTGRec outperforming traditional and generative recommendation baselines in effectiveness and scalability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that multiple related tokenizers provide more extensive and diverse pre-training data, supporting performance improvement through model scaling.

    Source: §1 · p. 2

Abstract

from arXiv · show

Generative recommendation autoregressively generates item identifiers to recommend potential items. Existing methods typically adopt a one-to-one mapping strategy, where each item is represented by a single identifier. However, this scheme poses issues, such as suboptimal semantic modeling for low-frequency items and limited diversity in token sequence data. To overcome these limitations, we propose MTGRec, which leverages Multi-identifier item Tokenization to augment token sequence data for Generative Recommender pre-training. Our approach involves two key innovations: multi-identifier item tokenization and curriculum recommender pre-training. For multi-identifier item tokenization, we leverage the RQ-VAE as the tokenizer backbone and treat model checkpoints from adjacent training epochs as semantically relevant tokenizers. This allows each item to be associated with multiple identifiers, enabling a single user interaction sequence to be converted into several token sequences as different data groups. For curriculum recommender pre-training, we introduce a curriculum learning scheme guided by data influence estimation, dynamically adjusting the sampling probability of each data group during recommender pre-training. After pre-training, we fine-tune the model using a single tokenizer to ensure accurate item identification for recommendation. Extensive experiments on three public benchmark datasets demonstrate that MTGRec significantly outperforms both traditional and generative recommendation baselines in terms of effectiveness and scalability.

1 Introduction

Generative recommendation autoregressively generates item identifiers, but one-to-one item tokenization can leave long-tail tokens with little supervision and limit sequence diversity. MTGRec uses semantically related tokenizers to create multiple training sequences and curriculum pre-training to adjust their sampling.

1 Introduction

  • One-to-one item tokenization leaves long-tail tokens low-frequency and weakly supervised, while limiting token-sequence diversity and potential gains from model scaling.These are the two potential issues identified for prior methods.Source: §1 · p. 2
  • MTGRec uses adjacent-epoch RQ-VAE checkpoints as semantically relevant tokenizers, associating items with multiple identifiers and expanding each interaction sequence into several training sequences.The resulting data is organized into groups derived from different tokenizers.Source: §1 · p. 2
  • A data-influence curriculum adaptively adjusts the sampling proportions of token-sequence groups during recommender pre-training.The method estimates influence using a first-order gradient approximation.Source: §1 · p. 2
  • After pre-training, MTGRec fine-tunes with a single item identifier to ensure precise item identification during recommendation.Source: §1 · p. 2
  • Experiments on three public datasets show MTGRec outperforming traditional and generative recommendation baselines in effectiveness and scalability.Source: §1 · p. 2

2 Methodology

MTGRec uses multi-identifier item tokenization to augment token sequences for generative recommender pre-training.

2 Methodology

  • MTGRec uses multi-identifier item tokenization to augment token sequences for generative recommender pre-training.Source: §2 · p. 2
  • The section presents MTGRec as a generative recommender framework.Source: §2 · p. 2
  • The described augmentation is applied to token sequences used in recommender pre-training.Source: §2 · p. 2

2.1 Overview

Sequential recommendation uses a user’s chronological interaction history to predict the next item; generative recommendation casts this as autoregressive generation of an item identifier. MTGRec associates items with multiple identifiers to create diverse training sequences and uses curriculum pre-training to schedule data groups.

2.1.1 Problem Formulation.

  • Sequential recommendation predicts the next item from a user’s chronological interaction history, while generative recommendation represents items as token sequences and autoregressively generates the target identifier.Source: §2.1.1 · p. 2

2.1.2 Method Overview.

  • MTGRec replaces one-to-one item-to-identifier mapping with multiple identifiers to construct more extensive and diverse token-sequence data for pre-training.Source: §2.1.2 · p. 2
  • Adjacent-epoch RQ-VAE checkpoints serve as semantically related tokenizers, letting one item sequence yield several token sequences with related but distinct semantic knowledge.Source: §2.1.2 · p. 3, Figure 1 · p. 3
  • A data-influence curriculum adaptively schedules the proportions of sequence groups during pre-training, then fine-tunes using a single item identifier for precise identification.The influence estimate uses first-order gradient approximation.Source: §2.1.2 · p. 3, Figure 1 · p. 3

2.2 Multi-Identifier Item Tokenization

MTGRec creates multiple related item tokenizers from adjacent RQ-VAE training checkpoints, then uses them to produce several token-sequence groups for recommender pre-training. The model samples among these groups and is fine-tuned with a single identifier for precise item identification.

2.2.1 Tokenizer Backbone.

  • RQ-VAE encodes an item semantic embedding into a latent representation, quantizes it into tokens through H-level residual quantization, and decodes the quantized representation to reconstruct the embedding.Its loss combines reconstruction and residual-quantization losses; beta balances optimization between encoder and codebooks and is typically 0.25.Source: §2.2.1 · p. 3

2.2.2 Semantically Relevant Tokenizers.

  • Independent randomly initialized RQ-VAEs can produce extraneous token sequences with no homogeneous knowledge and possible semantic conflicts; adjacent-epoch checkpoints instead yield related but distinct token semantics.The adjacent checkpoints share an initialization and have minimal codebook disparities between neighboring epochs.Source: §2.2.2 · pp. 3–4

2.2.3 Tokenize an Item Sequence to Multiple Token Sequences.

  • Different tokenizers map the same historical item sequence and target item to multiple input and target token sequences.Each tokenizer produces one corresponding input sequence and target identifier.Source: §2.2.3 · p. 4
  • Rather than use all augmented sequences at once, MTGRec samples one sequence per optimization step because large tokenizer counts make data volume unmanageable and group proportions difficult to adjust.Repeated sampling is described as approximately equivalent to using all the data.Source: §2.2.3 · p. 4

2.3 Curriculum Recommender Pre-training

MTGRec estimates each tokenizer’s data influence from its contribution to validation loss, then uses accumulated influence to adapt the proportions of token-sequence groups during pre-training.

2.3 Curriculum Recommender Pre-training

  • MTGRec dynamically adjusts the proportions of token-sequence data groups during pre-training according to estimated data influence.Influence is estimated for each tokenizer’s data group using first-order gradient approximation.Source: §2.3 · p. 4

2.3.1 Estimating Data Influence.

  • Data influence measures each training group’s contribution to validation loss, estimated using gradient information and a first-order Taylor expansion.The method considers a tokenizer’s data group as a whole rather than estimating influence for individual instances.Source: §2.3.1 · p. 4
  • Validation loss is computed over held-out leave-one-out data mixed across the tokenizers, with mean loss and cumulative gradient across validation examples.Training-group gradients are calculated by gradient accumulation, treating each tokenizer’s data group as a batch.Source: §2.3.1 · p. 4
  • Tokenizer influence is accumulated across model checkpoints to capture its contribution over multiple training steps.The method sums influence across checkpoints rather than using only a single step’s estimate.Source: §2.3.1 · p. 5

2.3.2 Curriculum Pre-training.

  • Training is divided into stages, and sampling probabilities are updated at each stage end using the latest tokenizer influence and prior cumulative influence.The temperature coefficient controls the smoothness of the sampling-probability distribution; groups begin with equal sampling probability.Source: §2.3.2 · p. 5
  • The sampled token-sequence pairs are fed to the generative recommender for optimization with negative log-likelihood loss.Source: §2.3.2 · p. 5

2.4 Fine-tuning and Inference

MTGRec fine-tunes its pre-trained recommender with one tokenizer to restore one-to-one item identification, then uses beam search to generate recommendations.

2.4 Fine-tuning and Inference

  • The pre-trained recommender is fine-tuned separately with each tokenizer, and the model with the best validation performance is selected for deployment and testing.This restores one-to-one item-to-identifier mapping, which is not guaranteed during multi-tokenizer pre-training.Source: §2.4 · p. 5
  • At inference, beam search decodes K token sequences from the full item set and maps them to items.The method omits a prefix tree to avoid hindering parallel decoding and reducing efficiency; rare invalid identifiers are ignored.Source: §2.4 · p. 5

3 Experiments

Experiments compare MTGRec with traditional and generative recommenders on three Amazon 2023 review subsets using full-ranking evaluation. MTGRec is reported to outperform both baseline groups across all cases.

3.1.1 Dataset.

  • The approach was evaluated on three Amazon 2023 review subsets—Musical Instruments, Industrial and Scientific, and Video Games—with user review data spanning May 1996 through September 2023.Preprocessing filtered low-activity users and items with fewer than five interaction records, then sorted each user's history chronologically and capped sequences at 20 items.Source: §3.1.1 · p. 5

3.1.2 Baseline Models.

  • The comparison includes traditional sequential recommenders and generative models TIGER, LETTER, and TIGER++.LETTER adds collaborative and diversity regularization to TIGER, while TIGER++ uses representation whitening and EMA for codebook learning.Source: §3.1.2 · pp. 5–6

3.1.3 Evaluation Settings.

  • Models are evaluated with Recall and NDCG at K=5 and 10 using leave-one-out splits and full ranking over the entire item set.Generative models use beam size 50.Source: §3.1.3 · p. 6

3.1.4 Implementation Details.

  • MTGRec uses Sentence-T5 item embeddings with an RQ-VAE tokenizer and a T5 generative recommender.RQ-VAE uses three codebooks of size 256 plus a collision-handling codebook; final-epoch checkpoints are selected as related tokenizers, with n tuned from 5 to 30. The recommender is pre-trained for 200 epochs, with sampling probabilities updated every 20 epochs after a 60-epoch warmup.Source: §3.1.4 · p. 6

3.2 Overall Performance

  • Generative recommendation models generally outperform traditional sequential models, while FDSA performs better than traditional models using only item IDs and collaborative information.The paper attributes the generative-model pattern to semantic item identifiers and the generative paradigm, and links FDSA’s performance to supplementary item-text features.Source: §3.2 · p. 6
  • MTGRec consistently achieves the best performance across all cases, substantially improving over traditional and generative recommendation baselines.The authors attribute its effectiveness and scalability to pre-training on larger, more diverse sequence data from multiple item tokenizers.Source: §3.2 · p. 6

3.3 Ablation Study

The ablations indicate that influence-based data curriculum, semantically relevant tokenizers, and pre-training on their augmented sequences each contribute to MTGRec's performance.

3.3 Ablation Study

  • Without influence-based data curriculum, equal-probability sampling performs worse than MTGRec across all datasets.This supports the contribution of curriculum learning to generative recommender pre-training.Source: §3.3 · pp. 6–7
  • Using randomly initialized, irrelevant tokenizers causes semantic conflicts, model-learning collapse, and significant performance degradation.The authors identify adjacent-epoch RQ-VAE checkpoints as semantically relevant tokenizers.Source: §3.3 · p. 7
  • Pre-training on augmented sequences from multiple semantically relevant tokenizers is a critical element of MTGRec's effectiveness.The no-pre-training variant instead learns from a single tokenizer.Source: §3.3 · pp. 7–8

3.4 Further Analysis

The further analyses examine model scale, tokenizer count, curriculum temperature, long-tail performance, transfer to other generative recommenders, identifier consistency, and training efficiency. They report benefits and trade-offs across these settings.

3.4.1 Performance Comparison w.r.t. Model Scale.

  • MTGRec outperforms TIGER and TIGER++ across tested model scales, with performance generally rising as layer count increases, though scaling is constrained by finite interaction-derived data.The figure reports MTGRec's best results across tokenizer counts; the authors note that maintaining semantic relevance while generating enough sequences limits larger-scale training.Source: §3.4.1 · p. 8

3.4.2 Performance Comparison w.r.t. Tokenizer Number.

  • Fewer tokenizers offer only marginal advancement, while too many can weaken semantic relevance; the optimal tokenizer count increases for 6-layer models.MTGRec must trade off sequence-data volume against tokenizer semantic relevance.Source: §3.4.2 · p. 8

3.4.3 Performance Comparison w.r.t. Temperature Coefficient.

  • The optimal temperature coefficient is 3 on Instrument and 1 on Scientific; smaller values favor high-probability tokenizers, while larger values approach uniform sampling.Both extremes adversely affect curriculum pre-training.Source: §3.4.3 · p. 8, Figure 4 · p. 8

3.4.4 Performance Comparison w.r.t. Long-tail Items.

  • MTGRec consistently outperforms the baseline across item-popularity groups, with greater improvement over TIGER and TIGER++ for unpopular targets such as group [0, 20).Figure 5's bars show test interactions per group, while its line chart shows Recall@10 improvement ratios versus TIGER.Source: Figure 5 · p. 8, §3.4.4 · p. 9

3.4.5 Applying MTGRec on Other Generative Recommendation Meth-

  • MTGRec consistently improves the performance of other generative recommendation methods, including TIGER and LETTER, on Instrument and Scientific.Table 4 reports the comparison on these two datasets.Source: §3.4.5 · p. 9, Table 4 · p. 9

3.4.6 Multiple Identifier Difference Analysis.

  • Adjacent tokenizers with a one-epoch interval show minimal identifier changes, while wider intervals increase changes and the risk of semantic conflict.Even with larger intervals, first-token changes typically remain below 1%, preserving core semantic information.Source: §3.4.6 · p. 9, Table 5 · p. 9

3.4.7 Efficiency Analysis.

  • MTGRec uses 200 pre-training epochs followed by low-cost fine-tuning, without excessive training-time cost versus baselines; curriculum learning accelerates convergence on Game.The efficiency comparison uses the same settings and hardware environment.Source: §3.4.7 · p. 9, Table 6 · p. 9

4 Related Work

Prior sequential recommenders model interaction histories, while generative recommendation represents items with token-list identifiers. The paper situates MTGRec against these approaches and their limitations.

4.1 Sequential Recommendation

  • Sequential recommendation predicts likely next items from users' historical behavior sequences.Methods span Markov chains, CNNs, RNNs, GNNs, and Transformers; ID- and collaborative-filtering-based approaches can overlook item content.Source: §4.1 · p. 9
  • Some sequential recommenders incorporate item attributes or pre-trained language-model embeddings to use item content and improve performance and generalization.These approaches are presented as attempts to enhance ID-sequence modeling.Source: §4.1 · p. 10

4.2 Generative Recommendation

  • Generative recommendation indexes each item with a token-list identifier; tokenization approaches include heuristic, text-based, and codebook-based methods, each with stated trade-offs.The paper contrasts MTGRec's multiple semantically relevant tokenizers with the prevalent one-to-one mapping, which it associates with long-tail, sparsity, and sequence-diversity challenges.Source: §4.2 · p. 10

5 Conclusion

MTGRec uses adjacent-epoch RQ-VAE checkpoints as semantically relevant tokenizers and curriculum pre-training to augment generative recommender training. Experiments on three public datasets show superior performance over traditional and generative recommendation baselines; future work will explore broader scenarios and larger models.

5 Conclusion

  • Adjacent-epoch RQ-VAE checkpoints serve as semantically relevant tokenizers, associating items with multiple identifiers and augmenting sequence data into distinct groups.Curriculum pre-training dynamically adjusts each group's sampling probability using data influence estimation; fine-tuning on each tokenizer supports accurate item identification, with the best model selected for deployment and testing.Source: §5 · p. 10
  • MTGRec outperformed traditional and generative recommendation baselines in experiments on three public datasets.The conclusion also reports in-depth analyses but gives no quantitative results here.Source: §5 · p. 10
  • Future work will adapt multi-identifier tokenization to transferable and multi-domain recommendation and investigate scaling model parameters to the billion level.The authors also plan to investigate the scaling effect as model parameters increase.Source: §5 · p. 10

Related papers

Loading 2504.04400v3…