PaperlayerView exact revision on arXiv

Source-linked AI summary

Tlow: Flow-based Item Tokenizer for Recommendation

Nian Li, Chonggang Song, Jingtao Ding, Lingling Yi, Yong Li, Qingmin Liao

arXiv:2608.24176v1cs.IRcs.AI

TL;DR

Traditional item tokenizers face parameter growth, cold-start difficulty, and embedding distributions that complicate independent quantization. Tlow transforms semantic embeddings into a standard normal latent space and adds codebook guidance, improving recommendation performance across general, cross-domain, multimodal, and cold-start settings, including online gains over random IDs.

  • Problem

    Random item-ID embeddings cause parameter growth with item count and struggle with new items, while semantic-embedding correlations complicate independent tokenization.

  • Method

    Tlow uses a flow-based tokenizer to transform semantic embeddings into a standard normal latent space and aligns codebook and token-embedding spaces through codebook guidance.

  • Results

    Tlow improves recommendation performance across general, cross-domain, multimodal, and cold-start scenarios; online CTR is 4.79% higher in single-domain and 6.23% higher in cross-domain settings than random-ID models.

  • Takeaways & Limitations

    The results support simplifying semantic embedding distributions before independent tokenization and using token IDs across general, cross-domain, multimodal, and cold-start recommendation settings.

Abstract

from arXiv · show

Item tokenizer encodes semantic embeddings into token IDs to replace the randomly assigned item IDs used in traditional recommendation models, fundamentally addressing the problems of excessive parameters and cold starts. However, the most common tokenizer, RQ-VAE, suffers from low decoding efficiency due to the inherent dependencies among its codebooks. Meanwhile, efficient independent tokenizers such as optimized product quantization (OPQ) still struggle with dimensional correlations and distribution complexity of semantic embeddings. In this work, we propose a f\underline{low}-based item \underline{T}okenizer (Tlow) to transform raw semantic embeddings into a latent space where embeddings conform to a unified standard normal distribution, achieving dual advantages of dimensional independence and distributional simplicity. Independent tokenization performed on these latent embeddings yields semantically clear token IDs. Additionally, we introduce a novel codebook guidance to align the codebook space with the token embedding space, further aiding the learning of more semantically distinct token embeddings. Offline experiments on four public datasets demonstrate that Tlow's tokenization and codebook guidance significantly improve recommendation performance. The improvement on cross-domain and multi-modal recommendations also proves the effectiveness of item tokenization in a simplified embedding space. Online experiments for a multi-modal retrieval task on China's largest social media platform WeChat validate Tlow's powerful distribution transformation capability. The retrieval model based on token IDs improves user CTR by 10.32\% globally and by 11.64\% for new items. Our codes are available at https://github.com/wjjln/Tlow.

1 Introduction

Tlow addresses the parameter and cold-start limitations of random item IDs by transforming semantic embeddings into a simpler latent space for independent tokenization. Its flow-based transformation and codebook guidance support clearer token semantics and improved recommendation performance.

  • Random item-ID embeddings make parameters grow linearly with item count and struggle with new items having few interaction records.
  • Semantic item tokenization creates a shared vocabulary whose token embeddings constrain model parameters and support cold-start recommendation.
  • Correlated dimensions and complex, anisotropic embedding distributions make independent quantization inaccurate, especially across diverse domains and modalities.
  • Tlow transforms raw semantic embeddings into standard-normal latent embeddings with independent dimensions, enabling more accurate independent tokenization and clearer token semantics.
  • Tlow improves recommendation performance across four datasets and supports gains in cross-domain, multimodal, and online retrieval settings.
  • Tlow contains multi-step flow blocks using ActNorm, invertible linear, and affine coupling layers before product-quantizing latent embeddings into discrete token IDs.
  • Codebook guidance directly aligns the token-embedding and codebook spaces, helping token embeddings preserve the clearer semantics encoded by codewords.

3 Experiments

Tlow is evaluated in general, cross-domain, and multi-modal sequential recommendation scenarios.

  • The experiments assess Tlow across general, cross-domain, and multi-modal sequential recommendation.

3.1 Experimental Setup

The study uses Amazon Reviews categories and a cross-domain dataset, compares traditional and tokenization-based models, and evaluates ranking with full-item leave-one-out protocols.

  • Datasets: Experiments use four Amazon Reviews categories plus the Cloth-Sports cross-domain dataset.Cloth-Sports combines Clothing Shoes and Jewelry with Sports and Outdoors, with overlapping users across categories.
  • Evaluation: The last historical item is reserved for testing and the second-to-last item for validation under leave-one-out evaluation.
  • Baselines: Baselines include traditional ID-based recommenders and tokenization-based models using different neural architectures and quantization techniques.
  • Evaluation: Performance is measured with Recall@5, Recall@10, NDCG@5, and NDCG@10 using full ranking over all items.
  • Implementation: All models use sentence-t5-base item semantic embeddings with dimension d_s = 768, while Tlow uses N = 4 blocks and M = 4 flow steps.

3.2 Overall Performance

Tokenization-based models outperform purely ID-based models, and Tlow achieves the best results across all reported metrics and four datasets.

  • Tokenization-based models significantly outperform purely ID-based models in recommendation performance.
  • Tlow performs best on all metrics across four datasets, indicating stronger tokenization in its transformed latent space.With the same sentence-t5-base embeddings, RPG does not consistently outperform TIGER.

3.3 Ablation Study

Ablation results show that both transformed semantic latent embeddings and codebook guidance are important for Tlow’s recommendation performance.

  • Removing codebook guidance reduces the semantic clarity of token embeddings and weakens recommendation performance.
  • Replacing transformed latent embeddings with randomly sampled standard-normal vectors causes a significant performance drop.The result confirms that Tlow preserves semantic information needed for behavior modeling during transformation.

3.4 Cross-domain and Multi-modal Recommendation

Tlow is evaluated for cross-domain and multi-modal recommendation by transforming heterogeneous semantic embeddings before tokenization. It significantly improves cross-domain results and consistently improves all reported multi-modal metrics, although the single-domain multi-modal margin is moderate.

  • Cross-domain Recommendation: Tlow significantly outperforms LLM4CDSR and RPG in cross-domain recommendation.The evaluation uses mixed item sequences from two domains.
  • Cross-domain Recommendation: Shared token IDs bridge domains by providing a common vocabulary for training token embeddings.The paper identifies this shared vocabulary as one reason for Tlow’s cross-domain effectiveness.
  • Multi-modal Recommendation: Tlow consistently improves over HM4SR and RPG on all four multi-modal metrics after distribution transformation.The setting combines text and image embeddings tokenized independently before merging their token IDs.
  • Multi-modal Recommendation: RPG provides limited and inconsistent gains over HM4SR in the combined text-image setting.RPG ties HM4SR on R@5 and underperforms on N@5 and N@10.
  • Multi-modal Recommendation: Tlow’s absolute improvement over HM4SR is moderate in this single-domain multi-modal setting.For example, R@10 increases from 0.0469 to 0.0521.

3.5 Cold-Start Recommendation

The cold-start analysis evaluates Tlow across user interaction depths and item popularity levels on the CDs dataset. Tlow performs significantly better than RPG across all examined groups.

  • Evaluation Setup: The cold-start evaluation groups users by interaction depth and test items by training-set popularity.It uses the largest CDs dataset to compare performance across these user and item groups.
  • Cold-Start Results: Tlow performs significantly better than RPG across all user and item groups.The result covers items across popularity levels and users with varying interaction depths.
  • Cold-Start Results: Tlow transforms embeddings, including long-tail item embeddings, into a standard normal distribution for tokenization.The paper links this transformation to more comprehensive tokenization across popularity levels.

3.6 Online Experiments

Tlow is deployed as a supplemental retrieval pathway in WeChat’s picture recommendation system alongside an existing DSSM-based retrieval method. Compared with random-ID embeddings, the Tlow pathway improves pathway-level CTR and UCTR, with larger advantages reported for newly published pictures.

  • Online Setup: Tlow and random-ID sequential models are added as supplemental retrieval pathways in a WeChat picture recommendation system.Both use recently clicked-item sequences and a 12-layer Transformer decoder, alongside the primary DSSM-based retrieval method.
  • Online Setup: The online setup evaluates both single-domain picture sequences and cross-domain sequences mixing clicked pictures and articles.The cross-domain setting tests Tlow’s tokenization across these two content types.
  • Online Performance: 4.79% and 6.23% higher CTR are achieved by Tlow than random IDs in single-domain and cross-domain scenarios, respectively.These are pathway-level CTR improvements reported in Table 6.
  • Online Performance: 10.32% and 7.20% higher UCTR are achieved by Tlow than random IDs in single-domain and cross-domain scenarios, respectively.The larger UCTR gains indicate improved retrieval across the reported user-base measure.
  • Cold-start Online Results: Tlow’s advantages become more pronounced when evaluation is restricted to newly published pictures.Random-ID embeddings require interaction data, whereas Tlow uses semantic features available at publication.
  • Online Performance: 24-hour overall and newly published picture CTR increase by 0.78% and 1.94% in the single-domain scenario.In the cross-domain scenario, per-capita picture CTR improves by 1.05%, while top-tier accounts’ exposure share decreases by 1.15%.

4 Related Work

Prior recommendation systems use item IDs, sequential architectures, and multi-modal representations, while semantic item tokenizers replace item IDs with discrete semantic tokens. Tlow addresses limitations of existing tokenizer families by combining independent tokenization with transformed embeddings.

  • Recommendation Models: ID-based recommendation evolved from collaborative filtering and matrix factorization to GNN, recurrent, convolutional, and self-attention models.Multi-modal systems additionally incorporate text and image information into item representations.
  • Item Tokenization: TIGER introduced RQ-VAE-based tokenization of item semantic embeddings followed by alignment-based improvements.These methods generate semantic token representations for recommendation.
  • Item Tokenization: RQ-VAE’s hierarchical codebooks reduce decoding efficiency because of codebook correlations.This limitation motivates independent tokenization methods.
  • Item Tokenization: PQ-based independent tokenizers enable parallel decoding, while heuristic parameter-free tokenizers remain limited in performance.The passage contrasts these approaches with hierarchical RQ-VAE tokenization.

5 Conclusion

Tlow transforms semantic embeddings into a standard normal distribution space, enabling dimensional independence and distributional simplicity for more accurate independent tokenization. It improves recommendation performance across general, cross-domain, multi-modal, and cold-start scenarios in offline and online experiments.

  • Tlow transforms semantic embeddings into a standard normal distribution space for more accurate independent tokenization.
  • Tlow improves recommendation performance across general, cross-domain, multi-modal, and cold-start scenarios in offline and online experiments.
Loading 2608.24176v1…