PaperlayerView exact revision on arXiv

Source-linked AI summary

REGER: Modeling inter-item Relations with Semantic IDs for Generative Recommendation

Yuxi Huang, Silin Zhou, Chenhao Wang, Zihan Li, Gangming Guo, Pei Ke, Shen Gao, Shuo Shang

arXiv:2610.37584v1cs.IRcs.AI

TL;DR

Generative recommendation models item IDs as discrete token sequences, but semantic relations may not be inherited by a model trained from scratch and collaborative relations remain largely implicit. REGER introduces semantic retrieval pretraining, transition-aware history attention, and hierarchical preference distillation; experiments report consistent gains across six real-world benchmarks.

  • Problem

    GR’s token-level modeling does not directly inherit item semantic relations, and its positive-item next-token supervision leaves collaborative preferences among competing items implicit.

    Source: §1 · p. 2, §3.2 · p. 3

  • Method

    REGER combines semantic-neighbor SID pretraining, SID-boundary-aligned local attention alongside global attention, and hierarchical distillation of teacher preferences over legal next tokens.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Experiments on six real-world benchmarks demonstrate consistent performance gains, while HCPD maintains practical training efficiency.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings support internalizing item knowledge within the generative recommendation model through explicit semantic and collaborative relation modeling.

    Source: §6 · p. 9

    The preference-distribution decomposition assumes a strictly positive item preference distribution over the catalog.

    Source: §4.3.1 · p. 5

Abstract

from arXiv · show

Personalized recommendation depends on learning semantic and collaborative relations among items from online content and user interactions. Generative recommendation (GR) offers a promising paradigm by autoregressively generating discrete semantic IDs (SIDs) of items. However, randomly initialized SID token embeddings and GR models trained from scratch do not directly inherit the semantic relations inherent in items, while inter-item collaborative relations are learned largely implicitly through token-level attention and next-token supervision. To address this limitation, we propose REGER, a unified framework that explicitly incorporates inter-item relations into semantic pretraining, history encoding, and SID generation. We pretrain the GR model through generative semantic retrieval, enabling it to inherit inter-item semantic similarity by learning to generate semantic neighbors' SIDs. We combine global attention with local attention aligned with SID boundaries to capture historical context and directed transitions between adjacent items. We further develop a theoretical framework for hierarchical preference decomposition over the SID prefix tree, proving an exact factorization of collaborative preference distributions and a weighted decomposition of item-level KL divergence into prefix-conditional KL divergences. Building on this analysis, we distill a discriminative teacher's full conditional distributions over all legal next tokens at ground-truth prefixes along the target SID path, jointly optimizing this objective with cross-entropy. Extensive experiments on real-world datasets demonstrate consistent improvements over existing GR baselines and support the effectiveness of the proposed preference distillation.

1 Introduction

Generative recommendation (GR) represents items with semantic IDs, but item-level semantic and collaborative relations remain largely implicit in token-level modeling. REGER explicitly addresses this gap across semantic learning, history encoding, and recommendation generation.

1 Introduction

  • GR autoregressively generates item token sequences, with shared SID prefixes reflecting semantic similarity and facilitating knowledge transfer across items.Residual quantization maps pretrained item representations to compact SIDs.Source: §1 · p. 1
  • Existing approaches improve tokenization, align semantic and collaborative representations, or combine GR with dense retrieval, but the link between item-level relations and discrete-token modeling remains a bottleneck.The cited approaches include co-occurrence-based subwords, hybrid retrieval, GFlowGR, and TopoTok.Source: §1 · pp. 1–2
  • Item text supplies semantic cues, interaction histories supply collaborative signals, and users’ differing strengths of preference across candidates induce ranking relations.GR uses SID tokens rather than complete items as its basic modeling units, leaving these item-level relations to be learned implicitly.Source: §1 · p. 2
  • REGER addresses three GR challenges: inheriting semantic similarity, modeling directed transitions between adjacent items, and representing collaborative preferences among candidate items.The framework targets inter-item relations in semantic learning, history encoding, and recommendation generation.Source: §1 · p. 2
  • REGER combines semantic-neighbor SID generation for pretraining, global and SID-boundary-aligned local attention for history encoding, and hierarchical collaborative preference distillation for SID generation.Its theory factorizes item preferences over legal next tokens in the SID prefix tree and decomposes item-level KL divergence into weighted prefix-conditional terms.Source: §1 · p. 2
  • Across six real-world benchmarks, experiments report consistent gains from explicit inter-item relation modeling and validate HCPD’s effectiveness and efficiency.The introduction identifies these as the framework’s experimental contributions.Source: §1 · p. 2

2 Related Work

Prior work has shaped SID tokenization to encode item relations and introduced explicit relation modeling in recommendation. REGER’s related-work discussion highlights that connecting item-level relations to token-level dependencies remains a challenge for GR.

2.1 Generative Recommendation

  • GR research has incorporated semantic and collaborative signals into SID construction, but semantic organization alone does not ensure that generators learn complete-item semantics and relations.Examples include residual quantization, collaborative alignment, co-occurrence-based tokenization, and contextual token representations.Source: §2.1 · p. 3

2.2 Inter-item Relation Modeling

  • Relation modeling spans attention structures, collaborative supervision, prefix ranking, and reward-based generation, yet item-level relations still connect only implicitly to GR’s discrete-token dependencies.The discussion contrasts GR’s shared SID tokens with continuous item representations and pretrained-LLM text tokens.Source: §2.2 · p. 3

3 Preliminary: Generative Recommendation

The preliminary section describes how items are represented by residual-quantized SIDs and how GR predicts the next item from a user’s history. Its standard cross-entropy objective trains on the positive item’s SID alone.

3.1 Item Tokenization

  • Residual quantization maps continuous item embeddings to discrete code sequences, with shared prefixes and codewords representing semantic relations.The paper adopts an RQ-VAE and appends a token to distinguish items with identical semantic codes.Source: §3.1 · p. 3

3.2 Problem Formulation

  • GR concatenates the chronologically ordered history items’ SIDs and autoregressively predicts the next item’s SID using next-token cross-entropy under teacher forcing.Each prediction conditions on the history and the ground-truth SID prefix.Source: §3.2 · p. 3
  • The standard objective uses the positive item’s SID as its sole target, without explicitly supervising preference relations among competing tokens.Such competitive relations are left to be learned implicitly.Source: §3.2 · p. 3

4 Methodology

REGER combines semantic-neighbor pretraining, global and local history attention, and hierarchical preference distillation to model inter-item relations in generative recommendation. Its training pipeline uses GSR to initialize the generator, then combines HCPD with cross-entropy during recommendation training.

4.1 Generative Semantic Retrieval

  • GSR trains the encoder-decoder to generate a semantic neighbor’s SID from a source item’s SID, using top-k neighbors retrieved by cosine similarity between item text embeddings.Pretraining updates SID token embeddings and the backbone, whose resulting parameters initialize recommendation training.Source: §4.1 · p. 4

4.2 Item Transition Structure Attention

  • ITSA combines token-global heads over the full history with item-local heads that connect each item’s SID to its own tokens and those of its immediate predecessor.The same encoder is used in both training stages, and changing attention masks leaves the backbone’s parameter count unchanged.Source: §4.2 · pp. 4–5

4.3.1 Preference and KL Decomposition.

  • The framework exactly factorizes item preferences into conditional distributions over legal next tokens and decomposes item-level KL into prefix-conditional KL terms weighted by prefix probability.These decompositions motivate conditional preference matching along ground-truth SID paths.Source: §4.3.1 · p. 5

4.3.2 Distillation and Training Objectives.

  • HCPD matches a frozen discriminative teacher’s and generator’s full distributions over legal next tokens at prefixes along the target SID path, jointly optimizing distillation with cross-entropy.The ground-truth-path objective uses observed-target prefix weights and does not generally reproduce the scaled full-tree KL; DuoRec is the adopted teacher.Source: §4.3.2 · pp. 5–6

5 Experiments

Experiments evaluate REGER on Amazon Reviews benchmarks, where it outperforms generative and hybrid baselines and ablations show contributions from GSR, ITSA, and HCPD. Further analyses examine HCPD against alternative objectives, attention patterns, and hyperparameter choices.

5.1 Experimental Setting

  • The evaluation covers Beauty, Sports, and Toys from Amazon Reviews 2014 and Instruments, Scientific, and Games from Amazon Reviews 2023, using 5-core filtering and histories truncated to 20 interactions.Methods are evaluated with Recall@K and NDCG@K for K∈{5,10}, with each test repeated across three random seeds.Source: §5.1 · p. 6

5.2 Main Results

  • REGER outperforms all generative and hybrid baselines across the three Amazon Reviews 2014 datasets and all metrics, and DuoRec in most settings.On Toys, REGER's Recall remains lower than DuoRec's, while its NDCG is comparable or slightly better.Source: §5.2 · p. 6, Table 1 · p. 7

5.3 Ablation Study

  • Removing GSR, replacing ITSA with standard bidirectional attention, or removing HCPD consistently degrades performance across all three datasets.Removing HCPD produces the largest drop, with Recall@5 decreasing by 10.9%–18.0%.Source: §5.3 · pp. 6–7, Table 3 · p. 7

5.4 Comparison of Training Objectives

  • Applied to TIGER, HCPD achieves the best recommendation results across all three datasets and metrics among the compared training objectives.HCPD matches teacher and generator conditional preferences over legal next-token branches at ground-truth prefixes; on Beauty, efficiency comparisons report modest training overhead and unchanged inference speed.Source: §5.4 · pp. 7–8, Table 2 · p. 7, Table 4 · p. 8

5.5 In-depth Analysis

  • On Beauty, REGER's item-pair similarities agree more closely with the semantic reference, while ITSA's attention combines broad history context with stronger attention to the current item and its predecessor.The heatmaps compare the same 160 items in the same order; position 0 in the attention visualization is the most recent item.Source: §5.5 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

5.6 Hyperparameter Analysis

  • Recall@10 peaks at K=10 on all three datasets, while NDCG@10 peaks at λ=0.8 for Beauty and Sports and λ=1.0 for Toys.Selected Local:Global head ratios are 2:4 for Beauty, 5:1 for Sports, and 3:3 for Toys; these outperform all-local and all-global attention.Source: §5.6 · pp. 8–9, Figure 6 · p. 9

6 Conclusion

REGER models inter-item semantic and collaborative relations through semantic pretraining, transition modeling, and preference distillation. Experiments across six real-world benchmarks show consistent gains, and HCPD maintains practical training efficiency.

6 Conclusion

  • REGER explicitly models inter-item semantic and collaborative relations through semantic pretraining, item transition modeling, and collaborative preference distillation.The framework uses an exact factorization of item preference distributions and a weighted decomposition of item-level KL divergence over SID prefix trees to motivate HCPD.Source: §6 · p. 9

A.1 Datasets

The appendix describes preprocessing for six Amazon Reviews datasets and summarizes the recommendation and training-objective baselines used in evaluation.

A.1 Datasets

  • The Amazon Reviews 2014 and 2023 datasets follow released preprocessing pipelines, with each user's latest interaction for testing, second latest for validation, and earlier interactions for training.Table 5 summarizes dataset statistics after preprocessing.Source: §A.1 · p. 10, Table 5 · p. 10

A.2 Baselines

  • The main baselines span discriminative, generative, and hybrid recommendation, including methods that alter tokenization, combine retrieval with generation, or align semantic and collaborative information.Examples include SASRec and DuoRec, TIGER and ActionPiece, and LIGER and MemGen-GR.Source: §A.2 · p. 10
  • Training-objective baselines include token-likelihood and auxiliary-KL methods, adaptive prefix likelihood, and flow-balance objectives for decoding transitions or complete trajectories.Additional Amazon Reviews 2023 baselines include DECOR, Latte, and TopoTok.Source: §A.2 · p. 11

A.3 Implementation Details

Implementation details specify item-text encoding and tokenizer training, followed by shared backbone and optimization settings for baselines and REGER.

A.3 Implementation Details

  • Item text is encoded with Sentence-T5 and reduced from 768 to 128 dimensions using PCA with whitening; an RQ-VAE tokenizer is trained once per dataset, with its SID mapping shared across recommender seeds.The tokenizer trains for up to 20,000 epochs using the default MTGRec configuration.Source: §A.3 · p. 11
  • Reproduced generative baselines use a T5 encoder-decoder, train for up to 200 epochs, and select checkpoints by validation NDCG@10.REGER uses up to 100 epochs of GSR pretraining and up to 200 epochs of joint CE and HCPD recommendation training; the teacher is frozen and K=10.Source: §A.3 · p. 11

B Results on Amazon Reviews 2023

On the larger 2023 benchmarks, REGER achieves the best or second-best results across all metrics, and HCPD improves it even when its teacher underperforms the student. The teacher comparison also shows that a teacher’s relative-preference supervision can help beyond its top-K recommendation performance.

B Results on Amazon Reviews 2023

  • REGER achieves the best or second-best results across all metrics on the larger 2023 benchmarks.The passage refers to Table 6, but its values are not supplied here.Source: B Results on Amazon Reviews 2023 · p. 11
  • HCPD consistently improves REGER even on Scientific, where DuoRec underperforms REGER without HCPD on every metric.The results suggest a teacher’s top-K recommendation performance does not fully reflect the supervisory value of its preference distribution.Source: B Results on Amazon Reviews 2023 · pp. 11–12

C Effect of the Distillation Teacher

  • A weaker SASRec teacher raises Recall@50 across all three datasets but lowers NDCG@5 and Recall@5, whereas DuoRec improves all reported metrics.The authors explain that KL matching may promote relevant candidates the student underestimates, but can also transfer inaccurate teacher preferences that weaken ranking.Source: C Effect of the Distillation Teacher · p. 12
  • The teacher comparison supports the importance of accurately modeling users’ relative preferences among candidate items; the authors identify DuoRec’s contrastive learning as a way to mitigate representation degeneration.These findings are discussed alongside the 2023 results.Source: C Effect of the Distillation Teacher · p. 12

D Theoretical Analysis of HCPD

The proof uses the notation and assumptions of Section 4.3. For Theorem 4.1, it obtains item probabilities by telescoping prefix probabilities from the root to each item’s leaf.

D Theoretical Analysis of HCPD

D.1 Proof of Theorem 4.1

  • For each item, the proof uses root mass 1 and leaf mass P(i) to telescope the prefix-probability product.Source: §D.1 · p. 12

D.2 Proof of Theorem 4.2

Theorem 4.2’s proof groups item-level KL terms by tree edges. The following section relates the full-tree KL decomposition to distillation evaluated only along ground-truth target paths.

D.2 Proof of Theorem 4.2

  • The proof of Theorem 4.2 expands item-level KL and groups its terms by tree edges.The edge probability mass is expressed as the parent-prefix mass times the conditional probability of the edge token.Source: §D.2 · p. 12

D.3 Full-Tree KL and Ground-Truth Path Distillation

  • Teacher-sampled paths provide an unbiased estimator of the scaled item-level KL, while observed-target prefix weights generally differ from the full-tree decomposition.Each instance evaluates conditional KL terms only at prefixes along its ground-truth path.Source: §D.3 · p. 12

D.4 Comparison with TCA-Aux

HCPD’s conditional KL fits teacher preferences over legal next tokens, while TCA-Aux’s full-vocabulary KL also penalizes insufficient legal probability mass. On Beauty, HCPD outperforms the full-vocabulary variant across all six metrics, with its largest reported relative gains at shorter ranking cutoffs.

D.4 Comparison with TCA-Aux

  • At a fixed prefix, HCPD and TCA-Aux use the same teacher distribution, legal-token set, and target prefixes, but TCA-Aux’s auxiliary KL uses full-vocabulary student probabilities.The comparison defines the teacher’s legal-token probabilities and the student’s legal probability mass Z.Source: §D.4 · p. 12
  • TCA-Aux’s fitting terms within the legal set match HCPD’s, but it adds a penalty for insufficient legal probability mass; the objectives are not generally equivalent under fixed reweighting or rescaling.Source: §D.4 · p. 13
  • When teacher and student conditional distributions match but legal probability mass is below one, HCPD’s KL and its logit gradients vanish, whereas TCA-Aux’s KL remains −log Z.At prefixes with only one legal continuation, HCPD’s KL is identically zero.Source: §D.4 · p. 13

E Scope of KL Supervision in HCPD

  • HCPD-Full retains ground-truth distillation paths but computes KL using full-vocabulary student probabilities, extending the teacher distribution by zero outside the legal set.Source: E Scope of KL Supervision in HCPD · p. 13
  • HCPD achieves higher mean performance than HCPD-Full across all six metrics, with relative gains of approximately 3.4% in NDCG@5 and 2.4% in Recall@5.The gaps narrow as recommendation-list length increases, with nearly identical Recall@20.Source: E Scope of KL Supervision in HCPD · p. 13, Table 8 · p. 13

Related papers

Loading 2610.37584v1…