PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Retrieval with Semantic Tree-Structured Item Identifiers via Contrastive Learning

Zihua Si, Zhongxiang Sun, Jiale Chen, Guozhang Chen, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, Jun Xu, Kun Gai

arXiv:2309.13375v2cs.IR

TL;DR

Recommendation retrieval needs both efficiency over large item pools and effectiveness in selecting candidates. SEATER uses an encoder-decoder with balanced tree-structured identifiers and contrastive learning; experiments report significant outperformance of several state-of-the-art methods across four datasets.

  • Problem

    Recommendation retrieval must efficiently select a small subset from a large item pool while returning effective candidates, and existing generative methods fall short on efficiency and effectiveness for large-scale recommendation.

    Source: §1 · pp. 1–2

  • Method

    SEATER uses an encoder-decoder to generate balanced k-ary tree-structured item identifiers and combines generation with InfoNCE alignment and triplet ranking losses.

    Source: §1 · p. 2

  • Results

    Across three public datasets and an industrial dataset, SEATER significantly outperforms several state-of-the-art dual-encoder, tree-based indexing, and generative methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The balanced tree structure maintains semantic consistency within each level and provides consistent, fast inference across items.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

The retrieval phase is a vital component in recommendation systems, requiring the model to be effective and efficient. Recently, generative retrieval has become an emerging paradigm for document retrieval, showing notable performance. These methods enjoy merits like being end-to-end differentiable, suggesting their viability in recommendation. However, these methods fall short in efficiency and effectiveness for large-scale recommendations. To obtain efficiency and effectiveness, this paper introduces a generative retrieval framework, namely SEATER, which learns SEmAntic Tree-structured item identifiERs via contrastive learning. Specifically, we employ an encoder-decoder model to extract user interests from historical behaviors and retrieve candidates via tree-structured item identifiers. SEATER devises a balanced k-ary tree structure of item identifiers, allocating semantic space to each token individually. This strategy maintains semantic consistency within the same level, while distinct levels correlate to varying semantic granularities. This structure also maintains consistent and fast inference speed for all items. Considering the tree structure, SEATER learns identifier tokens' semantics, hierarchical relationships, and inter-token dependencies. To achieve this, we incorporate two contrastive learning tasks with the generation task to optimize both the model and identifiers. The infoNCE loss aligns the token embeddings based on their hierarchical positions. The triplet loss ranks similar identifiers in desired orders. In this way, SEATER achieves both efficiency and effectiveness. Extensive experiments on three public datasets and an industrial dataset have demonstrated that SEATER outperforms state-of-the-art models significantly.

1 INTRODUCTION

The introduction frames recommendation retrieval as needing both efficiency at large scale and effective user-interest modeling. SEATER combines balanced tree-structured identifiers with contrastive learning, and the paper reports significant gains over several state-of-the-art method families.

1 INTRODUCTION

  • Retrieval must select a small subset from millions of items efficiently, while maintaining effectiveness because ranking depends on retrieved-item quality.The introduction also identifies limits of single-vector interest representations and notes a strong Euclidean-space assumption for ANN inner products may not hold in practice.Source: §1 · pp. 1–2
  • SEATER uses balanced k-ary identifiers and two contrastive tasks to learn token semantics, tree hierarchies, and inter-token dependencies.InfoNCE aligns token embeddings by hierarchical position, while triplet loss ranks similar identifiers in desired orders.Source: §1 · p. 2
  • Across three public datasets and an industrial dataset, SEATER significantly outperforms state-of-the-art dual-encoder, tree-based indexing, and generative methods.The introduction also reports experiments across four datasets validating effectiveness.Source: §1 · p. 2

2 RELATED WORK

Prior recommendation retrieval spans vector-based, tree-indexed, and generative approaches. SEATER focuses on optimizing identifier structure for generative retrieval while addressing inference costs associated with existing approaches.

2 RELATED WORK

  • Dual encoders represent users and items as vectors for ANN search; later work introduced multi-vector user-interest modeling, while TDM and JTM use tree-based indices.RecForest uses multiple trees and transformer-based routing.Source: §2 · p. 2
  • Generative retrieval produces document identifiers, and TIGER applies a T5-based architecture to recommendation, but multiple transformer layers can burden large-scale inference.Prior document methods use hierarchical IDs, titles or substrings, term sets, and n-gram sets as identifiers.Source: §2 · p. 2
  • Unlike recommendation approaches that leave codebook structures unoptimized, SEATER learns identifier hierarchies and relationships through contrastive learning.The related-work discussion frames this as optimizing identifiers in relation to their indices.Source: §2 · p. 3

3 METHOD

The method section covers SEATER’s model architecture, training, and inference stages.

3 METHOD

  • The section elaborates on SEATER’s model architecture.The passage identifies architecture as one of the section’s three areas of focus.Source: §3 · p. 3
  • The section elaborates on SEATER’s training stages.The passage identifies training as one of the section’s three areas of focus.Source: §3 · p. 3
  • The section elaborates on SEATER’s inference stages.The passage identifies inference as one of the section’s three areas of focus.Source: §3 · p. 3

3.1 Overview

SEATER maps a user’s historical interactions to item-identifier tokens. Its encoder-decoder generates identifiers autoregressively, while training combines identifier generation with contrastive tasks.

3.1 Overview

  • Generative retrieval maps a user’s historical item sequence to a sequence of tokens representing candidate item identifiers.The mapping is described as taking interacted items as input and generating candidate identifiers.Source: §3.1 · p. 3
  • The encoder receives user behavior, and the decoder generates the item identifier autoregressively, one step at a time.The model estimates the interaction probability between the user and item.Source: §3.1 · p. 3
  • Uniform-length identifiers encode item information from coarse to fine granularity, with sequence generation and two contrastive tasks optimizing the model and identifiers.The sequence-to-sequence task generates valid identifiers; the contrastive tasks learn token semantics and relationships.Source: §3.1 · p. 3

3.2 Retrieval Model

SEATER encodes a user's interaction history and autoregressively decodes an item identifier. Its balanced k-ary tree assigns equal-length, hierarchically semantic identifiers to items.

3.2.1 Encoder-Decoder Architecture.

  • A Transformer encoder represents the user's behavior sequence, while an autoregressive decoder predicts the item identifier token by token.The model uses one Transformer layer to maintain efficiency in large-scale retrieval.Source: §3.2.1 · p. 3

3.2.2 Item Identifiers.

  • Items are recursively clustered into equal-size k-way groups using constrained k-means on SASRec item embeddings, then assigned tree-node identifiers.Each tree node receives a unique token; leaf tokens correspond one-to-one with items and share their embeddings.Source: §3.2.2 · pp. 3–4
  • Balanced trees give items equal-length identifiers, consistent semantics within each level, and shorter maximum depth than imbalanced trees.The paper links these properties to faster inference and equivalent processing time across items.Source: §3.2.2 · p. 4
  • Collaborative-filtering-informed embeddings make items with similar interactions likely to receive similar identifiers, without extra training cost beyond SEATER.For example, the paper describes similar items sharing an identifier prefix.Source: §3.2.2 · p. 4

3.3 Training

SEATER trains item-identifier generation together with contrastive objectives that encode tree relationships and rank similar identifiers. Its identifier structure is also reported to support faster inference with affordable additional space.

3.3.4 Multi-task Training.

  • End-to-end multi-task training combines generation, alignment, and ranking objectives, with hyperparameters balancing the contrastive tasks.The method also applies L2 regularization to avoid over-fitting.Source: §3.3 · p. 4, §3.3.4 · p. 5

3.3.1 Generation Loss.

  • Teacher-forced sequence-to-sequence cross-entropy trains the model to generate the next interacted item's valid identifier from user history.The generation objective is one task in the broader multi-task scheme.Source: §3.3.1 · pp. 4–5, §3.3 · p. 4

3.3.2 Alignment Loss.

  • The infoNCE alignment objective pulls a token toward its parent representation and pushes it away from unrelated in-batch tokens.The parent token is the positive instance; other batch tokens excluding the token's parent and child are negatives.Source: §3.3.2 · p. 5

3.3.3 Ranking Loss.

  • Identifiers with different shared-prefix lengths represent varying similarity, and triplet loss teaches the model to rank the ground truth above closer-prefix negatives and those above weaker matches.The ranking objective uses a positive margin that adapts to differences in shared-token counts.Source: §3.3.3 · p. 5
  • Table 1 reports superior inference speed for SEATER with affordable additional space, analyzing beam-search time and identifier size.The accompanying complexity discussion states that DSI and NCI inference takes c times as long as SEATER.Source: Table 1 · p. 5, §3.3.3 · p. 5

3.4 Inference

Inference uses constrained beam search in the decoder to retrieve top items while restricting decoding to the identifier tree.

3.4 Inference

  • The decoder uses constrained beam search over tree-based identifiers to retrieve the top n items from the full candidate set.Restricting decoding to the designated prefix tree yields valid identifiers.Source: §3.4 · p. 5
  • The inference target is the top n items drawn from the entire candidate set.Source: §3.4 · p. 5
  • Decoding is constrained to the designated prefix tree so generated identifiers remain valid.Source: §3.4 · p. 5

4 DISCUSSION

SEATER integrates identifier structure into training through contrastive learning and balanced-tree identifiers. The discussion argues this design improves inference efficiency while adding relatively little identifier storage.

4.1 Comparison with Existing Work

  • SEATER trains on identifier structure through two contrastive tasks, unlike prior generative models that considered structure during identifier construction but not training.It reports superior performance with one transformer layer, compared with four for TIGER and six for P5.Source: §4.1 · p. 6

4.2 Efficiency Analyses

  • The discussion says Table 1’s complexity analysis validates that SEATER’s item-identifier structures are more efficient.Source: §4.2 · p. 6
  • Identifier leaves share item embeddings, so only non-leaf tokens add storage; the resulting identifier overhead is significantly smaller than the item embedding table.The paper sets k to 8 or 16 in its experiments.Source: §4.2 · p. 6
  • SEATER uses beam search, and its inference time is reported as superior to other tree-based and generative models.The discussion attributes longer inference for DSI and NCI to imbalanced trees whose maximum identifier lengths can be several times log_k N.Source: §4.2 · p. 6

5 EXPERIMENT

SEATER achieves the best reported performance across four datasets, with relative R@50 improvements on each. The experiments evaluate it against dual-encoder, tree-indexed, and generative baselines using public and industrial data.

5.1 Experimental Setup

  • The study follows standard item-retrieval practices for selecting datasets, baselines, and evaluation metrics.Source: §5.1 · p. 6

5.1.1 Dataset.

  • The evaluation spans Yelp2, Books3, MIND-small, and 0.75 million micro-video users; the industrial dataset is unfiltered and highly sparse.Source: §5.1.1 · p. 6

5.1.2 Evaluation Metrics & Protocal.

  • They report HR, NDCG, and Recall at top 20 or 50, computing NDCG’s ideal gain from all positive items rather than only recalled positives.Source: §5.1.2 · p. 6

5.2 Overall Performance

  • 6.45%, 5.43%, 11.56%, and 25.50% relative improvements in R@50 on Yelp, News, Books, and Micro-Video accompany SEATER’s best performance across all four datasets.Source: Table 3 · p. 7, §5.2 · pp. 7–8

5.1.3 Baseline and Implementation Details.

  • Baselines span dual-encoder, tree-indexed, and generative models; DSI/NCI comparisons omit SEATER’s extra losses and use their identifier structures, while tree-search beam size is 50.SEATER uses one encoder and decoder layer; item-embedding dimensions are 64 across models.Source: §5.1.3 · p. 7

5.3 Ablation Study

The ablations favor both contrastive losses and balanced identifiers, while identifier and model-depth studies show dataset-dependent choices and a branch-number trade-off.

5.3 Ablation Study

  • Removing either Lrank or Lali lowers overall performance; ranking negative samples improves over its omission but remains below Variant (0).The authors say both losses aid the model in learning identifier inter-token relationships and hierarchies.Source: §5.3 · p. 8
  • Compared with Variant (1), DSI- and NCI-identifier variants show a significant decline, with an average performance drop exceeding 10%.The variants retain the other variables, including identifier-building embeddings and model-layer count, for a fair comparison.Source: §5.3 · p. 8

5.4.1 Impact of Different Item Identifiers.

  • With identical item-embedding sources, balanced trees outperform imbalanced trees; SASREC embeddings perform best, while random identifiers perform worst.The comparison was conducted on the News dataset, the only dataset with item textual descriptions.Source: §5.4.1 · pp. 8–9, Figure 3 · p. 9

5.4.2 Effect of Branch Number k

  • Performance peaks at k=8 on Yelp and k=16 on Books; increasing k first shortens identifiers, but larger k later makes beam search harder.The reported decline occurs when identifier length stays relatively unchanged as k continues to increase.Source: §5.4.1 · p. 9, Figure 4 · p. 9

5.5 Analysis on Parameter Count

  • Yelp improves through 3 layers before declining, while Books improves with depth; deeper models add linear computation, making 1 layer a performance-efficiency balance.The experiments compare small-scale dense Yelp with large-scale sparse Books.Source: §5.5 · p. 9, Figure 5 · p. 9

6 CONCLUSION

SEATER combines balanced tree-structured item identifiers with contrastive learning for generative recommendation retrieval. Experiments and complexity analyses support its effectiveness and efficiency for large-scale retrieval.

6 CONCLUSION

  • SEATER consistently outperforms state-of-the-art models across three public datasets and an industrial dataset.The conclusion also reports that time and space complexity analyses validate the model’s efficiency for large-scale retrieval.Source: §6 · p. 9
  • A balanced k-ary tree gives identifier tokens consistent semantic granularity within each level and varying granularity across levels.The paper says this structure allocates semantic space rationally and supports fast inference.Source: §6 · p. 9
  • SEATER uses infoNCE to align token embeddings by hierarchical position and triplet loss to rank similar identifiers in desired orders.The two contrastive tasks capture token semantics, hierarchies, and inter-token relationships.Source: §6 · p. 9

Related papers

Loading 2309.13375v2…