Order-agnostic Identifier for Large Language Model-based Generative Recommendation

Xinyu Lin; Haihan Shi; Wenjie Wang; Fuli Feng; Qifan Wang; See-Kiong Ng; Tat-Seng Chua

Source-linked AI summary

Order-agnostic Identifier for Large Language Model-based Generative Recommendation

Xinyu Lin, Haihan Shi, Wenjie Wang, Fuli Feng, Qifan Wang, See-Kiong Ng, Tat-Seng Chua

arXiv:2502.10833v2Feb 15, 2025cs.IR

TL;DR

Existing item identifiers either lose important collaborative-filtering or semantic information, or suffer from beam-search local optima and inefficient sequential generation. The paper introduces order-agnostic set identifiers and SETRec, which combines CF and semantic tokens with sparse attention and query-guided simultaneous generation; experiments across models and recommendation settings validate its effectiveness and efficiency.

Problem
Existing token-sequence identifiers face beam-search local optima and inefficient sequential generation, while single-token identifiers inadequately capture semantic and collaborative-filtering information.
Method
SETRec represents each item with order-agnostic CF and semantic tokens, using sparse attention for history encoding and query-guided generation to remove token dependencies.
Results
Experiments on T5 and Qwen from 1.5B to 7B across full-, warm-, and cold-start settings validate SETRec's effectiveness, efficiency, generalization ability, and scalability.
Takeaways & Limitations
The results support order agnosticism and multi-dimensional information utilization as principles for item-identifier design in LLM-based generative recommendation.
Takeaways & Limitations
SETRec uses continuous rather than discrete tokens because discretization inevitably suffers information loss, and open-domain recommendation remains future work.

Abstract

from arXiv · show

Leveraging Large Language Models (LLMs) for generative recommendation has attracted significant research interest, where item tokenization is a critical step. It involves assigning item identifiers for LLMs to encode user history and generate the next item. Existing approaches leverage either token-sequence identifiers, representing items as discrete token sequences, or single-token identifiers, using ID or semantic embeddings. Token-sequence identifiers face issues such as the local optima problem in beam search and low generation efficiency due to step-by-step generation. In contrast, single-token identifiers fail to capture rich semantics or encode Collaborative Filtering (CF) information, resulting in suboptimal performance. To address these issues, we propose two fundamental principles for item identifier design: 1) integrating both CF and semantic information to fully capture multi-dimensional item information, and 2) designing order-agnostic identifiers without token dependency, mitigating the local optima issue and achieving simultaneous generation for generation efficiency. Accordingly, we introduce a novel set identifier paradigm for LLM-based generative recommendation, representing each item as a set of order-agnostic tokens. To implement this paradigm, we propose SETRec, which leverages CF and semantic tokenizers to obtain order-agnostic multi-dimensional tokens. To eliminate token dependency, SETRec uses a sparse attention mask for user history encoding and a query-guided generation mechanism for simultaneous token generation. We instantiate SETRec on T5 and Qwen (from 1.5B to 7B). Extensive experiments demonstrate its effectiveness under various scenarios (e.g., full ranking, warm- and cold-start ranking, and various item popularity groups). Moreover, results validate SETRec's superior efficiency and show promising scalability on cold-start items as model sizes increase.

1 INTRODUCTION

LLM-based generative recommendation depends on item identifiers, but existing token-sequence and single-token designs face efficiency, search, or information limitations. SETRec addresses these issues with order-agnostic set identifiers that combine semantic and CF information and support simultaneous generation.

Existing Identifier Limitations: Token-sequence identifiers can lose target items during beam search when low-probability initial tokens are pruned.They also require multiple serial LLM calls for autoregressive generation, reducing efficiency.
Design Principles: The paper proposes integrating semantic and CF information while removing token dependencies through order-agnostic identifiers.These principles target effective recommendation and simultaneous token generation for improved inference efficiency.
SETRec: SETRec represents each item with a set of order-agnostic tokens containing CF and semantic information.Its paradigm is designed to encode user history and generate item identifiers without token ordering dependencies.
SETRec: SETRec uses sparse attention for history encoding and query-guided generation to support simultaneous token generation.The method guides generation for specific information dimensions while retaining access to previous item identifiers.
Evaluation: The work instantiates SETRec on T5 and Qwen from 1.5B to 7B and evaluates it across full-ranking, warm-start, and cold-start settings.The reported experiments assess effectiveness, efficiency, generalization ability, and scalability.

2 PRELIMINARIES

Item identifiers bridge LLM language and item spaces for encoding user history and generating recommendations. Existing token-sequence and single-token identifiers trade off recommendation quality, efficiency, and representation of collaborative and semantic information, motivating order-agnostic identifiers that combine both.

LLM-based Generative Recommendation: LLM-based recommendation tokenizes users’ chronological interaction histories into item identifiers before generating the next item.The tokenizer transforms each historical item, and the recommender encodes the resulting sequence.
Existing Item Identifiers: Token-sequence identifiers represent items as discrete token sequences and generate them autoregressively with beam search.The next identifier is generated token by token from preceding identifier tokens.
Existing Item Identifiers: Token-sequence generation suffers from beam-search local optima and inference inefficiency caused by multiple serial LLM calls.Increasing beam size only slightly improves accuracy while remaining below globally optimal results.
Existing Item Identifiers: Single-token identifiers improve inference efficiency but struggle with sparse-interaction items or overlook collaborative-filtering information when using semantic embeddings.These limitations lead to suboptimal results.
Identifier Design Principles: Order-agnostic identifiers integrate semantic and collaborative-filtering information while eliminating unnecessary token dependencies to alleviate local optima and improve generation efficiency.The proposed set identifier paradigm represents multi-dimensional item information as a set of order-agnostic tokens.

3 SETREC

SETRec implements set identifiers by combining CF and semantic tokens, then uses query-guided simultaneous generation, grounding, and sparse attention to remove unnecessary token dependencies. Its sparse attention also reduces the computational cost of batch generation.

3.1 Order-agnostic Item Tokenization: The semantic tokenizer uses one unified autoencoder to reduce parameters and training instability while preserving semantic information through reconstruction.The unified autoencoder reduces parameters by an approximate ratio of 1/N compared with multiple independent autoencoders.
3.1 Order-agnostic Item Tokenization: SETRec represents each item with one CF embedding and N order-agnostic semantic embeddings collected into dimension-specific token corpora.The CF tokenizer uses a pretrained recommender model, while the semantic tokenizer uses an autoencoder to produce multiple semantic embeddings.
3.2.1 Query-guided Generation: Learnable query vectors guide simultaneous token generation so each generated token aligns with its designated CF or semantic information dimension.The generated tokens form an order-agnostic set identifier rather than a sequential token identifier.
3.2.1 Query-guided Generation: Token set grounding maps independently generated dimension-specific tokens to existing items using token-corpus grounding heads and combines CF and semantic scores.Semantic grounding heads can extend to new items, supporting generalization to items outside the original corpus.
3.2.2 Sparse Attention Mask: Sparse attention makes tokens within the same historical identifier independent while preserving access to tokens from previous identifiers.This removes within-item token dependencies from flattened user histories and supports simultaneous generation.
3.2.2 Sparse Attention Mask: The sparse attention mask reduces batch-generation time complexity from M^3L^2d to M^2L^2d.The reduction comes from avoiding duplicate computations over the shared prefix in the flattened input.

4 EXPERIMENT

The experiments evaluate SETRec against identifier baselines across architectures, model sizes, datasets, components, and recommendation settings. The study is organized around performance, component contributions, scaling, and broader evaluation scenarios.

Research Questions: Experiments compare SETRec with different identifier baselines across multiple LLM architectures and four real-world datasets.The evaluation also examines component effects, model-size scaling, and additional recommendation settings.

4.1 Experimental Settings

The experimental setup uses four recommendation datasets, compares SETRec with single-token and token-sequence baselines, and evaluates implementations across T5 and Qwen configurations. Table 1 reports overall T5 performance and inference time under a stated statistical-testing convention.

4.1.1 Datasets: The datasets comprise Amazon Toys, Beauty, and Sports reviews plus Steam video-game interactions, with textual item metadata and chronologically ordered user interactions.The Amazon datasets focus on specific e-commerce categories, while Steam contains video-game interactions and textual semantic information.
4.1.2 Baselines: SETRec is compared with single-token baselines DreamRec and E4SRec and token-sequence baselines including BIGRec, IDGenRec, CID, SemID, TIGER, and LETTER.The baselines use ID embeddings, human-vocabulary tokens, clustering, hierarchical categories, or quantized semantic information.
4.1.3 Implementation Details: Implementations use T5-small and Qwen2.5, with Qwen evaluated at 1.5B, 3B, and 7B parameters.The paper abbreviates T5-small and Qwen2.5 as T5 and Qwen, respectively.
4.1.3 Implementation Details: Table 1 reports overall T5 performance, marks best and second-best results, tests significance at p-value < 0.01, and measures inference time on one NVIDIA RTX A5000 GPU.Inference time covers all tested users.

4.2 Overall Performance (RQ1)

SETRec outperforms the evaluated identifier baselines across all, warm, and cold settings, while also improving inference efficiency on T5 and maintaining strong results on Qwen.

Overall comparison: Token-sequence identifiers generally outperform single-token identifiers because multiple tokens encode richer item information across dimensions.This pattern holds under all, warm, and cold evaluation settings.
Cold-start performance: Semantic-integrating identifiers generalize better to cold items than identifiers relying only on CF information.CF-based methods depend heavily on substantial interactions, whereas human-vocabulary identifiers can leverage LLM world knowledge.
Overall comparison: SETRec significantly outperforms all baselines across the all, warm, and cold settings on four datasets.The authors attribute this to combining CF and semantic information with order-agnostic identifiers.
Inference efficiency: 15×, 11×, 18×, and 8× speedups are achieved by SETRec on Toys, Beauty, Sports, and Steam, respectively, versus token-sequence identifiers.The improvement comes from simultaneous generation of multiple tokens in one LLM call.
Qwen-1.5B: SETRec consistently outperforms baselines on Qwen-1.5B and achieves an average 20× speedup across Toys and Beauty.On Qwen, SETRec also surpasses its T5 counterpart, especially under the cold setting.

4.3 In-depth Analysis

Component, popularity, inference-weighting, and scaling analyses show that SETRec benefits from semantic and CF information, order-agnostic generation, and larger models particularly for cold items.

Ablation Study: Removing any SETRec component reduces performance under all, warm, and cold settings, validating the contribution of each component.The ablations remove semantic tokens, CF tokens, query vectors, or the sparse attention mask.
Ablation Study: Removing semantic tokens causes the largest degradation on cold items, demonstrating the importance of semantic information and multiple semantic dimensions.For T5, removing CF tokens also worsens cold-item performance, while CF tokens can hurt Qwen cold-item performance.
Item popularity: SETRec outperforms LETTER and E4SRec across all popularity groups, with larger improvements on sparse items.Performance generally declines from the most popular group G1 to the least popular group G4.
Model scaling: SETRec’s cold-item performance continues improving as Qwen scales from 1.5B to 7B, whereas warm-item performance does not improve continuously.The authors describe this as promising cold-start scalability but limited warm-item scalability.
Inference analysis: Semantic scores improve performance especially on cold items when the inference weight β increases above zero.Using only semantic scores at β = 1 remains competitive on warm items.

5 RELATED WORK

Related work places LLM-based recommendation within discriminative and generative research lines, with identifier methods differing in their use of human vocabulary, external tokens, semantic information, and CF information.

LLM-based Recommendation: LLM-based recommendation research includes discriminative methods that assist conventional recommender models.These methods involve LLMs in different stages of recommendation pipelines.
Item identifiers: Generative recommendation identifiers use item titles, descriptions, tags, or external tokens to leverage LLM knowledge and encode item information.External tokens are also motivated by their potential to include hierarchical information.
Item identifiers: Token-sequence identifiers can integrate semantic and CF information but suffer from local optima and inference inefficiency.These limitations motivate approaches that improve generation efficiency.

6 CONCLUSION

The paper concludes that SETRec’s set identifiers combine CF and semantic information without token dependencies, yielding effective, efficient, generalizable, and scalable generative recommendation results.

Conclusion: SETRec implements order-agnostic set identifiers with CF and semantic tokens, sparse attention, and query-guided simultaneous generation.The approach was evaluated across four datasets and varied recommendation scenarios.
Conclusion: Experiments demonstrate SETRec’s effectiveness, efficiency, generalization ability, and scalability across four datasets and various scenarios.The conclusion summarizes the empirical support for the proposed identifier design.
Future work: Future work includes discrete set identifiers and applying SETRec to open-ended recommendation with open-domain user behaviors.These directions extend the method beyond the evaluated settings.