PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations

Yuhao Yang, Zhi Ji, Zhaopeng Li, Yi Li, Zhonglin Mo, Yue Ding, Kai Chen, Zijian Zhang, Jie Li, Shuanglong Li, Lin Liu

arXiv:2503.02453v1cs.IRcs.AI

TL;DR

Generative recommendation methods can lose information through separated quantization and sequence-modeling stages, limiting fine-grained modeling relative to dense retrieval. COBRA cascades sparse semantic IDs with dense vectors through end-to-end generation, and experiments report improved accuracy and diversity, including production gains in conversion and ARPU.

  • Problem

    Existing generative recommendation methods suffer information loss and struggle with fine-grained similarity modeling compared with sequential dense retrieval.

  • Method

    COBRA alternates sparse-ID and dense-vector generation, using sparse IDs as conditions for dense representations in a coarse-to-fine process with BeamFusion.

  • Results

    COBRA outperforms state-of-the-art methods in accuracy and diversity, while online tests report a 3.60% conversion increase and 4.15% ARPU increase.

  • Takeaways & Limitations

    The evaluations support COBRA’s effectiveness and practical applicability for large-scale recommendation tasks.

Abstract

from arXiv · show

Generative models have recently gained attention in recommendation systems by directly predicting item identifiers from user interaction sequences. However, existing methods suffer from significant information loss due to the separation of stages such as quantization and sequence modeling, hindering their ability to achieve the modeling precision and accuracy of sequential dense retrieval techniques. Integrating generative and dense retrieval methods remains a critical challenge. To address this, we introduce the Cascaded Organized Bi-Represented generAtive retrieval (COBRA) framework, which innovatively integrates sparse semantic IDs and dense vectors through a cascading process. Our method alternates between generating these representations by first generating sparse IDs, which serve as conditions to aid in the generation of dense vectors. End-to-end training enables dynamic refinement of dense representations, capturing both semantic insights and collaborative signals from user-item interactions. During inference, COBRA employs a coarse-to-fine strategy, starting with sparse ID generation and refining them into dense vectors via the generative model. We further propose BeamFusion, an innovative approach combining beam search with nearest neighbor scores to enhance inference flexibility and recommendation diversity. Extensive experiments on public datasets and offline tests validate our method's robustness. Online A/B tests on a real-world advertising platform with over 200 million daily users demonstrate substantial improvements in key metrics, highlighting COBRA's practical advantages.

1 Introduction

Generative recommendation methods offer direct item prediction but can lose fine-grained information compared with dense retrieval. COBRA addresses this gap by cascading sparse IDs and dense vectors, with coarse-to-fine generation and BeamFusion for flexible inference.

  • Generative models directly predict target items from user behavior sequences, extending sequential recommendation beyond ranking-based methods.
  • COBRA combines sparse IDs for coarse semantic information with dense vectors for fine-grained detail in a unified generative framework.Its decoder sequentially predicts the sparse ID followed by the dense vector.
  • During inference, COBRA first generates a sparse ID and then uses it to condition dense-vector generation for more precise recommendations.BeamFusion combines beam search with nearest-neighbor scores to support flexible and diverse inference.
  • COBRA is presented as an end-to-end framework intended to balance recommendation precision and diversity.The contributions include cascaded representation learning, trainable dense representations, coarse-to-fine generation, and BeamFusion.

2 Related Work

Related work spans sequential dense retrieval and generative recommendation. Prior generative methods use semantic IDs, alignment tasks, collaborative signals, or language models to improve item generation.

  • Sequential Dense Recommendation: Sequential dense recommendation models learn dense user and item representations from interaction sequences to capture long-term preferences and short-term dynamics.
  • Generative Recommendation: Generative recommendation models directly generate item identifiers instead of computing ranking scores for every item.
  • Generative Recommendation: LC-Rec aligns semantic identifiers with collaborative filtering through additional alignment tasks.
  • Generative Recommendation: IDGenRec uses large language models to generate unique, semantically rich textual identifiers and reports strong potential in zero-shot settings.
  • Generative Recommendation: ColaRec aligns content-based semantic spaces with collaborative interaction spaces to improve recommendation.

3 Methodology

COBRA is introduced as a generative recommendation framework built around cascaded sparse-dense representations and coarse-to-fine generation. Figure 2 presents its overall architecture.

  • COBRA integrates cascaded sparse and dense representations into a generative recommendation framework.
  • The framework uses coarse-to-fine generation to enhance recommendation performance.
  • Figure 2 illustrates the overall COBRA framework.

3.1 Sparse-Dense Representation

COBRA represents each item with both sparse semantic IDs and dense vectors. Sparse IDs provide categorical structure, while trainable dense representations encode fine-grained item information within the cascaded model.

  • Sparse Representation: COBRA generates sparse IDs from item attributes by embedding textual descriptions and applying Residual Quantization.
  • Dense Representation: A Transformer-based text encoder produces trainable dense vectors from flattened item-attribute sentences.The vector is extracted from the output corresponding to the [CLS] token.
  • Decoder Architecture: The COBRA architecture feeds these representations to a Transformer Decoder that alternates between predicting sparse IDs and dense vectors.The predicted outputs are used to compute L_sparse and L_dense.
  • Sparse-Dense Representation: Each item’s cascaded representation combines its sparse ID and dense vector as (ID_t, v_t).
  • Sparse-Dense Representation: Sparse IDs provide discrete categorical constraints, while dense vectors preserve continuous feature resolution.

3.2 Sequential Modeling

COBRA factorizes next-item generation into sparse ID prediction followed by dense-vector prediction, using cascaded representations to combine categorical and fine-grained information. A Transformer models these paired representations and sequential context.

  • Sequential Modeling: COBRA predicts the next item’s sparse ID and dense vector in separate stages, combining categorical information with fine-grained details.The dense-vector stage is conditioned on the generated sparse ID and interaction history.
  • Sequential Modeling: Each historical item is represented by a sparse ID and dense vector, which together form the Transformer’s cascaded input sequence.The sequence is augmented with item position and type embeddings to model context.
  • Sequential Modeling: Given the interaction history, the Transformer decoder produces contextualized representations used to predict the next sparse ID and dense vector.Sparse-ID logits are derived from the decoder outputs, while dense-vector prediction uses the sequence with the next sparse-ID embedding appended.

3.3 End-to-End Training

COBRA trains sparse-ID and dense-vector prediction jointly with a composite end-to-end objective. The dense loss refines vectors to distinguish similar from dissimilar items while the combined loss balances both prediction tasks.

  • End-to-End Training: COBRA’s composite loss combines sparse-ID prediction loss and dense-vector prediction loss for joint end-to-end optimization.The overall objective is L = Lsparse + Ldense.
  • End-to-End Training: The sparse-ID loss trains the model to predict the next sparse ID from the historical interaction sequence.It is defined over the interacted item’s ground-truth sparse ID and the set of all sparse IDs.
  • End-to-End Training: The dense-vector loss refines predicted vectors by encouraging similarity to positive-item vectors and discrimination from vectors of other batch items.Cosine similarity measures alignment between predicted and ground-truth dense vectors.
  • End-to-End Training: End-to-end training dynamically refines dense vectors for the recommendation task while jointly optimizing sparse and dense representations.The dual objective is described as guiding dense-vector refinement with sparse IDs.

3.4 Coarse-to-Fine Generation

During inference, COBRA uses a coarse-to-fine cascade: beam search generates sparse IDs, dense vectors refine each beam, and ANN retrieves candidate items. BeamFusion combines beam and similarity scores before selecting the top K recommendations.

  • Coarse-to-Fine Generation: The coarse-to-fine process captures both categorical aspects and fine-grained details of user-item interactions through sequential sparse-ID and dense-vector generation.The two stages are explicitly presented as sparse-ID generation followed by dense-vector refinement.
  • Sparse ID Generation: COBRA first uses BeamSearch on the sparse-ID distribution to generate the top M sparse IDs for the next item.Each generated ID receives a beam score and is appended to the cascaded sequence.
  • Dense Retrieval: For each generated sparse ID, COBRA produces a corresponding dense vector and applies Approximate Nearest Neighbor search to retrieve candidate items.ANN retrieves the top N items associated with each beam.
  • BeamFusion: BeamFusion creates a globally comparable score that reflects differences across sparse IDs and fine-grained differences among items sharing an ID.The mechanism is designed to balance precision and diversity during retrieval.
  • BeamFusion: COBRA ranks all candidate items by BeamFusion Score and selects the top K items as final recommendations.The final recommendation set is defined by the TopK operation over candidate scores.

4 Experiment

COBRA is evaluated on public recommendation benchmarks using standard ranking metrics and comparisons with diverse baseline methods. It consistently outperforms the baselines, including TIGER, across the reported datasets and metrics.

  • Datasets and Evaluation: COBRA is evaluated on Beauty, Sports and Outdoors, and Toys and Games subsets of the Amazon Product Reviews dataset.Items are represented using attributes including title, price, category, and description, with 5-core filtering applied.
  • Datasets and Evaluation: The evaluation uses Recall@K and NDCG@K at K=5 and K=10 to measure recommendation accuracy and ranking quality.These metrics assess relevant-item retrieval and ranking order.
  • Baselines: COBRA is compared with sequential, generative, convolutional, recurrent, gated, and self-attentive recommendation methods.The baselines include P5, Caser, HGN, GRU4Rec, SASRec, FDSA, BERT4Rec, and S3-Rec.
  • Results: On Beauty, COBRA achieves Recall@5=0.0537 and Recall@10=0.0725, exceeding TIGER by 18.3% and 11.9%, respectively.These are the reported public-dataset improvements for the two Beauty recall metrics.
  • Results: On Sports and Outdoors, COBRA records Recall@5=0.0305 and NDCG@10=0.0215, outperforming TIGER by 15.5% and 18.8%, respectively.The cited results cover one recall metric and one ranking metric.
  • Results: On Toys and Games, COBRA attains Recall@10=0.0462 and NDCG@10=0.0515, surpassing TIGER by 24.5% and 19.2%, respectively.These values are reported as improvements over TIGER on the corresponding metrics.

4.2 Industrial-scale Experiments

Industrial-scale experiments evaluate COBRA and its variants on a large advertising dataset, including component analyses and embedding visualizations. COBRA outperforms its variants, while BeamFusion becomes increasingly advantageous as recall size grows.

  • Datasets: The Baidu Industrial Dataset contains five million users and two million advertisements across list-page, dual-column, and short-video recommendation scenarios.User interactions and advertising content are represented through attributes such as titles, industry labels, brands, and campaign text.
  • Baselines: The industrial ablations remove sparse IDs, dense vectors, or BeamFusion to isolate their contributions.COBRA w/o ID uses only dense vectors, COBRA w/o Dense uses only sparse IDs, and COBRA w/o BeamFusion removes the fusion module.
  • Embedding Analysis: COBRA’s dense embeddings show intra-ID cohesion and inter-ID separation, while COBRA w/o ID shows weaker category separation.The difference matrix quantifies the enhancement when sparse IDs are incorporated.
  • Embedding Analysis: A t-SNE visualization of 10,000 randomly sampled advertisement embeddings shows distinct clustering centers for various IDs.The visualization reports category-based clustering in a two-dimensional space.
  • Results: At K=500, COBRA achieves Recall@500=0.3716, a 42.2% improvement over COBRA w/o Dense.This comparison highlights the reported benefit over the sparse-ID-only variant.
  • Results: At K=800, COBRA attains Recall@800=0.4466, improving 43.6% over COBRA w/o ID and 36.1% over COBRA w/o BeamFusion.The comparison covers both the dense-only ablation and the ablation without BeamFusion.
  • Results: As recall size K increases, the advantages associated with BeamFusion become increasingly evident in practical industrial recall systems.The passage connects the trend to BeamFusion’s reported effectiveness in industrial recall.
  • Ablation Results: Removing sparse IDs reduces recall by 26.7%–41.5%, removing 3-level semantic IDs by 30.3%–48.3%, and removing BeamFusion by 27.5%–36.1%.The reported declines identify the measured contribution ranges for semantic categorization, fine-grained modeling, and BeamFusion.

4.3 Further Analysis

COBRA’s further analyses examine its learned advertisement representations, the accuracy–diversity trade-off in BeamFusion, and real-world performance. The results show category-structured embeddings, tunable retrieval behavior, and gains in online business metrics.

  • Representation Learning: COBRA’s dense embeddings show intra-ID cohesion and inter-ID separation, while removing sparse IDs weakens category separation.The comparison indicates that sparse IDs contribute to preserving semantic consistency while dense embeddings capture detailed item characteristics.
  • Representation Learning: 10,000 sampled advertisements form distinct category clusters in t-SNE visualizations, indicating strong within-category cohesion.The observed clusters correspond primarily to different advertisement categories.
  • BeamFusion Analysis: Increasing BeamFusion’s τ generally decreases diversity, with an optimal balance at τ=0.9 and ϕ=16.Diversity is measured by the number of different IDs among recalled items, allowing practitioners to adjust accuracy–diversity emphasis.
  • Online Results: COBRA increased conversion by 3.60% and ARPU by 4.15% in an online A/B test on 10% of user traffic.The test used the Baidu Industrial Dataset in January 2025, with conversion and ARPU as primary metrics.

5 Conclusions

COBRA integrates cascaded sparse and dense representations through coarse-to-fine generation. Experiments on public and industrial datasets, together with online A/B tests, support improvements in accuracy, diversity, and practical applicability.

  • 5 Conclusions: COBRA integrates sparse IDs and dense vectors through coarse-to-fine generation to improve recommendation accuracy and diversity.It first generates a sparse ID capturing an item’s categorical essence, then refines it with a dense vector.
  • 5 Conclusions: Extensive evaluations on public and industrial datasets, including online A/B tests, confirm COBRA’s effectiveness and practical applicability.The conclusion characterizes COBRA as a robust solution for large-scale recommendation tasks.
Loading 2503.02453v1…