Source-linked AI summary
Generative Retrieval for E-commerce: Jointly Learning Embedding and Codebook with Same Product Cluster
Songtao Fang, Zihao Xu, Shaowei Wei, Jin Zhang, Zhuojun Wang
TL;DR
Existing generative retrieval pipelines separately learn product embeddings and a codebook, which can accumulate representation errors and produce inconsistent IDs within product clusters. The paper jointly trains both components with same-product-cluster supervision. On real-world e-commerce data and multiple retrieval tasks, the method significantly outperforms existing approaches while improving retrieval accuracy and within-cluster product-ID consistency.
Problem
Two-stage generative retrieval can propagate embedding errors into codebook learning and assign inconsistent IDs within the same product cluster.
Method
The method jointly trains the product embedding model and codebook, using same-product-cluster information as additional supervision.
Results
The method significantly outperforms existing approaches on real-world e-commerce data and multiple retrieval tasks.
Takeaways & Limitations
Joint training and cluster supervision together improve retrieval accuracy and product-ID consistency within the same cluster.
Abstract
from arXiv · showhide
With the development of large language models (LLMs), generative retrieval is becoming increasingly important in e-commerce scenarios. Current mainstream approaches typically use a two-stage training strategy: first train a product embedding model, and then learn a codebook that maps embeddings to product IDs. This cascaded approach suffers from two major issues: (1) error accumulation-if the embedding model in the first stage produces biased representations, the codebook in the second stage cannot correct these errors, degrading final retrieval performance; and (2) codebook learning relies solely on product embeddings and lacks modeling of query-to-product and product-to-product interactions. As a result, products belonging to the same cluster may be assigned inconsistent IDs by the codebook, further hurting retrieval accuracy. To address these problems, we propose a novel method that jointly trains the embedding model and the codebook, and incorporates same product cluster information as an additional supervision signal. Experimental results demonstrate that our method significantly improves e-commerce retrieval performance while simultaneously enhancing both embedding and codebook learning.
1 Introduction
Generative retrieval can better interpret complex or implicit-intent e-commerce queries, but existing two-stage product-ID learning accumulates embedding errors and models insufficient interactions. The proposed joint framework adds product-cluster supervision and outperforms existing approaches across multiple e-commerce retrieval tasks.
- Motivation: Generative retrieval uses large generative models to interpret user intent and produce relevant results for complex or implicit-intent queries.Sparse and dense methods can struggle with shallow semantic matching, colloquial queries, semantic variation, and term mismatch.
- Limitations of Existing Methods: Two-stage approaches first learn product embeddings and then derive a codebook mapping embeddings to product IDs, causing error accumulation and insufficient interaction modeling.Embedding inaccuracies can propagate into codebook training, while the codebook relies solely on static product embeddings.
- Proposed Approach: The proposed framework jointly optimizes product embeddings and the codebook under a unified objective.This design addresses error propagation in conventional cascaded pipelines.
- Proposed Approach: Product cluster assignments provide auxiliary supervision that enhances semantic consistency among IDs and improves embeddings for identical products.The framework uses cluster information to align products within the same semantic group.
- Results: The method outperforms existing approaches on multiple e-commerce retrieval tasks.This is reported as an extensive experimental finding in the introduction.
2 Related Work
Sparse and dense retrieval improve retrieval through keyword matching and neural representations, but both can fail to capture complex query intent. Generative retrieval instead uses large language models to better understand such queries.
- Sparse and Dense Retrieval: Sparse retrieval methods such as BM25 use inverted indexes and TF-IDF–style term matching for efficient lookup.They struggle with semantic variation and term mismatch.
- Sparse and Dense Retrieval: Dense retrieval methods such as DPR and ColBERT use neural embeddings and pretrained models to capture semantic relationships.Their fixed features or learned representations can still fail to capture complex query intent.
- Sparse and Dense Retrieval: Large language models excel at understanding complex queries, offering an advantage over sparse and dense retrieval approaches.This motivates generative retrieval for queries whose intent is difficult to represent through fixed features or term matching.
3 Model
The model jointly trains product embeddings and a codebook, using same-product-cluster information to improve semantic consistency. It quantizes embeddings into product identifiers, applies contrastive and reconstruction objectives, and trains an LLM to map queries to those identifiers.
- Product Identifier Training: Products are tokenized into fixed-length discrete identifiers, while same-product clusters provide auxiliary semantic supervision.A cluster can contain functionally or visually identical items, such as different sellers offering the same SKU.
- Product Identifier Training: Residual quantization converts each product embedding into a sequence of code indices selected hierarchically from L codebooks.Each level selects the most similar code embedding for the current semantic residual, producing a quantized identifier.
- Product Identifier Training: The quantized code sequence is decoded to reconstruct the product embedding, with RQ-VAE combining reconstruction and codebook commitment losses.The decoder reconstructs the input embedding from the quantized representation.
- Product Identifier Training: The framework jointly optimizes product embeddings and the codebook under a combined reconstruction, contrastive, and cluster-consistency objective.The model loss combines RQ-VAE loss with InfoNCE and MSE terms, whose weights control contrastive learning and same-product-cluster constraints.
- Product Identifier Training: InfoNCE is applied to both product embeddings and reconstructed embeddings so query-product relevance contributes to joint training.The positive is the query-relevant product, while hard negatives or in-batch negatives provide contrasting products.
- LLM Training: An LLM is trained to map product information and user queries to product-ID token sequences through progressive learning.The first stage aligns product information with newly introduced identifier tokens; the sequence-to-sequence loss then models target ID generation.
4 Experiments
The experiments evaluate the method against sparse, dense, and generative retrieval baselines on large-scale e-commerce data. Results indicate that joint training and product-cluster supervision improve retrieval effectiveness and ID consistency.
- Experimental Setup: 20 million products support embedding and codebook training, while 40 million query–product-ID mappings support LLM training; validation and test sets each contain 10,000 queries.Both evaluation sets are evenly divided between regular and implicit-intent queries.
- Evaluation: The evaluation compares sparse, dense, and generative retrieval models using Recall@K and Average Length of the Shared ID Prefixes among same-cluster products.Recall@K measures relevant items retrieved in the top K, while ALSP evaluates shared prefixes in product IDs.
- Results: The proposed model outperforms sparse and dense retrieval methods on a large-scale real-world dataset, especially for complex or implicit-intent queries.The passage attributes sparse retrieval’s difficulty to keyword matching and dense retrieval’s limitations to its sample construction and inability to leverage general world knowledge.
- Results: Compared with generative retrieval baselines, the method yields higher shared-prefix consistency within product clusters and improves key metrics when cluster supervision is retained.Removing the cluster-based constraint substantially degrades ALSP and other key metrics.
- Analysis: The joint framework addresses bias amplification and information loss associated with embedding-based clustering, while aligning item IDs with semantic product groupings.The reported analysis connects end-to-end training with performance improvements and cluster constraints with more consistent ID representations.
5 Conclusion
The paper concludes that jointly training product embeddings and the codebook with same-cluster supervision improves retrieval accuracy and product ID consistency on real-world e-commerce data.
- Conclusion: Joint training and cluster supervision are both crucial for enhancing retrieval accuracy and improving product ID consistency within the same cluster.The conclusion reports these findings from ablation studies.