Source-linked AI summary
SA-RSQ: A Versatile Sparse Representation Framework for Multi-modal Recommender Systems
Xiang Wang, Shigang Quan, Tingzhen Chang, Kang Yang, Sitong Chen, Yabo Fan, Xingxing Wang, Zhaodian He
TL;DR
Industrial recommender systems face storage and latency costs from high-dimensional multimodal features, while hard and dense soft quantization impose different quality or budget constraints. SA-RSQ uses sparse Top-K routing with softmax-weighted reconstruction and compact (Index, Probability) tuples, yielding favorable evaluated trade-offs and online lifts of +2.51% in CTR and +3.66% in CPM.
Problem
High-dimensional multimodal features create substantial storage and latency overhead, while existing hard or dense soft quantization methods limit the trade-off between compactness and representation quality.
Method
SA-RSQ selects Top-k code words with masked-softmax weights to store compact (Index, Prob) tuples and enables differentiable weighted reconstruction for fixed supports without a straight-through estimator.
Results
+2.51% in CTR and +3.66% in CPM over the production baseline were reported in a one-week online A/B test, alongside favorable storage–accuracy trade-offs across evaluated 8–48 byte configurations.
Takeaways & Limitations
SA-RSQ shows practical potential for compressing high-dimensional embeddings in industrial recommender systems across the evaluated storage budgets and deployment setting.
Takeaways & Limitations
The evidence is limited by withheld online trial counts, absent significance analysis, and missing p99 latency or throughput measurements.
Abstract
from arXiv · showhide
Deploying high-dimensional multimodal features in industrial recommender systems incurs substantial storage and latency overhead. Hard quantization is compact but introduces boundary distortion, whereas dense soft quantization couples representation quality to the limited storage budget. We propose Sparse Activation-based Residual Soft Quantization (SA-RSQ), which uses Top-K sparse routing and softmax weights to store compact (Index, Probability) tuples. The stored tuples decouple per-item storage from codebook dimensionality; for a fixed selected support, gradients propagate through the routing weights and weighted reconstruction without relying on a straight-through estimator. Experiments on a proprietary food-delivery advertising dataset show favorable reconstruction-performance and CTR trade-offs across storage budgets of 8-48 bytes per item. A preliminary Next-Distribution Prediction study and a one-week online A/B test further demonstrate the practical potential of SA-RSQ, with relative lifts of +2.51% in CTR and +3.66% in CPM.
1 Introduction
Industrial recommender systems need to compress high-dimensional multimodal representations without sacrificing fine-grained semantic discrimination. SA-RSQ uses sparse differentiable routing to provide flexible storage–performance trade-offs, with favorable offline results and online gains.
- 2048D multimodal representations create prohibitive storage and inference-latency costs when deployed across hundreds of millions of items.
- Hard quantization can cause boundary distortion and semantic collapse, while dense soft quantization faces severe fitting pressure under narrow storage budgets.
- SA-RSQ selects Top-K codewords and applies masked-softmax probabilities, storing compact (Index, Prob) tuples while retaining differentiable weighted reconstruction for fixed supports.
- SA-RSQ establishes a sparse semantic space that decouples storage constraints from representation dimensionality and bridges hard compression with dense embeddings.
2 Related Work
Prior work compresses representations through vector and residual quantization, while generative recommendation commonly predicts discrete semantic IDs. SA-RSQ motivates continuous sparse probability distributions to support differentiable, more error-tolerant generation.
- Vector Quantization for Representation Compression: Product Quantization partitions vectors into subspaces, OPQ learns rotations to reduce distortion, and VQ-VAE and RQ-VAE introduce learnable discrete codebooks and residual hierarchies.
- Vector Quantization for Representation Compression: Hard semantic-ID methods achieve extreme compression but suffer item collision and straight-through-estimator optimization instability.
- Semantic IDs for Generative Recommendation: Generative recommendation has used autoregressive Next-Token Prediction to generate RQ-VAE semantic IDs, alongside tokenizers and systems incorporating collaborative signals or end-to-end optimization.
- Semantic IDs for Generative Recommendation: Next-Token Prediction inherits nondifferentiable boundaries and autoregressive error accumulation, motivating Next-Distribution Prediction over continuous sparse codebook probabilities.
3 Methodology
SA-RSQ compresses high-dimensional embeddings with sparse, differentiable residual soft quantization while separating per-item storage from codebook dimensionality. Its optimization and deployment designs support reconstruction, codebook utilization, and downstream gradient alignment under compact storage budgets.
- Residual soft quantization: SA-RSQ uses an Encoder-Decoder architecture with L residual stages and learnable codebooks to progressively quantize item embeddings.The input embedding is mapped to a lower-dimensional latent space, then quantized through successive residual stages.
- Differentiable sparse routing: For fixed Top-k support, residual updates, weighted sums, and routing probabilities are differentiable, avoiding straight-through estimation despite discrete support selection.The support is treated as fixed within each forward pass, allowing gradients through selected logits and weighted reconstruction.
- Differentiable sparse routing: Top-k routing retains the most relevant codewords, applies masked Softmax weights, and reconstructs each residual as a probability-weighted convex combination.Non-Top-k probabilities are zero, while selected routing weights remain differentiable.
- Industrial integration: Index+Prob storage uses 4LK bytes per item, with reported configurations of 8, 32, and 48 bytes under the stated accounting assumptions.The stored entries are active indices and non-zero probabilities; shared codebooks and model parameters are excluded from the per-item budget.
- Optimization objectives: Cosine sparsity annealing starts with dense activation and decays the active budget to the target before inference, balancing semantic exploration and sparsity.This schedule addresses restricted gradient flow caused by applying Top-k masking from the outset.
- Optimization objectives: Training combines global reconstruction, layer-wise residual approximation, geometric orthogonality, and mutual-information objectives to preserve reconstructions and codebook utilization.Orthogonality expands the expressive span of selected codewords, while mutual information balances sharp sample assignments with broad batch-level codeword exploration.
4 Experiments
Experiments evaluate SA-RSQ on proprietary industrial recommendation data across aligned storage budgets, generative recommendation, ablations, and online deployment. Results show favorable storage–accuracy trade-offs, component sensitivity, preliminary NDP gains, and positive online lifts, with deployment and statistical limitations.
- Experimental Setup: The evaluation uses hundreds of millions of proprietary food-delivery advertising items with 2048D MLLM semantics and a shared DIN backbone with 16D final representations.Exact dataset statistics are withheld, and transfer to public benchmarks is not claimed.
- Main Results: SA-RSQ consistently outperforms evaluated baselines under strictly aligned storage budgets, including 64.616 AUC at 8 bytes with L=1, K=4.At 32 bytes, SA-RSQ reaches AUC 64.836; at 48 bytes, it reaches AUC 64.913 and gAUC 63.013.
- Main Results: At 48 bytes, SA-RSQ reduces Reconstruction Loss to 0.1553, while traditional methods have RL > 0.3; lower RL is observed alongside stronger CTR performance.The reported interpretation links improved structural metrics and preserved fine-grained semantics to mitigation of item collision.
- Generative Recommendation: In the preliminary generative study, Next Distribution Prediction improves R@10 from 0.0088 to 0.0096 over Next Token Prediction for SA-RSQ.The authors frame these results as evidence that probabilistic targets are feasible, not as proof that NDP is generally superior.
- Ablation Study: Ablations show that removing residual loss yields AUC 64.622 and RL 0.3212, removing MIR lowers Usage to 24.5% and SC to 0.2446, and removing the Top-k mask lowers AUC to 64.604.Removing Proxy Co-training lowers SC from 0.9179 to 0.1014, while removing Curriculum Learning produces RL 0.4942 and 15.9% Usage.
5 Conclusion
SA-RSQ compresses high-dimensional MLLM embeddings with compact sparse tuples and supports differentiable reconstruction. On a proprietary dataset, it shows favorable trade-offs across evaluated storage budgets, while broader claims remain limited by evaluation scope and uncertainty.
- SA-RSQ compresses high-dimensional MLLM embeddings using Top-k support selection and masked-softmax weights to form compact (Index, Prob) tuples.For a fixed support, weighted reconstruction and routing probabilities remain differentiable without a straight-through estimator.
- SA-RSQ provides favorable reconstruction-performance and CTR trade-offs across the evaluated 8–48 byte configurations.The evaluation uses a proprietary dataset, with a preliminary generative study and one-week online A/B test suggesting practical potential.
- Broader claims are limited by the lack of public data, repeated-run uncertainty, and detailed latency statistics.These limitations constrain assessment of generality, statistical reliability, and system-level efficiency.
- Future work should evaluate Next-Distribution Prediction on public or shareable benchmarks and report uncertainty and system-level latency.The paper also calls for robustness studies across domains before treating distribution prediction as a general generative recommendation paradigm.