Source-linked AI summary
QARM: Quantitative Alignment Multi-Modal Recommendation at Kuaishou
Xinchen Luo, Jiangxia Cao, Tianyu Sun, Jinkai Yu, Rui Huang, Wei Yuan, Hezheng Lin, Yichen Zheng, Shiyao Wang, Qigen Hu, Changqing Qiu, Jiaqi Zhang, Xu Zhang, Zhiheng Yan, Jingming Zhang, Simin Zhang, Mingxing Wen, Zhaojie Liu, Kun Gai, Guorui Zhou
TL;DR
Cached pre-trained multi-modal representations are added as fixed inputs to recommendation models, while the two-stage approach avoids directly adding a large multi-modal model to massive online recommendation services. QARM aligns representations to downstream tasks and quantizes them as semantic IDs; online tests report gains in advertising and shopping.
Problem
The common pretrain-and-cache approach supplies fixed multi-modal representations to downstream recommendation models rather than training them end-to-end with recommendation interactions.
Source: §1 · p. 2Method
QARM fine-tunes multi-modal representations for downstream tasks and converts them into VQ or RQ quantitative codes for recommendation features.
Source: §1 · p. 2, §2.3 · p. 4Results
Online tests report advertising Revenue +9.704%/+3.147% and +9.555%/+1.950%, and online-shopping GMV +2.296%/1.568%.
Source: §4.3 · p. 7Takeaways & Limitations
QARM has been deployed at Kuaishou since March 2024, supporting services that serve 400 million users every day.
Source: §6 · pp. 8–9
Abstract
from arXiv · showhide
In recent years, with the significant evolution of multi-modal large models, many recommender researchers realized the potential of multi-modal information for user interest modeling. In industry, a wide-used modeling architecture is a cascading paradigm: (1) first pre-training a multi-modal model to provide omnipotent representations for downstream services; (2) The downstream recommendation model takes the multi-modal representation as additional input to fit real user-item behaviours. Although such paradigm achieves remarkable improvements, however, there still exist two problems that limit model performance: (1) Representation Unmatching: The pre-trained multi-modal model is always supervised by the classic NLP/CV tasks, while the recommendation models are supervised by real user-item interaction. As a result, the two fundamentally different tasks' goals were relatively separate, and there was a lack of consistent objective on their representations; (2) Representation Unlearning: The generated multi-modal representations are always stored in cache store and serve as extra fixed input of recommendation model, thus could not be updated by recommendation model gradient, further unfriendly for downstream training. Inspired by the two difficulties challenges in downstream tasks usage, we introduce a quantitative multi-modal framework to customize the specialized and trainable multi-modal information for different downstream models.
1 INTRODUCTION
Kuaishou’s short-video and live-streaming content is difficult to represent with item IDs alone, while directly adding large multi-modal models is impractical for massive recommendation traffic. QARM aligns multi-modal representations with downstream business data and converts them into learnable semantic IDs.
1 INTRODUCTION
- Kuaishou’s short-video and live-stream content is difficult to understand through item IDs alone, while directly adding large multi-modal models is impractical for massive online recommendation requests.A two-step pretraining-and-cache approach is used to alleviate computation pressure.Source: §1 · pp. 1–2
- The common two-step deployment caches pre-trained multi-modal representations as fixed downstream inputs, leaving a gap between self-supervised image-text objectives and interaction-based recommendation supervision.The paper frames these as representation unlearning and representation unmatching, respectively.Source: §1 · p. 2
- QARM fine-tunes the pre-trained multi-modal model with business-specific item relationships to align its representations with real user-item interaction patterns.Shopping may benefit from relationships between different product categories, while usual short-video recommendation emphasizes relationships between items in the same category.Source: §1 · p. 2
- Its quantitative code mechanism turns fine-tuned representations into Semantic IDs whose embeddings can be optimized end-to-end with real user-item interaction data.The paper describes using residual K-means to obtain a codebook, then measuring representations with the frozen codebook to produce Semantic IDs.Source: §1 · pp. 2–3
- QARM’s online tests report advertising Revenue +9.704% and online-shopping GMV +2.296%; it has supported Kuaishou services serving 400 million users daily since March 2024.These are the overall gains and deployment scale reported among the paper’s contributions.Source: §1 · p. 3
2 METHODOLOGY
Industrial recommendation separates retrieval from ranking, while QARM addresses mismatched multimodal representations by aligning them to downstream behavior and converting them into learnable codes. The codes can serve as item-, user-, and target-aware features in recommendation models.
2.1 Background of Industrial Recommender
- Retrieval models find a small candidate set from a billion-item pool, then more complex ranking models score candidates to select items.The section describes User2Item and Item2Item retrieval designs and a multi-task ranking paradigm.Source: §2.1 · p. 3
2.2 Item Alignment of QARM
- QARM trains a multimodal-only item-to-item alignment model on pairs derived from retrieval models, using their learned knowledge to reflect downstream business characteristics.Pairs include targets linked to similar items from a user's latest 50 positive clicks and stable similar pairs from an Item2Item model.Source: §2.2 · p. 3
2.3 Quantitative Code of QARM
- QARM converts aligned item representations into VQ top-k codes or RQ residual codes, enabling multimodal information to be represented with ID-style features.VQ uses aligned item representations as its codebook; RQ uses K-means codebooks in cascading levels.Source: §2.3 · p. 4
2.4 Usage of QARM
- The codes become learnable item embeddings, describe users through recent positive-interaction sequences, and support target-item-aware matching sequences for ranking.The model combines item-side, user-side, target-aware cross features, and other features for multi-task prediction.Source: §2.4 · p. 5
3 SYSTEM DEPLOYMENT
QARM operates within Kuaishou’s distributed training system as part of feature engineering, preparing newly uploaded short videos as ID-based codes for training and serving.
3 SYSTEM DEPLOYMENT
- QARM preprocesses newly uploaded short-video information into ID-based codes; daily interaction pairs and labels update model parameters offline before real-time synchronization to online inference.The offline data engine assembles hundreds of billions of user-item pair features each day.Source: §3 · p. 5
4 EXPERIMENTS
The experiments evaluate QARM in Kuaishou’s Shopping and Advertising services through offline and online tests, item-group analyses, and quantitative-code parameter studies. The reported findings include online business-metric gains and improvements for long-tail items.
4.3 Online A/B Test
- Revenue rose +9.704%/+3.147% and +9.555%/+1.950% across advertising item groups, while online-shopping GMV rose +2.296%/1.568%.The advertising figures distinguish cold-start items from another group; the online results use the “+ VQ & RQ Code” variant.Source: Table 3 · p. 6, Table 4 · p. 6, §4.3 · p. 7
4.2 Offline Performance
- In offline tests, item-alignment representations improved advertising prediction accuracy; VQ or RQ codes improved on alignment alone, and using both codes improved further.The authors describe VQ as using top-K similar-item neighbors and RQ as encoding the representation through a hierarchical residual path.Source: §4.2 · pp. 6–7
4.4 Case Study of QARM
- Long-tail items showed the largest CTR-AUC prediction improvement, while online exposure increased for L1–L2 and was lower for L4–L6.Groups L1 through L6 are ordered from least to most frequently purchased.Source: Figure 5 · p. 6, §4.4 · p. 7, Table 5 · p. 7
- In the mall scenario, item alignment placed two different vests closest together, whereas the original MLLM representations did not measure them correctly.The visualization also compares image-only, image-text, and item-alignment t-SNE representations.Source: §4.4 · pp. 7–8, Figure 6 · p. 7
4.5 Parameter Analysis of Quantitative Code
- The default quantitative-code settings were VQ length K=25, RQ length L=6, and dimension d=64; tested VQ lengths gave relatively stable improvement, and increasing d enhanced performance.The parameter tests varied VQ code length in Shopping and code embedding dimension in Advertising.Source: §4.5 · p. 8, Table 7 · p. 8
5 RELATED WORKS
Prior recommender work incorporated multimodal features through fixed representations, pretraining, and discrete semantic IDs. QARM combines downstream business-specific alignment with quantitative codes for end-to-end recommendation training.
5 RELATED WORKS
- Earlier multimodal recommenders used fixed visual or textual features and affinity graphs, while later work added multimodal pretraining and task alignment.Examples include VBPR [9], LATTICE [33], BM3 [36], DVBPR [11], AlignRec [18], and Sheng et al. [25].Source: §5 · p. 8
- QARM aligns multimodal representations with downstream business-specific item-item relationships during pretraining and uses quantitative codes for end-to-end recommendation training.Source: §5 · p. 8
- Product quantization divides vectors into subvectors for independent quantization, while residual quantization quantizes the residual left by earlier steps.Source: §5 · p. 8
- Semantic IDs can let semantically related items share codes to support recommendation generalization; TIGER uses RQ-VAE and predicts the next item's ID autoregressively.Source: §5 · p. 8
6 CONCLUSION
QARM aligns multimodal representations to downstream item relationships and quantizes them into learnable codes for recommendation training. Experiments report effectiveness across advertising and shopping, and the system serves 400 million users daily at Kuaishou.
6 CONCLUSION
- QARM mines downstream item-item pairs to fine-tune the multimodal model, then quantizes its representations into codes for end-to-end recommendation optimization.The conclusion describes heuristic VQ and RQ code methods for user-side, item-side, and target-item-aware features.Source: §6 · p. 8
- Experiments on Kuaishou advertising and shopping scenarios demonstrate QARM's effectiveness, with analyses finding it more friendly to cold-start and long-tailed items.Source: §6 · p. 8
- Since March 2024, QARM has been deployed at Kuaishou to support services serving 400 million users every day.Source: §6 · pp. 8–9