Source-linked AI summary
M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems
Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang
TL;DR
Industrial recommender systems need one foundation model that can cover open-ended domains and tasks without violating real-world efficiency constraints. M6-Rec builds on M6 by expressing behavior and recommender tasks as text-based language understanding or generation, then adds parameter-efficient adaptation and deployment optimizations. It supports zero/few-shot recommendation across varied use cases and is deployed on cloud servers and edge devices.
Problem
Industrial recommender systems use separate algorithms across diverse domains and tasks, motivating a unified model that supports open-ended downstream settings under efficiency constraints.
Method
M6-Rec extends the M6 pretrained model by representing user behavior as text, converting tasks to language understanding or generation, and adding efficient adaptation and deployment techniques.
Results
M6-Rec performs well across tasks and domains including retrieval, ranking, zero-shot recommendation, explanation generation, personalized content creation, and conversational recommendation, with deployment on cloud and edge devices.
Takeaways & Limitations
A single generative pretrained model can unify recommender-system subtasks and support open-domain, zero-shot recommendation within cloud and edge deployments.
Takeaways & Limitations
The paper primarily considers text, while extending the framework to multimodal settings remains future work.
Abstract
from arXiv · showhide
Industrial recommender systems have been growing increasingly complex, may involve \emph{diverse domains} such as e-commerce products and user-generated contents, and can comprise \emph{a myriad of tasks} such as retrieval, ranking, explanation generation, and even AI-assisted content production. The mainstream approach so far is to develop individual algorithms for each domain and each task. In this paper, we explore the possibility of developing a unified foundation model to support \emph{open-ended domains and tasks} in an industrial recommender system, which may reduce the demand on downstream settings' data and can minimize the carbon footprint by avoiding training a separate model from scratch for every task. Deriving a unified foundation is challenging due to (i) the potentially unlimited set of downstream domains and tasks, and (ii) the real-world systems' emphasis on computational efficiency. We thus build our foundation upon M6, an existing large-scale industrial pretrained language model similar to GPT-3 and T5, and leverage M6's pretrained ability for sample-efficient downstream adaptation, by representing user behavior data as plain texts and converting the tasks to either language understanding or generation. To deal with a tight hardware budget, we propose an improved version of prompt tuning that outperforms fine-tuning with negligible 1\% task-specific parameters, and employ techniques such as late interaction, early exiting, parameter sharing, and pruning to further reduce the inference time and the model size. We demonstrate the foundation model's versatility on a wide range of tasks such as retrieval, ranking, zero-shot recommendation, explanation generation, personalized content creation, and conversational recommendation, and manage to deploy it on both cloud servers and mobile devices.
1 INTRODUCTION
M6-Rec explores a unified foundation model for open-ended recommender-system domains and tasks, addressing the challenge of broad coverage under strict efficiency constraints. It builds on M6, represents behavior as text, adapts through language understanding or generation, and uses efficient deployment techniques.
- Motivation: Industrial recommender systems span diverse domains and tasks, yet typically use separate algorithms and systems for each one.Examples include retrieval, ranking, explanation generation, trend forecasting, and AI-assisted content production.
- Motivation: A recommender foundation model must support distinct domains and tasks while meeting strict inference-speed, memory, and storage constraints.
- Approach: M6-Rec converts recommender tasks into language understanding or generation and represents anonymous user behavior as plain text.The text-based representation supports sample-efficient adaptation by leveraging M6’s pretrained ability.
- Approach: Option tuning replaces fine-tuning, uses negligible task-specific parameters, and can outperform fine-tuning when combined with adapter tuning despite tuning only 1% of parameters.The approach leaves pretrained parameters unchanged and supports serving multiple tasks with one model.
- Deployment: M6-Rec combines late interaction, parameter sharing, pruning, and early exiting to reduce inference cost and model size for real-world deployment.
- Scope and deployment: The framework unifies recommender subtasks from content supply, delivery, and presentation, supports zero/few-shot learning, and is deployed on cloud and edge devices.
2 RELATED WORK
Prior recommendation pretraining mainly targets transferable behavior modeling or narrow scoring applications. M6-Rec addresses these limits by using web-pretrained language models, text rather than item IDs, and generative modeling for broader domains and tasks under efficiency constraints.
- Limitations of prior work: Existing recommendation pretraining mainly focuses on transferable user modeling and has three identified limitations.
- Limitations of prior work: Prior approaches often neglect non-behavior web corpora, whereas M6-Rec uses M6’s web-pretrained knowledge to support zero-shot recommendation.The paper gives recommending turkeys around Thanksgiving without related click behavior as an example.
- Limitations of prior work: Using item texts instead of item-ID embeddings allows open-domain recommendation with unseen target-domain items.
- Limitations of prior work: Earlier approaches typically handle scoring tasks, while M6-Rec’s generative language-model foundation also supports generation tasks such as dialog-based recommendation.
- Language-model foundations: Language-model foundation research has emphasized scaling model size and unifying modalities, while combining language models with user behavior data remains less developed.
- Efficient inference: Late interaction divides computation into a cached deep stage and a real-time shallow stage, and M6-Rec extends it from two coarse entities to multiple finer-grained segments.
- Efficient deployment: Existing tiny language models often exceed 10M parameters, whereas the paper estimates about 2M are needed to avoid degrading mobile-user experience.
3 METHOD: M6-REC
M6-Rec converts recommender-system tasks and user behavior into language understanding or generation problems, adapting an industrial pretrained language model for diverse use cases. Option tuning and deployment-oriented architectural techniques support efficient adaptation and inference.
- 3.1 M6: An Industrial-Strength Backbone: M6-Rec uses M6, a visual-linguistic pretrained model, as its backbone for recommendation tasks.The selected M6 backbone supports Chinese and English, multimodal features, and deployment in Alibaba’s ecosystem.
- 3.2 Behavior Modeling as Language Modeling: User behavior is represented as natural-language text, and downstream recommendation tasks are converted into language understanding or generation.The approach uses plain-text descriptions of user attributes, past behaviors, candidate items, and interactions.
- 3.2 Behavior Modeling as Language Modeling: For scoring tasks, M6-Rec summarizes the input at the [EOS] position and feeds that vector to a task-specific softmax classifier.CTR prediction uses a two-class classifier trained with cross-entropy loss.
- 3.2 Behavior Modeling as Language Modeling: The same text-generation framework supports explanations, personalized product design, personalized search-query generation, and conversational recommendation.Explanation generation conditions on text through “because,” while other tasks fill structured blanks or continue speaker-marked dialogue.
- 3.2 Behavior Modeling as Language Modeling: M6-Rec also supports zero-shot scoring by comparing the likelihood of alternative events described in natural language.This avoids requiring a trained classification layer for zero-shot comparisons.
- 3.4 Option Tuning for Efficient Adaptation: Option-adapter tuning augments option tuning with adapters and reportedly outperforms fine-tuning while tuning far fewer parameters.Option tuning addresses slow convergence, while multi-segment late interaction is designed for reduced-capacity, low-latency adaptation.
4 EXPERIMENTS
M6-Rec is evaluated across ranking, retrieval, explanation, content creation, conversational recommendation, and deployment-oriented settings. Across these experiments, it outperforms conventional or baseline methods, supports unseen feature values, and demonstrates online gains and edge deployment.
- M6-Rec outperforms conventional methods in various recommender-system stages.
- Ranking: M6-Rec outperforms DIN on the AlipayQuery and TaoProduct CTR datasets.Both DIN and M6-Rec use user behavior sequences and text features, while DIN additionally uses item IDs.
- Mobile deployment: M6-Edge outperforms public tiny language models of similar scale on CLUE, and an Alipay edge ranker built on it increases user clicks by around 0.4%.The edge ranker was successfully deployed in Alipay.
- Retrieval: M6-Rec outperforms TwinBERT and YouTubeDNN for mini-app retrieval, including cases with feature values unseen during training.The ID-based YouTubeDNN fails on such cases, while M6-Rec still performs well.
- Retrieval: Over 1.0% relative online CTR improvement was achieved when M6-Rec replaced a TwinBERT-like mini-app retrieval baseline.The model was fully deployed in Alipay’s system since July 2021.
- Generation: M6-Rec performs better than baselines on explainable recommendation and produces useful search queries, personalized product titles, and conversational responses.The reported gains include better explainability and text quality, useful query mining, personalized content supply, and higher lexical diversity with responses closer to ground truth.
4.2 Zero-Shot Recommendation
M6-Rec is evaluated for zero-shot recommendation across three domains and can also match a traditional ID-based ranker after fitting language loss on few samples.
- M6-Rec performs zero-shot ranking on three datasets from different domains.The evaluation expresses recommendation events in natural language.
- After fitting language loss on a few samples, M6-Rec matches a traditional ID-based ranker trained on a million samples.
4.3 Effects of Late Interaction
Multi-segment late interaction reduces online computation for real-time CTR prediction by caching results from the model’s early layers while retaining predictive performance close to a small student model.
- Caching results from the first L′ = 21 layers leaves only the last L−L′ = 3 layers to compute per request.
- Multi-segment late interaction provides latency similar to a distilled 3-layer student model with much less predictive-performance loss.The method is applied to CTR prediction, where low-latency real-time inference is critical.
4.4 Effects of Option-Adapter Tuning
The deployment experiments examine parameter-efficient tuning and model distillation for constrained hardware. Option-adapter tuning exceeds full-model fine-tuning while tuning only 1% of parameters, and M6-Edge supports an edge-ranker deployment with click gains.
- Motivation: Parameter-efficient tuning is important for mobile deployment because one extra MB can harm user experience and shared models can serve many tasks.
- Option-adapter tuning: Adding soft options improves prompt-tuning convergence, while option-adapter tuning outperforms full-model fine-tuning despite tuning only 1% of parameters.
- Edge deployment: M6-Edge is a distilled tiny language-model foundation for edge rankers on users’ mobile phones.
- Edge deployment: An edge ranker built on M6-Edge outperforms public tiny language models of similar scale on CLUE and increases user clicks by around 0.4% in Alipay.
5 CONCLUSION
M6-Rec unifies various industrial recommender-system tasks and generalizes to open-ended domains while supporting zero-shot learning. The paper identifies multimodal extension as a future direction.
- M6-Rec unifies various tasks in an industrial recommender system.
- M6-Rec generalizes to open-ended domains.
- M6-Rec is capable of performing zero-shot learning.
- Extending the framework to multimodal settings is identified as a future direction.
A.1 Datasets
The appendix describes datasets collected from real-world Alipay and Taobao recommender systems, with task-specific temporal splits and filtering or sampling choices.
- The appendix reports statistics for datasets collected from the authors’ real-world systems.The listed datasets include TaoProduct, AlipayQuery, and AlipayMiniApp.
- TaoProduct: TaoProduct comes from Taobao’s ranking-stage product recommender and uses days one through seven for training and validation, with day eight as test.Negative samples are down-sampled by a factor of ten, and infrequent items are filtered out.
- AlipayQuery: AlipayQuery comes from Alipay’s search-query ranking stage and uses the same days-one-to-seven training-validation and day-eight test split.Negative samples are down-sampled by a factor of ten, and infrequent queries and mini-apps are filtered out.
- AlipayMiniApp: AlipayMiniApp comes from Alipay’s mini-app retrieval stage and uses thirty days of sampled training-validation data followed by seven sampled test days.Rare or noisy items are filtered out, and the time gap leaves many test items unseen during training.
A.2 Hyper-parameters
The hyper-parameter appendix specifies sequence, adapter, prompt, and batch-size settings, while noting different learning-rate needs for fine-tuning and prompt-like tuning. Training uses four machines with eight V100 GPUs each and reduced data subsets for several datasets.
- Prompt-like tuning requires a larger learning rate than fine-tuning to achieve optimal performance.The passage states that this rate is at least higher, but does not provide the full value.
- M6-Rec is trained on four machines equipped with eight Nvidia Tesla V100 GPUs each.Because dense-model training is slower than sparse shallow baselines, M6-Rec uses only a small subset of training data on three datasets.