An Unified Search and Recommendation Foundation Model for Cold-Start Scenario

1 INTRODUCTION

• Jointly modeling search and recommendation can capture users’ complete intents across scenarios, while addressing data imbalance, heterogeneous item sets, and negative transfer.The authors note that data from a single domain may not fully reflect users’ intents and that many items are shared between search and recommendation.Source: §1 · p. 1
• LLMs provide domain-invariant query and item text features that the authors use to constrain task divergence and mitigate negative transfer and item heterogeneity.The paper contrasts this use of text features with traditional multi-domain multi-task models.Source: §1 · p. 2
• The proposed foundation model combines ID, LLM text, and sparse features with Aspect Gating Fusion and Domain Adaptive Multi-Task learning.For cold-start scenarios, the authors report offline and online experiments with supervised fine-tuning, and deployment in Alipay’s mobile application.Source: §1 · p. 2

2.1 Problem Formulation

• The foundation model jointly trains search and recommendation tasks, predicting click-through rate and, for search, query-item relevance; new scenarios use pretrained parameters followed by supervised fine-tuning.Recommendation examples use an empty query, and the downstream fine-tuning setup tests different freeze-finetune splits.Source: §2.1 · p. 2

2.2 S&R Multi-Domain Foundation Model

• LLMs extract domain-invariant query and item text features, which are fused with ID and sparse features before domain-adaptive multi-task prediction.The architecture has user-query-item encoding, Aspect Gating Fusion, and Domain-Adaptive Multi-Task components.Source: §2.2 · p. 2

2.2.1 User Query and Item Encoding.

• User behavior is encoded as a chronological sequence of search and recommendation clicks, while query and item representations combine ID, token-level text, and sparse-feature embeddings.LLM encodings are mean-pooled and shared-projected to form domain-invariant query and item text representations, which are useful when new items or queries have few training samples.Source: §2.2.1 · p. 2

2.2.2 Aspect Gating Fusion.

• Aspect Gating Fusion combines ID, text, and sparse-feature representations as a weighted sum, with mean, [CLS]-based, and domain-specific gating strategies described.The gating network supplies the weights used to balance the feature aspects.Source: §2.2.2 · p. 3

2.2.3 Domain Adaptive Multi-Task Learning.

• The Domain Adaptive Layer appends a domain embedding to instance features and applies a domain-specific linear transformation to map inputs into a common vector space.A divergence regularizer constrains distributions across domains; the module then feeds a standard multi-task model to predict CTR and query-item relevance.Source: §2.2.3 · pp. 3–4

2.2.4 Loss of S&R Foundation model.

• The training objective combines CTR prediction loss, query-item relevance loss, and domain-adaptive regularization across the domains and tasks that provide each objective.Each domain calculates either or both task objectives depending on whether its task is search or recommendation.Source: §2.2.4 · p. 4

2.3 Supervised Fine-Tuning Downstream Tasks

• For downstream tasks, the model restores foundation-model parameters, freezes some layers, and fine-tunes the rest, while initializing and fine-tuning a new domain embedding.The paper tests different freeze-finetune splits and compares transfer against training a single-domain model without transfer learning.Source: §2.3 · p. 4

• RQ1 asks whether the joint S&R Multi-Domain Foundation model achieves state-of-the-art performance against other multi-domain and multi-task models.Source: §3 · p. 4
• RQ2 compares LLM-extracted domain-invariant text features and Aspect Gating Fusion with other methods for query and item representations.Source: §3 · p. 4
• RQ3 asks whether the S&R foundation model and supervised finetuning benefit cold-start scenarios.Source: §3 · p. 4

3.1.1 Dataset.

• Experiments use seven Alipay Search Ranking and Query Recommendation datasets, categorized as search, recommendation, or hybrid S/R tasks.Search tasks have explicit queries, recommendation tasks do not, and S/R tasks combine explicit queries with CTR prediction.Source: §3.1.1 · p. 4

3.1.2 Comparison Methods.

• The proposed model is compared with Shared Bottom MTL, MMoE, and PLE, with ablations of encoding, Aspect Gating Fusion, and Domain Adaptive Multi Task modules.The MLP variant follows concatenated user-query-item representations with multiple MLP layers; MMoE-DA-JS adds a JS-divergence domain-adaptive layer and MMoE.Source: §3.1.2 · p. 4
• Task 4 compares text encoders and restored pretrained embeddings, including mean pooling, randomly initialized Transformer, BERT, ChatGLM-6B, and ChatGLM with prompts.ChatGLM encodings are reduced from 4096 to 32 dimensions through two MLP layers; fusion variants include Mean-Pooling, [CLS]-Gating, and Domain-Gating.Source: §3.1.2 · pp. 4, 6
• Cold-start tests compare supervised finetuning of the foundation model with single-domain training for Service Card Recommendation and Content Query Recommendation.Source: §3.1.2 · p. 6

3.2.1 S&R Multi-Domain Foundation model.

• MMoE-DA-JS achieved the best AUC on tasks 4–7, with improvements over baseline of +0.0404, +0.0192, +0.0297, and +0.0131, respectively.These results compare multi-domain models across seven search and recommendation datasets.Source: §3.2.1 · p. 6
• The S&R Foundation MLP embedding is scattered, whereas the MMoE-DA-JS embedding is coherently aligned.The comparison is shown in the t-SNE visualization of domain-adaptive layers.Source: §3.2.1 · p. 6

3.2.2 Domain-Invariant Text Features and Aspect Gating Fusion.

• Restoring pretrained BERT BASE parameters, freezing the encoder, and fine-tuning remaining networks achieved 0.7580 AUC, a 0.0056 gain over baseline; Domain-Gating reached 0.7524 AUC, a 0.0139 gain over mean pooling.Freezing pretrained parameters at levels L0 and L1 gave a 0.0043 AUC gain over the other freeze-finetune splits.Source: §3.2.2 · p. 6

3.2.3 Supervised Finetuning in Cold Start Scenarios.

• +0.0216 AUC on Service Card Recommendation and +0.0279 AUC on Content Query Recommendation over single-domain models; the fine-tuned foundation model also gained +17.54% relative PVCTR over baseline online.Cold-start evaluation used new scenarios with few samples; online testing compared against a single-domain DNN model.Source: §3.2.3 · p. 6, §3.2.4 · p. 6

3.2.4 Online AB Testing.

• The fine-tuned model performed better than baseline consistently in the online cold-start scenario.The Service Card Recommendation A/B test ran from day 1 to day 7.Source: §3.2.4 · p. 6

4 CONCLUSION

• The S&R Multi-Domain Foundation model combines LLM-encoded domain-invariant text features with Aspect Gating Fusion of ID, text, and sparse features, and uses domain adaptation for cold-start scenarios.The paper reports better cold-start performance than a single-domain model and deployment in Alipay search and recommendation scenarios.Source: §4 · p. 6