PaperlayerView exact revision on arXiv

Source-linked AI summary

CTRL: Connect Collaborative and Language Model for CTR Prediction

Xiangyang Li, Bo Chen, Lu Hou, Ruiming Tang

arXiv:2306.02841v4cs.IR

TL;DR

CTR prediction models can lose feature semantics through one-hot encoding, while PLM-based methods may omit collaborative signals and incur expensive online inference. CTRL aligns collaborative and semantic representations, then fine-tunes a lightweight collaborative model for serving. It outperforms state-of-the-art collaborative and semantic models, with AUC improvements over the best collaborative model across three datasets.

  • Problem

    One-hot CTR models discard feature semantics, while PLM-based approaches may omit collaborative relations and are computationally expensive for online inference.

    Source: §1 · pp. 1–2

  • Method

    CTRL converts tabular data into text, aligns its textual and tabular modalities, and fine-tunes the collaborative CTR model for online serving.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    CTRL improves AUC over the best collaborative model by 1.90%, 3.08%, and 4.45% on three datasets, and outperforms state-of-the-art collaborative and semantic models.

    Source: §5.2 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    CTRL is described as model-agnostic and industrial-friendly, with only the lightweight fine-tuned collaborative model deployed for efficient online serving.

    Source: §1 · p. 2

    CTRL’s alignment assumes correlated representations should be close under a distance metric and uncorrelated representations far apart.

    Source: §4.2.1 · p. 4

Abstract

from arXiv · show

Traditional click-through rate (CTR) prediction models convert the tabular data into one-hot vectors and leverage the collaborative relations among features for inferring the user's preference over items. This modeling paradigm discards essential semantic information. Though some works like P5 and CTR-BERT have explored the potential of using Pre-trained Language Models (PLMs) to extract semantic signals for CTR prediction, they are computationally expensive and suffer from low efficiency. Besides, the beneficial collaborative relations are not considered, hindering the recommendation performance. To solve these problems, in this paper, we propose a novel framework \textbf{CTRL}, which is industrial-friendly and model-agnostic with superior inference efficiency. Specifically, the original tabular data is first converted into textual data. Both tabular data and converted textual data are regarded as two different modalities and are separately fed into the collaborative CTR model and pre-trained language model. A cross-modal knowledge alignment procedure is performed to fine-grained align and integrate the collaborative and semantic signals, and the lightweight collaborative model can be deployed online for efficient serving after fine-tuned with supervised signals. Experimental results on three public datasets show that CTRL outperforms the state-of-the-art (SOTA) CTR models significantly. Moreover, we further verify its effectiveness on a large-scale industrial recommender system.

1 INTRODUCTION

Collaborative CTR models use feature co-occurrences but lose semantic information, while PLM-based approaches face efficiency challenges and may omit collaborative signals. CTRL aligns collaborative and semantic signals, then serves a fine-tuned lightweight CTR model without the language model.

1 INTRODUCTION

  • CTR prediction estimates users’ willingness to click on items from historical behavior data and informs whether items are displayed.Accurate CTR prediction is described as critical to user experience, product sales, and advertising platform revenue.Source: §1 · p. 1
  • Collaborative CTR models lose feature semantics through one-hot encoding, limiting recommendations for cold-start and low-frequency long-tailed features.PLM-based approaches can extract semantic information and contribute external knowledge beyond training data and scenarios.Source: §1 · p. 1
  • Semantic-only prediction may omit indispensable feature co-occurrences and user-item interactions, while online language-model inference is computationally expensive.These shortcomings create challenges in combining collaborative and semantic signals and serving efficiently without extensive engineering optimization.Source: §1 · pp. 1–2
  • CTRL aligns collaborative and semantic signals in two stages, then serves only the fine-tuned lightweight CTR model online.The framework converts tabular data into text, aligns the two modalities, and fine-tunes the collaborative model with supervised signals.Source: §1 · p. 2
  • CTRL achieves SOTA performance on three public datasets and is also verified on a large-scale industrial recommender system.The framework is described as model-agnostic, compatible with collaborative models and PLMs including LLMs, and efficient for inference.Source: §1 · p. 2

2 RELATED WORK

Collaborative recommendation models use one-hot tabular features and interaction functions to capture co-occurrences, but cannot capture original feature semantics. Language-model approaches provide semantic modeling, yet may neglect collaborative experience and face online latency challenges.

2.1 Collaborative Models for Recommendation

  • Collaborative models encode tabular features as one-hot vectors and use interaction functions to capture feature co-occurrences.Examples include Wide&Deep’s nonlinear layers, DeepFM’s inner products, and DCN and EDCN’s cross layers.Source: §2.1 · p. 2
  • Collaborative models cannot capture original feature semantics, which can hinder predictions for cold-start and low-frequency long-tailed features.Source: §2.1 · p. 2

2.2 Semantic Models for Recommendation

  • Language-model recommendation work includes generative, discriminative, and two-tower approaches, as well as studies using LLMs.P5 [14] uses T5 [47] for recommendation text generation; P-Tab [35] prompts and pre-trains tabular data; CTR-BERT uses two BERT models for user and item text.Source: §2.2 · p. 2
  • Semantic-only recommendation may discard collaborative experience, while language models’ parameter counts make low-latency online deployment challenging.The related-work discussion presents CTRL as combining collaborative and semantic signals through two-stage training.Source: §2.2 · p. 2

3 PRELIMINARY

The preliminary section defines CTR prediction as supervised binary classification over multi-field user, item, and context features. It outlines how collaborative CTR models encode and interact features, and why this can lose useful semantic information.

3 PRELIMINARY

  • CTR prediction is a supervised binary-classification task over instances (x, y), where y indicates the click label.The feature vector x contains multiple fields.Source: §3 · pp. 2–3
  • The multi-field input includes user profiles, item features, and contextual information used to estimate P(y=1|x).Examples include gender and occupation, category and price, and time and location [16].Source: §3 · p. 3
  • Collaborative CTR models one-hot encode tabular fields, map sparse vectors into embeddings, and apply interaction layers before predicting a CTR score.The score is optimized against the ground-truth label with Binary Cross Entropy.Source: §3 · p. 3
  • One-hot encoding discards feature semantics, leaving feature co-occurrences as the usable signals and potentially limiting performance when relations are weak.The passage identifies cold-start and low-frequency long-tailed features as such scenarios, motivating language models to supply semantic information.Source: §3 · p. 3

4 METHOD

CTRL converts tabular instances into text prompts, aligns collaborative and semantic representations across modalities, then fine-tunes the collaborative model for CTR prediction. Its stages combine cross-modal contrastive learning with fine-grained alignment before supervised downstream training.

4 METHOD

  • CTRL uses a two-stage training paradigm: cross-modal knowledge alignment followed by supervised fine-tuning of the collaborative model.Paired tabular and textual data enter the collaborative and language models, respectively, for alignment; the collaborative model is then fine-tuned on the downstream task.Source: §4 · p. 3, Figure 3 · p. 4

4.1 Prompt Construction

  • Prompt templates transform each training instance's tabular data into text describing user-side features and item-side features.The example prompt summarizes user profiles and viewing history in one sentence, then item attributes in another.Source: §4.1 · p. 3

4.2 Cross-modal Knowledge Alignment

  • CTRL aligns collaborative and semantic spaces by pairing tabular and textual information and applying contrastive learning.The same instance's two modalities form a positive pair; different samples' modalities form negative pairs through in-batch sampling.Source: §4.2 · p. 4, §4.2.1 · p. 4

4.2.1 Cross-modal Contrastive Learning.

  • The contrastive-learning stage encodes paired modalities, projects their representations to a common dimension, and averages two directional losses while updating both models.The losses use same-instance cross-modal representations as positives and representations from other samples as negatives.Source: §4.2.1 · pp. 4–5

4.2.2 Fine-grained Alignment.

  • CTRL addresses the global similarity objective's lack of fine-grained information alignment by transforming both modalities into M sub-spaces and summing maximum similarities across them.The collaborative and language-model parameters are updated during this alignment.Source: §4.2.2 · p. 5

4.3 Supervised Fine-tuning

  • After alignment, CTRL fine-tunes the collaborative model on the CTR task with supervised signals and deploys only that lightweight model online.The cross-modal stage produces an aligned hybrid representation space in which feature relations are mutually strengthened.Source: §4.3 · p. 5

5 EXPERIMENTS

Across three datasets, CTRL significantly outperforms the compared collaborative and semantic CTR models. The experiments also examine serving efficiency, modal alignment, model compatibility, ablations, and prompt design.

5.1.1 Datasets and Evaluation Metrics; 5.1.2 Competing Models; 5.1.3 Implementation Details

  • Experiments use MovieLens, Amazon (Fashion), and Taobao, evaluate AUC and Logloss, and compare CTRL with collaborative and semantic baselines.The reported implementation uses AutoInt and RoBERTa by default for alignment, then fine-tunes the collaborative model for CTR prediction.Source: §5.1.1 · p. 5, §5.1.2 · p. 5, §5.1.3 · p. 5

5.2 Performance Comparison

  • CTRL outperforms all compared collaborative and semantic models across three datasets, with AUC gains of 1.90%, 3.08%, and 4.45% over the best collaborative model.The table reports significance against the best baselines at p-value < 0.05.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 Serving Efficiency

  • With AutoInt as its collaborative backbone, CTRL retains AutoInt’s model parameters and inference time because the semantic model is not needed for online inference.The comparison is motivated by industrial serving limits of 10–20 milliseconds.Source: §5.3 · p. 6, Table 3 · p. 6

5.4 Visualization of Modal Alignment

  • In MovieLens t-SNE visualizations, tabular and textual representations move from separate spaces before alignment to a unified multimodal space afterward.The authors interpret this as alignment that injects semantic information and external general knowledge into the collaborative model.Source: §5.4 · pp. 6–7, Figure 4 · p. 7

5.5.1 Compatibility for semantic models

  • CTRL improves over AutoInt with different language-model backbones; ChatGLM achieves the best reported performance, while its gains over BERT-Large are mild.The study includes models from TinyBERT at 14.5M parameters to ChatGLM at 6B parameters, with most ChatGLM parameters frozen during training.Source: §5.5.1 · p. 7, Table 4 · p. 7

5.5.2 Compatibility for collaborative models

  • CTRL improves average RelaImpr across Wide&Deep, DeepFM, DCN, and AutoInt by 1.31%, 1.13%, 1.57%, and 2.61%, respectively.The study uses RoBERTa as the semantic model.Source: §5.5.2 · p. 7, Table 5 · p. 7

5.6.1 Ablation Study Analysis

  • Removing pre-trained language-model weights significantly hurts performance, replacing maxsim with cosine similarity degrades it, and end-to-end training underperforms CTRL’s pre-training and fine-tuning.The authors attribute the weight-removal result to the language model’s world knowledge and semantic modeling, and conjecture that multi-objective training may hurt CTR performance.Source: §5.6.1 · pp. 7–8, Figure 5 · p. 8

5.6.2 Prompt Analysis

  • Natural-language Prompt-1 performs best; comparisons also indicate that retaining feature-field semantics matters, while changing the field-value connective has little effect.The authors recommend prompts with feature fields and fluent, grammatically correct sentences that retain semantic information.Source: §5.6.2 · p. 8, Figure 6 · p. 8

6 APPLICATION IN INDUSTRY SYSTEM

CTRL was trained on Huawei user behavior data and deployed with its collaborative model at ranking. In a seven-day online test, it improved CTR over the base model.

6.1 Deploying Details of CTRL Online

  • Training used seven days of Huawei behavior data across more than 30 features, took about five hours, and deployed the collaborative model at ranking.CTRL trained for one epoch in stage one and five epochs in stage two, supporting daily model updates.Source: §6.1 · p. 8

6.2 Offline and Online Performance

  • CTRL added no serving latency compared with its AutoInt backbone in the reported industrial deployment.The offline model paired AutoInt with RoBERTa.Source: §6.2 · p. 8
  • CTRL significantly outperformed baselines in offline AUC and Logloss and gained 5% CTR over the base model in seven-day online A/B testing.The deployed service caters to tens of millions of Huawei users.Source: Table 6 · p. 8, §6.2 · p. 8

7 CONCLUSION

CTRL aligns collaborative and semantic signals for CTR prediction, then serves a fine-tuned lightweight collaborative model. The paper reports better performance than state-of-the-art collaborative and semantic models while maintaining inference efficiency.

7 CONCLUSION

  • CTRL uses contrastive learning to align tabular and converted textual data as collaborative and semantic modalities.After supervised fine-tuning, the lightweight collaborative model is deployed online.Source: §7 · p. 9
  • Experiments show CTRL outperforms state-of-the-art collaborative and semantic models while maintaining good inference efficiency.Source: §7 · p. 9
  • Future work includes applying CTRL to sequence recommendation and explainable recommendation.Source: §7 · p. 9

A.1 Datasets and Evaluation Metrics

The appendix describes the dataset splits and evaluation metrics, then outlines the collaborative and semantic models used as competing baselines.

A.1 Datasets and Evaluation Metrics

  • MovieLens and Amazon are split by user interaction time at 8:1:1, while Alibaba uses seven prior days for training and validation at 9:1 and the eighth day for testing.MovieLens ratings equal to 3 are removed; Amazon ratings above 3 are positive and the rest negative.Source: §A.1 · p. 10
  • AUC measures positive-over-negative ranking probability, while Logloss measures distance between two distributions in binary classification.Source: §A.1 · p. 10

A.2 Competing Models

  • Collaborative baselines include Wide&Deep, DeepFM, DCN, AutoInt, PNN, xDeepFM, and FiBiNET.Their described interaction mechanisms include cross-networks, self-attention, and factorization machines.Source: §A.2 · p. 10
  • Semantic baselines include P5 text generation with T5, CTR-BERT BERT towers, and P-Tab pre-training with masked language modeling.Source: §A.2 · p. 10

B HYPERPARAMETER ANALYSIS

The temperature coefficient affects CTRL's contrastive alignment performance on MovieLens and Amazon, with an observed optimum below 1 on both datasets.

B.1 The Impact of Contrastive Learning Temperature Coefficient

  • The optimal contrastive-learning temperature coefficient was below 1 on both MovieLens and Amazon in these experiments.Source: §B.1 · p. 10
  • Performance tended to improve and then decline as the temperature coefficient increased, making its value consequential for alignment.Source: §B.1 · p. 10
  • The temperature experiments evaluated CTRL on MovieLens and Amazon, with results reported in Figure 8(a).Source: §B.1 · p. 10

B.2 The Impact of First Stage Batch Size

Larger first-stage batch sizes are associated with improved performance on both datasets. The authors link this pattern to better cross-modal alignment and prediction accuracy.

B.2 The Impact of First Stage Batch Size

  • Larger batches during contrastive pre-training are described as conducive to better cross-modal alignment and prediction accuracy.Source: §B.2 · pp. 10–11
  • Performance improves on both datasets as first-stage batch size increases.Figure 8 examines the batch-size effect.Source: §B.2 · p. 10, Figure 8 · p. 11

Related papers

Loading 2306.02841v4…