PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Interest Network for Click-Through Rate Prediction

Guorui Zhou, Chengru Song, Xiaoqiang Zhu, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, Kun Gai

arXiv:1706.06978v4stat.MLcs.LG

TL;DR

Fixed-length representations in conventional deep CTR models make it difficult to capture diverse user interests from rich historical behaviors. DIN learns ad-specific interest representations through local activation, while two training techniques support large-scale networks; experiments report superior performance and Alibaba deployment.

  • Problem

    Fixed-length user representations in Embedding&MLP CTR models limit their ability to capture diverse interests from historical behaviors.

  • Method

    DIN uses a local activation unit to weight historical behaviors by relevance to a candidate ad, complemented by mini-batch aware regularization and a data adaptive activation function.

  • Results

    DIN achieves superior performance across public and Alibaba datasets, including 0.0113 absolute AUC gain and 11.65% RelaImpr over BaseModel on Alibaba.

  • Takeaways & Limitations

    DIN’s ad-varying user representation improves expressive ability, and the proposed techniques support industrial-scale training and Alibaba deployment.

  • Takeaways & Limitations

    LSTM modeling of historical behavior sequences showed no improvement, and sequence modeling of concurrent interests remains future work.

Abstract

from arXiv · show

Click-through rate prediction is an essential task in industrial applications, such as online advertising. Recently deep learning based models have been proposed, which follow a similar Embedding\&MLP paradigm. In these methods large scale sparse input features are first mapped into low dimensional embedding vectors, and then transformed into fixed-length vectors in a group-wise manner, finally concatenated together to fed into a multilayer perceptron (MLP) to learn the nonlinear relations among features. In this way, user features are compressed into a fixed-length representation vector, in regardless of what candidate ads are. The use of fixed-length vector will be a bottleneck, which brings difficulty for Embedding\&MLP methods to capture user's diverse interests effectively from rich historical behaviors. In this paper, we propose a novel model: Deep Interest Network (DIN) which tackles this challenge by designing a local activation unit to adaptively learn the representation of user interests from historical behaviors with respect to a certain ad. This representation vector varies over different ads, improving the expressive ability of model greatly. Besides, we develop two techniques: mini-batch aware regularization and data adaptive activation function which can help training industrial deep networks with hundreds of millions of parameters. Experiments on two public datasets as well as an Alibaba real production dataset with over 2 billion samples demonstrate the effectiveness of proposed approaches, which achieve superior performance compared with state-of-the-art methods. DIN now has been successfully deployed in the online display advertising system in Alibaba, serving the main traffic.

1 INTRODUCTION

CTR prediction matters directly for advertising revenue, but fixed-length user representations limit deep models’ ability to capture diverse interests. DIN addresses this with ad-specific interest activation, alongside training techniques for large industrial networks.

  • CTR prediction directly affects advertising revenue because CPC systems rank ads using eCPM, the product of bid price and CTR.
  • Embedding&MLP methods map sparse features to embeddings, pool them into fixed-length vectors, and use an MLP to learn nonlinear feature relations.
  • Fixed-length user vectors constrain representation of users’ diverse interests, while enlarging embeddings increases parameter size and can burden industrial systems.
  • DIN uses a local activation unit to weight historical behaviors by relevance to each candidate ad, producing an ad-dependent interest representation.
  • The paper proposes mini-batch aware regularization and a data adaptive activation function to support training industrial deep networks.
  • Experiments on public and Alibaba datasets support the effectiveness of DIN and its training techniques, with deployment in Alibaba’s commercial advertising system.

2 RELATEDWORK

CTR models evolved from embedding-based shallow networks toward deeper architectures that learn feature relations with MLPs and specialized interaction structures. Attention-based methods motivate DIN’s local activation design, which was also deployed at scale.

  • CTR model development has progressed from shallow to deep architectures as datasets and feature dimensions have grown.
  • Embedding-based models learn dense representations of sparse inputs and apply transformation functions to capture feature combinations.
  • Deep Crossing, Wide&Deep, and related models replace simpler transformations with MLPs, while PNN and DeepFM add specialized feature-interaction structures.
  • Attention methods use weighted sums to focus on information relevant to a target, and DIN adapts this idea with a local activation unit.
  • The paper reports public code and successful deployment of DIN with techniques for training networks containing hundreds of millions of parameters.

3 BACKGROUND

Alibaba’s display advertising system matches users with candidate ads and then ranks those candidates by predicted CTR. Large-scale user behavior data supports both stages and reflects users’ diverse interests.

  • Figure 1 illustrates the running procedure of Alibaba’s display advertising system and the role of user behavior data.
  • Alibaba’s display advertising system has matching and ranking stages: matching generates relevant candidate ads, while ranking predicts CTR and selects top-ranked ads.
  • User behavior data contributes critically to building matching and ranking models in the system.
  • The system handles hundreds of millions of users whose rich historical behaviors contain diverse interests.

4 DEEP INTEREST NETWORK

The section represents sparse, multi-group user and advertisement features with embeddings before applying an Embedding&MLP base model or DIN’s ad-conditioned local activation. DIN addresses the fixed-length bottleneck by adaptively weighting historical behaviors for each candidate ad.

  • 4.1 Feature Representation: Industrial CTR inputs are multi-group categorical data commonly encoded as high-dimensional sparse binary vectors.Feature groups may be one-hot or multi-hot, and user behavior features are typically multi-hot.
  • 4.1 Feature Representation: Embedding tables transform sparse feature indicators into low-dimensional dense vectors through table lookup.One-hot groups produce a single embedding, whereas multi-hot groups produce a list of embeddings.
  • 4.1 Feature Representation: Pooling and concatenation convert variable-length behavioral embeddings into fixed-length group representations for the downstream MLP.Sum and average pooling are common choices, after which group representations are concatenated.
  • 4.2 Base Model(Embedding&MLP): The base Embedding&MLP model pools all user behaviors into one representation that remains unchanged across candidate ads.Expanding this vector can increase overfitting, computation, and storage burdens in industrial systems.
  • 4.3 The structure of Deep Interest Network: DIN applies a local activation unit to compute an ad-dependent weighted sum of historical behavior embeddings.The activation network uses behavior and ad embeddings, their product, and learned activation weights to model relevance.
  • 4.3 The structure of Deep Interest Network: DIN preserves activation-weight magnitude rather than normalizing weights with softmax, retaining the intensity of user interests.The local activation unit is related to attention methods but relaxes the traditional sum-to-one constraint.

5 TRAINING TECHNIQUES

This section addresses the computational difficulty of regularizing industrial networks with hundreds of millions of sparse-feature parameters and presents mini-batch-aware regularization alongside Dice, a data-adaptive activation function.

  • 5.1 Mini-batch Aware Regularization: Industrial networks with hundreds of millions of sparse-feature parameters make conventional regularization computationally impractical.Traditional ℓ2 regularization requires computing the norm over all parameters for every mini-batch, despite only non-zero sparse-feature parameters being updated.
  • 5.1 Mini-batch Aware Regularization: The mini-batch-aware regularizer computes the ℓ2 norm only for sparse-feature parameters appearing in the current mini-batch.This restricts regularization computation to active features and makes regularization feasible for large embedding dictionaries.
  • 5.1 Mini-batch Aware Regularization: The method approximates full ℓ2 regularization by aggregating contributions across mini-batches according to whether each feature appears in a batch.The indicator α_mj is one when at least one instance in mini-batch B_m contains feature j.
  • 5.2 Data Adaptive Activation Function: PReLU switches between s and αs using a hard control point at zero, with α learned as the parameter of the second channel.The control function is p(s)=I(s>0).
  • 5.2 Data Adaptive Activation Function: Dice generalizes PReLU by shifting the rectification point according to input distribution and switching smoothly between its two channels.The rectification point is set to the input mean; when the mean and variance are zero, Dice degenerates into PReLU.

6 EXPERIMENTS

The experiments evaluate the proposed approach on two public user-behavior datasets and a large Alibaba display-advertising dataset, using dataset-specific CTR prediction setups and reporting effectiveness against state-of-the-art methods.

  • 6 EXPERIMENTS: Experiments on two public datasets and an Alibaba production dataset report that the proposed approach outperforms state-of-the-art CTR prediction methods.The public datasets contain user behaviors, while the Alibaba data come from an online display advertising system.
  • 6.1 Datasets and Experimental Setup: The Amazon Electronics subset contains 192,403 users, 63,001 goods, 801 categories, and 1,689,188 samples.Its features include goods and category identifiers plus users’ reviewed-good and category lists.
  • 6.1 Datasets and Experimental Setup: MovieLens contains 138,493 users, 27,278 movies, 21 categories, and 20,000,263 samples converted into binary classification data.Ratings of 4 or 5 are labeled positive, and the remaining ratings negative.
  • 6.1 Datasets and Experimental Setup: The Alibaba dataset uses about 2 billion training samples and 0.14 billion testing samples collected from online display advertising traffic.The setup uses embedding dimension 12 across 16 feature groups and an MLP of 192 × 200 × 80 × 2.
  • 6.1 Datasets and Experimental Setup: Alibaba’s dataset volume is much larger than Amazon’s and MovieLens’s, creating greater experimental challenges.The paper describes the Alibaba data as substantially larger than both public datasets.

6.2 Competitors

The comparison includes LR, BaseModel, Wide&Deep, PNN, and DeepFM, evaluated with AUC and RelaImpr; the Alibaba regularization figure contrasts training behavior.

  • LR serves as a weak shallow baseline, while BaseModel provides the strong Embedding&MLP baseline for deep-model comparisons.
  • Wide&Deep combines manually designed cross-product features with a deep BaseModel component, whereas PNN adds a product layer and DeepFM adds a factorization-machines wide module.
  • AUC measures ranking quality for predicted ads, and the experiments use a user-weighted variant that averages intra-user AUC across users.
  • RelaImpr measures relative improvement over models, with AUC equal to 0.5 for a random guesser.
  • On Alibaba, fine-grained goods features overfit without regularization, while the proposed mini-batch aware regularization performs best among the compared regularizers.

6.4 Result from model comparison on Amazon Dataset and MovieLens Dataset

Across Amazon and MovieLens, DIN performs best among the compared models, especially on Amazon, and Dice further improves DIN.

  • DIN performs best among all competitors on the Amazon and MovieLens datasets.The experiments average results over five runs, with random initialization affecting AUC by less than 0.0002.
  • DIN stands out particularly on Amazon, which contains rich user behaviors.
  • DIN’s local activation unit soft-searches historical behaviors relevant to the candidate ad, producing an adaptively varying interest representation.

6.5 Performance of regularization

Fine-grained goods identifiers create severe overfitting in Alibaba training without regularization; compared methods mitigate it to different degrees, with MBA performing best.

  • Without regularization, fine-grained goods-id features cause serious overfitting after the first training epoch.The feature dimension reaches about 0.6 billion in the Alibaba setting.
  • The comparison evaluates dropout, frequency filtering, DiFacto regularization, and the proposed mini-batch aware regularization.
  • Mini-batch aware regularization performs best and significantly prevents overfitting.Dropout prevents quick overfitting but slows convergence, while frequency filtering only partly relieves overfitting.
  • Well-trained models using fine-grained goods identifiers achieve better AUC than models without them because the features contain richer information.Frequency filtering may discard low-frequency identifiers and reduce the model’s opportunity to use fine-grained features.

6.6 Result from model comparison on Alibaba Dataset

On Alibaba’s full feature set, DIN outperforms competing deep models, while MBA regularization and Dice provide additional gains; online testing reports improvements in CTR and RPM.

  • DIN achieves a 0.0059 absolute AUC gain and 6.08% RelaImpr over BaseModel.Under the same activation function and regularization, DIN outperforms BaseModel, Wide&Deep, PNN, and DeepFM.
  • MBA regularization adds a 0.0031 absolute AUC gain over dropout, and Dice adds a 0.0015 absolute AUC gain over PReLU.
  • DIN with MBA regularization and Dice achieves 11.65% RelaImpr and a 0.0113 absolute AUC gain over BaseModel.It also achieves a 0.009 absolute AUC gain over DeepFM.
  • Online A/B testing reports up to 10.0% CTR and 3.8% RPM promotion versus BaseModel, and DIN was deployed to serve the main traffic.The testing ran in Alibaba’s display advertising system from 2017-05 to 2017-06.
  • The Alibaba comparison uses full feature sets, and the table reports RelaImpr relative to BaseModel.

6.8 Visualization of DIN

The visualization study examines DIN’s adaptive activation and learned embeddings. Relevant behaviors receive higher weights, while goods from the same category form clusters in the learned embedding space.

  • Adaptive activation: High-relevance behaviors receive high activation weights for a candidate ad.The case study uses this pattern to examine the local activation unit.
  • Embedding visualization: DIN visualizes goods embeddings with shapes denoting categories and colors denoting CTR prediction values.The visualization uses t-SNE on learned goods embeddings.
  • Embedding visualization: Goods from the same category almost form one cluster in DIN’s embedding space.The visualization uses nine categories and 100 goods from each category as candidate ads for a young mother.
  • Interpretation: The candidate-ad-dependent embedding space helps DIN capture a user’s diverse interests.The representation is examined across candidate ads rather than compressed independently of the candidate.

7 CONCLUSIONS

The paper addresses CTR prediction in e-commerce display advertising, where fixed-length representations limit modeling of diverse user interests. DIN activates related behaviors to create ad-dependent user-interest representations, supplements this design with two training techniques, and is deployed in Alibaba’s online display advertising system.

  • 7 CONCLUSIONS: DIN targets CTR prediction with rich user behavior data in e-commerce display advertising.The paper identifies fixed-length representations as a bottleneck for capturing diverse user interests.
  • 7 CONCLUSIONS: DIN activates related user behaviors to obtain an adaptive user-interest vector that varies across ads.This design is intended to improve expressive ability under limited dimensionality.
  • 7 CONCLUSIONS: Mini-batch aware regularization and data adaptive activation function support training industrial deep networks and further improve DIN’s performance.The paper states these techniques can generalize to other industrial deep learning tasks.
  • 7 CONCLUSIONS: DIN has been deployed in Alibaba’s online display advertising system.The deployment serves the system’s main traffic.
Loading 1706.06978v4…