PaperlayerView exact revision on arXiv

Source-linked AI summary

FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction

Kelong Mao, Jieming Zhu, Liangcai Su, Guohao Cai, Yuru Li, Zhenhua Dong

arXiv:2304.00902v4cs.IR

TL;DR

CTR models often augment MLPs because vanilla MLPs are inefficient at learning multiplicative feature interactions, but the strength of simply using two MLP streams has been underreported. The paper proposes FinalMLP with stream-specific feature gating and interaction aggregation, and reports better performance than many sophisticated two-stream models across benchmarks and an industrial A/B test.

  • Problem

    Vanilla MLPs are inefficient at learning multiplicative feature interactions, motivating two-stream models that add specialized interaction networks.

  • Method

    FinalMLP enhances two parallel MLP streams with pluggable stream-specific feature gating and multi-head bilinear interaction aggregation.

  • Results

    FinalMLP outperforms existing two-stream models on four open datasets and shows significant improvement over the deployed baseline in online A/B testing.

  • Takeaways & Limitations

    A well-tuned two-stream MLP can be surprisingly effective, and FinalMLP is proposed as a strong baseline for future two-stream CTR research.

  • Takeaways & Limitations

    Direct bilinear fusion becomes parameter-intensive and computationally expensive when stream outputs are high-dimensional.

Abstract

from arXiv · show

Click-through rate (CTR) prediction is one of the fundamental tasks for online advertising and recommendation. While multi-layer perceptron (MLP) serves as a core component in many deep CTR prediction models, it has been widely recognized that applying a vanilla MLP network alone is inefficient in learning multiplicative feature interactions. As such, many two-stream interaction models (e.g., DeepFM and DCN) have been proposed by integrating an MLP network with another dedicated network for enhanced CTR prediction. As the MLP stream learns feature interactions implicitly, existing research focuses mainly on enhancing explicit feature interactions in the complementary stream. In contrast, our empirical study shows that a well-tuned two-stream MLP model that simply combines two MLPs can even achieve surprisingly good performance, which has never been reported before by existing work. Based on this observation, we further propose feature gating and interaction aggregation layers that can be easily plugged to make an enhanced two-stream MLP model, FinalMLP. In this way, it not only enables differentiated feature inputs but also effectively fuses stream-level interactions across two streams. Our evaluation results on four open benchmark datasets as well as an online A/B test in our industrial system show that FinalMLP achieves better performance than many sophisticated two-stream CTR models. Our source code will be available at MindSpore/models.

Introduction

Existing CTR research commonly pairs MLPs with specialized interaction networks because vanilla MLPs are inefficient at learning multiplicative interactions. This paper finds that a tuned two-stream MLP is surprisingly strong and extends it into FinalMLP with feature gating and interaction aggregation.

  • Introduction: DualMLP, formed by combining two parallel MLPs, achieves performance comparable to or better than many well-designed two-stream models.This is presented as the first reported characterization of DualMLP against other two-stream models.
  • Introduction: Existing two-stream CTR models typically differentiate streams by using distinct network structures or learning different interaction types.Examples include low- versus high-order, explicit versus implicit, and vector-wise versus bit-wise interactions.
  • Introduction: Feature gating produces stream-specific feature importance weights for soft feature selection, while interaction aggregation enhances fusion between the streams.The gating mechanism can be conditioned on learnable parameters, user features, or item features.
  • Introduction: FinalMLP outperforms existing two-stream models on four open benchmark datasets and shows significant improvement over the deployed baseline in online A/B testing.The benchmark results are reported as new state-of-the-art performance.
  • Introduction: FinalMLP combines two MLP streams with feature gating and interaction aggregation, and is evaluated on benchmark datasets and in production.The proposed modules differentiate stream inputs and fuse stream-level interactions.

Background and Related Work

Two-stream CTR models use parallel networks to learn complementary feature interactions, transform sparse features into embeddings, and fuse stream outputs into click probabilities. Prior methods commonly use summation or concatenation, while this work motivates differentiated inputs and bilinear stream-level interaction aggregation.

  • Feature Embedding: Embedding maps high-dimensional sparse raw features into dense vectors, which are concatenated and passed to subsequent layers.Raw features may be categorical, multi-valued, or numerical.
  • Feature Embedding: Soft feature selection uses gating to assign feature-importance weights that amplify important features and suppress noisy ones, with this work applying it separately to each stream.Stream-specific gating is intended to create differentiated stream inputs.
  • Background and Related Work: Two-stream CTR models employ parallel networks that learn feature interactions from different views before producing a final click probability.The streams may use different interaction networks to capture complementary patterns.
  • Stream-Level Fusion: Existing fusion commonly sums or concatenates stream outputs, but these first-order operations do not capture stream-level feature interactions.The paper therefore explores a second-order bilinear function for interaction aggregation.
  • Representative Two-Stream CTR Models: Representative two-stream models pair MLPs with generalized linear, FM, cross, CIN, self-attention, logarithmic-transformation, interaction-machine, or mask-based networks.These designs target complementary explicit and implicit or otherwise differentiated feature interactions.

Our Model

The paper develops DualMLP and enhances it into FinalMLP with stream-specific feature gating and bilinear interaction aggregation. These modules differentiate feature inputs and model interactions between stream outputs while reducing bilinear-fusion cost through multi-head computation.

  • DualMLP: DualMLP combines two independent MLP networks as parallel streams whose sizes can differ by data.The two streams use feature inputs h1 and h2 and produce output representations o1 and o2.
  • Feature Gating: Feature gating reweights inputs separately for each stream using conditional features or learnable parameters, producing differentiated feature importance weights.The gating weights are transformed to [0, 2] with average 1 before producing weighted feature outputs.
  • Interaction Aggregation: Bilinear interaction aggregation adds second-order interactions between the two stream outputs instead of relying only on summation or concatenation.The bilinear term models interactions between o1 and o2; setting W3 to zero recovers linear concatenation fusion.
  • Multi-Head Bilinear Fusion: Multi-head bilinear fusion chunks both stream outputs into k paired subspaces, fuses each pair, and sum-pools the results into the predicted click probability.This reduces parameter and computation complexity theoretically by a factor of k, from O(d1d2) to O(d1d2/k).
  • FinalMLP: FinalMLP combines stream-specific feature gating with stream-level interaction aggregation as pluggable modules on two MLP networks.The model is trained with binary crossentropy using true labels and estimated click probabilities.

Experimental Setup

The evaluation uses four open CTR benchmark datasets and compares single-stream explicit-interaction networks with representative two-stream models under standardized settings. Models are implemented with FuxiCTR, and DualMLP and FinalMLP receive tuning for stream diversity.

  • Datasets: Experiments use the Criteo, Avazu, MovieLens, and Frappe open benchmark datasets with reused preprocessing and data-splitting settings.Table 1 summarizes the dataset statistics.
  • Evaluation Metric: AUC is used as an evaluation metric, with a 0.1-point increase recognized as a significant CTR-prediction improvement.The significance criterion is attributed to prior CTR-prediction studies.
  • Baselines: The study evaluates single-stream explicit-interaction networks spanning first-, second-, third-, and higher-order interaction designs.The listed baselines include LR, FM-family models, CrossNet variants, CIN, AutoInt, FiGNN, AFN, and SAM.
  • Baselines: Representative two-stream CTR models are also evaluated alongside the single-stream baselines.The passage identifies this as a separate experimental group following the related-work discussion.
  • Implementation Settings: Models are implemented with FuxiCTR using embedding dimension 10, batch size 4096, default MLP size [400, 400, 400], and learning rates 1e-3 or 5e-4.For DualMLP and FinalMLP, the two MLPs are tuned from 1 to 3 layers to enhance stream diversity.

MLP vs. Explicit Feature Interactions

The paper directly compares MLPs with explicit feature-interaction networks and finds that MLPs can match or outperform them, despite their recognized weakness in learning multiplicative interactions.

  • MLP achieves the best performance on Criteo, MovieLens, and Frappe, and the second-best performance on Avazu among compared feature-interaction methods.On Avazu, its AUC is within 0.01 of the best result.
  • The results indicate that MLPs are highly expressive for learning feature interactions implicitly, helping explain their use alongside explicit interaction networks in two-stream models.

DualMLP and FinalMLP vs. Two-Stream Baselines

Two-stream models generally outperform single-stream baselines, while DualMLP performs unexpectedly well and FinalMLP consistently surpasses the compared two-stream baselines across four datasets.

  • Two-stream models generally outperform single-stream baselines, especially the single MLP model, by learning complementary features.
  • DualMLP achieves comparable or better performance than sophisticated two-stream baselines when its two MLP streams are carefully tuned.Using different MLP sizes to increase stream diversity improves DualMLP performance.
  • FinalMLP extends the strong DualMLP baseline through pluggable feature gating and fusion modules.
  • FinalMLP consistently outperforms DualMLP and all compared two-stream baselines across four open datasets.It surpasses DCNv2, xDeepFM, and AutoInt+ by 0.12, 0.23, and 0.11 AUC points on Avazu, MovieLens, and Frappe, respectively.

Ablation Studies

Ablations show that both feature selection and bilinear fusion contribute to FinalMLP, with bilinear fusion producing the larger performance effect; head count also requires balancing capacity and redundancy.

  • The ablation variants include DualMLP, FinalMLP without feature selection, and summation, concatenation, and element-wise-product fusion.
  • Removing feature selection or replacing bilinear fusion with summation, concatenation, or element-wise product reduces performance.The comparisons are presented as an ablation study of FinalMLP’s key designs.
  • Bilinear fusion is more important than feature selection because replacing it causes greater performance degradation.
  • Using more fusion parameters through smaller head counts does not always improve performance.An appropriate number of heads balances multi-view stream interaction learning against redundant interactions.

Industrial Evaluation

Industrial evaluation shows that FinalMLP improves offline AUC over deployed baselines and achieves a significant online CTR gain in a production news-recommendation system.

  • FinalMLP obtains over one AUC point improvement over the deployed BaseModel in offline evaluation using 1.2 billion samples from three-day click logs.
  • FinalMLP improves offline AUC by an additional 0.44 points over EDCN.
  • FinalMLP achieves a 1.6% average improvement in CTR during a five-day online A/B test.CTR measures users’ clicks divided by total news impressions, and the improvement is significant in the production system.
  • Using eight fusion heads reduces latency from 70ms to 47ms, matching the deployed BaseModel’s 45ms latency level.

Conclusion and Outlook

FinalMLP is presented as a simple yet effective two-stream MLP for CTR prediction, enhanced with stream-specific feature gating and multi-head bilinear fusion. Results across four open datasets and industrial settings support its effectiveness and motivate it as a strong baseline for future two-stream CTR research.

  • Conclusion and Outlook: FinalMLP employs MLPs in both streams, with stream-specific feature gating and multi-head bilinear fusion to differentiate inputs and enable stream-level interaction.The modules are described as pluggable enhancements to the two-stream model.
  • Conclusion and Outlook: Evaluation on four open datasets and industrial settings demonstrates FinalMLP's strong effectiveness for CTR prediction.
  • Conclusion and Outlook: FinalMLP's results question the effectiveness and necessity of explicit feature-interaction modeling in two-stream CTR research.
  • Conclusion and Outlook: The authors propose FinalMLP as a strong baseline for future two-stream CTR models and suggest extending its modules to other architectures.
Loading 2304.00902v4…