PaperlayerView exact revision on arXiv

Source-linked AI summary

MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask

Zhiqiang Wang, Qingyun She, Junlin Zhang

arXiv:2102.07619v2cs.IR

TL;DR

CTR ranking models need to capture complex feature interactions, but feed-forward layers can be inefficient for common feature crosses. MaskNet introduces instance-guided element-wise multiplication through MaskBlock and reports significant outperformance of state-of-the-art models on three real-world datasets.

  • Problem

    Feed-forward neural networks used in DNN ranking models can be inefficient at capturing common feature crosses and complex feature interactions.

  • Method

    MaskNet uses an instance-guided mask for element-wise multiplication on feature embeddings and feed-forward layers, organized through MaskBlock with layer normalization and a feed-forward layer.

  • Results

    On three real-world datasets, the proposed MaskNet models significantly outperform state-of-the-art models such as DeepFM and xDeepFM.

  • Takeaways & Limitations

    MaskBlock is presented as an effective basic building unit for composing high-performance ranking systems.

  • Takeaways & Limitations

    The instance-guided mask always takes its input from the input instance, specifically the feature embedding layer.

Abstract

from arXiv · show

Click-Through Rate(CTR) estimation has become one of the most fundamental tasks in many real-world applications and it's important for ranking models to effectively capture complex high-order features. Shallow feed-forward network is widely used in many state-of-the-art DNN models such as FNN, DeepFM and xDeepFM to implicitly capture high-order feature interactions. However, some research has proved that addictive feature interaction, particular feed-forward neural networks, is inefficient in capturing common feature interaction. To resolve this problem, we introduce specific multiplicative operation into DNN ranking system by proposing instance-guided mask which performs element-wise product both on the feature embedding and feed-forward layers guided by input instance. We also turn the feed-forward layer in DNN model into a mixture of addictive and multiplicative feature interactions by proposing MaskBlock in this paper. MaskBlock combines the layer normalization, instance-guided mask, and feed-forward layer and it is a basic building block to be used to design new ranking model under various configurations. The model consisting of MaskBlock is called MaskNet in this paper and two new MaskNet models are proposed to show the effectiveness of MaskBlock as basic building block for composing high performance ranking systems. The experiment results on three real-world datasets demonstrate that our proposed MaskNet models outperform state-of-the-art models such as DeepFM and xDeepFM significantly, which implies MaskBlock is an effective basic building unit for composing new high performance ranking systems.

1 INTRODUCTION

CTR ranking models need to capture complex feature interactions, but feed-forward layers can be inefficient at modeling common feature crosses. MaskNet addresses this by adding instance-guided multiplicative operations and using MaskBlock as a compositional building block.

  • CTR prediction supports personalized advertising and recommender systems, making effective modeling of complex feature interactions important.
  • Prior work reports that feed-forward neural networks are inefficient at capturing common feature crosses and complex interactions.
  • MaskNet introduces an instance-guided mask that performs element-wise products on feature embeddings and feed-forward layers using information from the input instance.
  • MaskBlock combines layer normalization, an instance-guided mask, and a feed-forward hidden layer to mix multiplicative and feed-forward feature interactions.
  • Serial MaskNet stacks MaskBlocks, whereas parallel MaskNet places multiple MaskBlocks in parallel over a shared feature embedding layer.
  • Experiments on three real-world datasets show that the two proposed MaskNet models significantly outperform state-of-the-art models.

2 RELATED WORK

CTR research includes classical, neural, explicit-interaction, gating, and normalization approaches. These lines of work provide the context for MaskNet’s use of feature-wise masking and layer normalization.

  • FNN uses FM to pre-train embeddings, Wide & Deep combines wide linear and deep components, and DeepFM replaces the wide component with FM while sharing embeddings.
  • DCN and xDeepFM explicitly model feature interactions, contrasting with DNN ranking models that use MLP layers to model high-order interactions implicitly.
  • Feature-wise gating has been explored in vision, natural language processing, and recommendation systems, including channel recalibration and gated linear units.
  • BatchNorm and LayerNorm are prominent normalization techniques, with LayerNorm introduced to ease optimization of recurrent neural networks.

3 OUR PROPOSED MODEL

The proposed model introduces instance-guided, element-wise multiplicative masking into DNN ranking systems and combines it with normalization and feed-forward layers in MaskBlock. MaskNet then composes these blocks through serial or parallel configurations.

  • 3.1 Embedding Layer: The embedding layer converts sparse and dense inputs into a fixed-length concatenated vector of field embeddings.Each field embedding has dimension k, producing an embedding length of f × k.
  • 3.2 Instance-Guided Mask: The mask network uses a wider aggregation layer to collect global context, followed by a projection layer that matches the embedding or hidden-layer dimensionality.Different masks are used for the feature embedding and different MLP layers.
  • 3.2 Instance-Guided Mask: Instance-guided masks use the current feature embedding to emphasize informative elements and suppress uninformative or noisy elements in embeddings and hidden layers.The mask applies element-wise products and uses input-instance context to guide both feature and hidden representations.
  • 3.3 MaskBlock: MaskBlock combines layer normalization, an instance-guided mask, and a following feed-forward layer to mix multiplicative and additive feature interactions.Normalization is used on embeddings and hidden layers, while the feed-forward layer aggregates masked information.
  • 3.4 MaskNet: MaskNet uses MaskBlock as its basic building unit in two configurations: serial stacking and parallel blocks sharing the feature embedding.The serial model feeds blocks sequentially, whereas the parallel model places multiple blocks on the shared embedding and uses a final prediction layer.

4 EXPERIMENTAL RESULTS

The experiments evaluate MaskNet on three real-world datasets and use ablation and hyper-parameter analyses to investigate performance and component effects.

  • Experiments evaluate MaskNet on three real-world datasets.
  • The study compares MaskNet with existing state-of-the-art deep learning CTR models.
  • Ablation studies examine the influence and necessity of MaskBlock components.
  • Additional analyses study how network hyper-parameters affect the two proposed MaskNet models.

4.1 Experiment Setup

The evaluation uses three datasets, AUC-based metrics, several CTR baselines, and standardized neural-network settings for model comparison.

  • Datasets: The experiments use Criteo, Malware, and Avazu datasets for evaluation.Criteo contains categorical and continuous fields; Malware is formulated as binary classification; Avazu contains 23 impression fields.
  • Datasets: Instances are randomly split into training, validation, and test sets in an 8:1:1 ratio.
  • Metrics: AUC is the primary evaluation metric, while RelaImp measures relative AUC improvement over a corresponding baseline.Higher AUC indicates better performance, and RelaImp removes the constant 0.5 random-strategy component.
  • Baselines: Models are compared against FM, DNN, DeepFM, DCN, xDeepFM, and AutoInt, with FM serving as the evaluation base model.
  • Implementation Details: Table 2 reports AUC across three datasets with embedding size 10 and three MaskBlocks for each proposed model.

4.2 Performance Comparison (RQ1)

Both serial and parallel MaskNet outperform the evaluated state-of-the-art methods across all three datasets, with statistically significant gains and comparable performance between the two variants.

  • Both serial and parallel MaskNet achieve better performance on all three datasets than the state-of-the-art methods.The reported significance test uses α=0.01.
  • 3.12% to 11.40% is MaskNet’s reported accuracy improvement over the FM baseline.
  • 1.55% to 5.23% is MaskNet’s reported accuracy improvement over the DeepFM baseline.
  • 1.27% to 4.46% is MaskNet’s reported accuracy improvement over the xDeepFM baseline.
  • Serial and parallel MaskNet show comparable performance across the three evaluation datasets.The paper presents MaskBlock as an effective basic building unit for composing ranking systems.

4.3 Ablation Study of MaskBlock (RQ2)

Ablation results support the necessity of instance-guided masking and layer normalization, while the feed-forward layer affects serial and parallel MaskNet differently.

  • Removing either the instance-guided mask or layer normalization decreases model performance.The authors therefore identify both components as necessary for MaskBlock effectiveness.
  • Removing the feed-forward layer dramatically degrades serial MaskNet performance.
  • Removing the feed-forward layer appears not to harm parallel MaskNet performance.
  • The authors attribute this difference to parallel MaskNet’s additional feed-forward layers above its parallel MaskBlocks.These layers may provide a function similar to the removed MaskBlock feed-forward layer.

4.4 Hyper-Parameter Study(RQ3)

The study examines how feature embedding size, MaskBlock count, and mask reduction ratio affect MaskNet performance on Criteo. Larger embeddings help initially, while deeper stacking benefits the two architectures differently and reduction ratio has little effect.

  • Number of Feature Embedding Size: Both MaskNet models improve initially with larger feature embedding sizes, but performance degrades beyond 50 for SerMaskNet and 30 for ParaMaskNet.The experiments use three MaskBlocks on Criteo.
  • Number of MaskBlock: SerMaskNet performance increases with additional MaskBlocks until more than five, whereas ParaMaskNet performance increases slowly as blocks are added.The experiment stacks one to nine MaskBlocks for both models.
  • Reduction Ratio in Instance-Guided Mask: Different reduction ratios have little influence on model performance, allowing small ratios to save computation in applications.Ratios from 1 to 5 were tested by changing the aggregation-layer size.

4.5 Instance-Guided Mask Study(RQ4)

The instance-guided mask produces mostly small values while selectively boosting informative signals, and its outputs vary across input instances. These observations support input-dependent attention over different embedding and feed-forward elements.

  • Study Design: The experiment evaluates instance-guided masks by sampling 100,000 instances and comparing mask outputs for two individual instances.The analysis examines distributions across blocks and instance-specific differences.
  • Mask Value Distribution: More than 50% of mask values are near zero, while larger values boost some informative signals and suppress uninformative or noisy elements.This distribution was observed across 100,000 sampled Criteo instances and different MaskNet blocks.
  • Instance-Specific Mask Values: Different input instances direct masks toward different feature-embedding regions and feed-forward-layer elements.Two sampled instances produced distinct attention patterns in both kinds of mask values.

5 CONCLUSION

The paper introduces instance-guided feature-wise multiplication and MaskBlock for CTR ranking, then builds two MaskNet models from this block. On three real-world datasets, the proposed models significantly outperform DeepFM and xDeepFM.

  • Conclusion: Instance-guided masks apply element-wise products to feature embeddings and feed-forward layers, introducing multiplicative operations into DNN ranking systems.The masks are guided by the input instance.
  • Conclusion: MaskBlock combines layer normalization, an instance-guided mask, and a feed-forward layer to mix multiplicative and additive feature interactions.It serves as a basic building block for ranking models.
  • Conclusion: The proposed MaskNet models significantly outperform state-of-the-art models such as DeepFM and xDeepFM on three real-world datasets.The paper presents two MaskNet models based on MaskBlock.
Loading 2102.07619v2…