PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoNet: Memorizing All Cross Features' Representations Efficiently via Multi-Hash Codebook Network for CTR Prediction

Pengtao Zhang, Junlin Zhang

arXiv:2211.01334v3cs.IRcs.AIcs.LG

TL;DR

Modern deep CTR models show little performance benefit from increasing parameters, despite the value of cross features. MemoNet combines a DNN backbone with HCNet, an independent memory for cross-feature representations; the paper reports superior results over state-of-the-art methods and scaling gains as the codebook grows.

  • Problem

    The paper identifies limited gains from increasing modern deep CTR-model parameters and argues that network structures inefficiently extract and memorize useful cross-feature knowledge.

    Source: §1 · p. 1

  • Method

    HCNet stores and retrieves cross-feature representations through a multi-hash codebook, and MemoNet combines HCNet with a DNN backbone.

    Source: §1 · p. 2

  • Results

    MemoNet outperforms state-of-the-art models on three public datasets, improves performance in online tests, and shows scaling gains as its codebook grows.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors present learning and memorizing cross-feature representations with limited resources as feasible and as a promising research direction.

    Source: §1 · p. 2

Abstract

from arXiv · show

New findings in natural language processing (NLP) demonstrate that the strong memorization capability contributes a lot to the success of Large Language Models (LLM). This inspires us to explicitly bring an independent memory mechanism into CTR ranking model to learn and memorize cross features' representations. In this paper, we propose multi-Hash Codebook NETwork (HCNet) as the memory mechanism for efficiently learning and memorizing representations of cross features in CTR tasks. HCNet uses a multi-hash codebook as the main memory place and the whole memory procedure consists of three phases: multi-hash addressing, memory restoring, and feature shrinking. We also propose a new CTR model named MemoNet which combines HCNet with a DNN backbone. Extensive experimental results on three public datasets and online test show that MemoNet reaches superior performance over state-of-the-art approaches. Besides, MemoNet shows scaling law of large language model in NLP, which means we can enlarge the size of the codebook in HCNet to sustainably obtain performance gains. Our work demonstrates the importance and feasibility of learning and memorizing representations of cross features, which sheds light on a new promising research direction.

1 INTRODUCTION

MemoNet introduces an independent memory mechanism for learning and memorizing cross-feature representations, motivated by the limited gains from simply increasing CTR-model parameters. Its experiments report improved performance over state-of-the-art methods and scaling gains as the memory grows.

1 INTRODUCTION

  • The paper argues that increasing parameters in modern deep CTR models has little effect, while useful cross features hold important knowledge in CTR data.This motivates testing whether an independent memory can improve CTR performance as its size increases.Source: §1 · p. 1
  • HCNet stores cross-feature representations in a multi-hash codebook and uses multi-hash addressing, memory restoring, and feature shrinking.Addressing locates codebook chunks, restoring recovers feature representations, and shrinking compresses them into a thinner embedding layer.Source: §1 · p. 2
  • MemoNet combines HCNet with a DNN backbone, assigning cross-feature memory to HCNet and generalization across feature orders to the DNN.The authors present this division of labor as reducing the DNN's learning burden.Source: §1 · p. 2
  • MemoNet significantly improves performance over state-of-the-art CTR methods on three public datasets and online tests, and exhibits gains as its codebook grows.The authors describe it as the first CTR model with this scaling ability.Source: §1 · p. 2
  • HCNet is proposed as a general memory module for learning and memorizing cross-feature representations, and the authors identify this as a distinct alternative to specialized interaction sub-networks.The paper contrasts independent memory with modeling cross features through a DNN sub-network.Source: §1 · p. 2

2 RELATED WORKS

Deep CTR models explicitly model feature interactions, but manual feature engineering and multi-stage interaction mining remain limited. The paper motivates an independent memory mechanism as an alternative for learning cross features.

2.1 Deep CTR Methods Modeling Feature Interactions

  • Explicit interaction models capture pairwise or higher-order features, with reported results showing advantages over implicit approaches.Examples include DeepFM, DCN, DCN V2, xDeepFM, AutoInt, FiBiNet, MaskNet, and CAN.Source: §2.1 · p. 2
  • Human-designed cross features remain common in real-life CTR systems, indicating inefficiency in explicitly modeling interactions with deep networks.The paper proposes independent memory for cross-feature representations so the MLP can focus on generalization.Source: §2.1 · p. 3

2.2 Deep CTR Methods Mining Feature Interactions

  • Multi-stage interaction mining can miss newly beneficial interactions in online learning and long-tail useful cross features.The paper states its approach can be deployed in online learning systems and capture long-tail beneficial cross features efficiently.Source: §2.2 · p. 3

3 PRELIMINARIES

The preliminaries describe the DNN components used for CTR prediction: embedding, multilayer perceptron, prediction, and log-loss training.

Feature embedding

  • One-hot feature inputs are mapped to dense, low-dimensional embeddings before complex transformations.The embedding matrix maps n features into dimension d.Source: §3 · p. 3

MLP

  • The MLP stacks feed-forward layers over concatenated feature embeddings to learn high-order interactions.The layers use ReLU activation.Source: §3 · p. 3

Prediction and loss

  • A prediction layer maps the final feed-forward representation through a sigmoid to produce a CTR estimate, trained with log loss.The output is a predicted value in (0, 1).Source: §3 · p. 3

4 METHODOLOGY

HCNet uses a multi-hash codebook to store and recover cross-feature representations, then shrinks them for use in a DNN. MemoNet combines HCNet with a DNN backbone, while key interaction fields reduce the number of computed pairs.

4.1 Multi-Hash Codebook

  • A multi-hash codebook stores cross-feature representations in codewords, using multiple addresses to distinguish features despite codeword collisions.The representation is segmented into chunks indexed by different hash functions; unlike bins, codewords can hold many collision items and HCNet aims to recover each item's exact representation.Source: §4.1 · pp. 3–4

4.2 HCNet

  • HCNet has three phases: multi-hash addressing, memory restoring, and feature shrinking.Addressing locates codebook chunks for cross features; shrinking compresses pairwise features from a 1 × f(f−1)d layer to a 1 × fd embedding.Source: §4.2 · pp. 4–5
  • LMR maps flattened codeword embeddings into a cross-feature representation, while AMR uses the corresponding first-order embeddings to guide recovery.The authors report AMR performs better with relatively small codebooks, while LMR performs better as codeword count increases.Source: §4.2 · p. 5

4.3 MemoNet

  • MemoNet combines HCNet for memorizing cross features with a DNN responsible for generalizing across first-, second-, and higher-order features.The authors describe this division of labor as reducing the DNN's learning difficulty; HCNet is also proposed as a plug-in for other CTR models.Source: §4.3 · pp. 5–6

4.4 Key Interaction Field

  • Key interaction fields are the small subset of fields identified as contributing most of HCNet's interaction performance gain.Restricting pairwise interactions to these fields reduces the count from f × (f−1)/2 to m × (f−1)/2; the paper describes feature-number and field-attention ranking approaches.Source: §4.4 · p. 6

5 EXPERIMENTAL RESULTS

Experiments use three real-world CTR datasets, AUC and Logloss metrics, and comparisons against a set of established CTR models.

5.1 Experiment Setup

  • The evaluation covers three real-world datasets and reports AUC and Logloss.The datasets are randomly split 8:1:1 into training, validation, and testing.Source: §5.1 · p. 6
  • The datasets are Avazu, KDD12, and Criteo, with field counts and feature types described for each.Avazu has 23 fields, KDD12 has 13, and Criteo has 26 categorical and 13 continuous feature fields.Source: §5.1 · p. 6
  • MemoNet is compared with FM, Wide&Deep, DNN, DeepFM, DCN, AutoInt, DCN V2, xDeepFM, FiBiNet, CAN, and MaskNet.Other models, including LR, FwFM, NFM, AFM, and CCPM, are omitted because more recent models were reported to outperform them in FuxiCTR experiments.Source: §5.1 · p. 6
  • HCNet's default setup uses two hash functions and one million codewords, except for KDD12, which uses 0.5 million.Embedding dimensions are fixed by dataset, and DNN models use three hidden layers of 400 ReLU neurons.Source: §5.1 · p. 6

5.2 Performance Comparison

Small MemoNet variants outperform the baselines across all benchmarks, and larger codebooks further improve performance. The authors attribute the gains to memorizing cross features rather than simply increasing model size.

5.2 Performance Comparison

  • +0.91, +1.06, and +0.57 AUC: the best small MemoNet models outperform the DNN backbone on the three datasets, respectively.Both small variants, MemoNet-L and MemoNet-A, achieve the best results on all benchmarks.Source: §5.2 · p. 7
  • +0.75, +0.71, and +0.32 AUC: best-performance MemoNets outperform the best baselines on the three datasets, respectively.The authors note that a 0.1-level AUC improvement is difficult in real-life CTR tasks.Source: §5.2 · p. 7
  • Big MemoNet, with 500M and 100M parameters in HCNet on Avazu and Criteo, respectively, continues to improve over small MemoNet; best-performing MemoNets beat best baselines by +0.75, +0.71, and +0.32 absolute AUC.Source: §5.2 · p. 7
  • Small MemoNet increases model size by 65%, 9%, and 100% over DNN, while simply enlarging the DNN yields only 0.1–0.2 AUC improvement.The authors report that further increasing baseline model size can reduce performance through overfitting.Source: §5.2 · p. 7
  • MemoNet’s memory is presented as retaining long-tail useful cross-feature representations that mining-based methods such as Wide & Deep and AutoFIS select only in limited numbers.The authors contrast memorizing cross features with modeling them through sub-networks, citing CAN and DCN v2 as examples.Source: §5.2 · pp. 7–8

5.3 Effect of Codeword Number

Increasing codebook size improves MemoNet-L and MemoNet-A on Avazu and Criteo through the tested range, while the best codeword count varies by dataset.

5.3 Effect of Codeword Number

  • At 100M codewords, Avazu reaches 79.91% AUC and Criteo reaches 81.52% AUC; both models continue improving through the tested range.The sweep stopped at 100M on these datasets because of hardware-resource limits; KDD12’s optimal codeword count is 0.5M.Source: §5.3 · p. 8
  • MemoNet-A outperforms MemoNet-L with smaller codebooks, but the comparison reverses after dataset-specific turning points: 0.2M on KDD12 and Criteo, and 5M on Avazu.The paper links attentive restoring’s small-codebook advantage to more cross features sharing each codeword embedding.Source: §5.3 · p. 8

5.4 Effect of Hash Function Number

The effect of hash-function count on MemoNet-L depends on the dataset.

5.4 Effect of Hash Function Number

  • MemoNet-L performance fluctuates within a limited range on Criteo, rises continuously on Avazu, and rises below five hash functions before fluctuating on KDD12.The authors suggest Avazu contains more useful long-tail cross features than the other datasets.Source: §5.4 · p. 8

5.5 Plugging HCNet into SOTA Models

Adding HCNet to strong CTR models improves their reported performance. Ranking fields with FNR or FAR identifies beneficial interactions, with FAR’s top four fields capturing much of the gain from using all fields.

5.5 Plugging HCNet into SOTA Models

  • Enhanced xDeepFM, DCN v2, and MaskNet significantly outperform their original versions after incorporating 2-order HCNet.Table 3 reports AUC performance gains for the enhanced models.Source: §5.5 · p. 9, Table 3 · p. 9

5.6 Effect of Methods to Identify KIFs

  • FNR and FAR identify beneficial fields; FAR’s top four fields capture 93.90%, 74.58%, and 65.79% of HCNet’s all-field performance gain on Avazu, Criteo, and KDD12, respectively.Adding only a small fraction of fields can obtain the majority of the gain and reduce HCNet’s time complexity.Source: §5.6 · p. 9, Figure 5 · p. 9

5.7 Online Deployment and A/B Test

The paper reports online gains from deploying 2-order HCNet as a plug-in, and finds that adding higher-order HCNet consistently improves MemoNet-L performance across three datasets.

5.7 Online Deployment and A/B Test

  • CTR improved 9.73% and Watch Time improved 6.86% versus the last deployed DNN-based production model in a 14-day online A/B test.The deployment used only 4 KIFs among 96 feature fields, with online latency increasing 10.33%.Source: §5.7 · p. 9

5.8 Effect of Higher Order HCNet

  • Performance consistently increased on all three datasets when higher-order HCNet was added to MemoNet-L.Each higher-order HCNet received 1M codewords of new storage; the figure presents the benchmark performances.Source: §5.8 · p. 9, Figure 6 · p. 9

6 CONCLUSION AND FUTURE WORK

The paper introduces HCNet for memorizing cross-feature representations and combines it with a DNN backbone in MemoNet; experiments on three public datasets show MemoNet outperforming state-of-the-art models.

6 CONCLUSION AND FUTURE WORK

  • MemoNet combines HCNet with a DNN backbone and outperforms state-of-the-art models on three public datasets.The authors identify exploration of more complex codebook structures as future work.Source: §6 · p. 9

Related papers

Loading 2211.01334v3…